Release date: 11 April 2016
Release type: major
Just over a year after the last major release (MSNoise 1.3) we are proud to announce the new MSNoise 1.4. It is a major release, with a massive amount of work since the last one: in GitHub numbers , it’s over 125 commits and about 5500 new lines of code and documentation added!
MSNoise 1.4 introduces four major new features : a new ultra-intuitive web-based admin interface, the support for plugins and extensions, the phase weighted stack and the instrument response removal. It also brings the possibility to parallel/thread process the cross-correlation and the MWCS steps. MSNoise is now “tested” automatically on Linux (thanks to TravisCI) & Windows (thanks to Appveyor), for Python versions 2.7, 3.4 and 3.5. Yes, MSNoise is Python 3 compatible !!!
This version has benefited from outputs/ideas/pull requests/questions from several users/friends:
Raphaël De Plaen
Lukas E. Preiswerk
all others (don’t be mad :-) )
Thanks to all for using MSNoise, and please, let us know why/how you use it (and please cite it!)!
To date, we found/are aware of 12 publications using MSNoise ! That’s the best validation of our project ever ! See the full list on the MSNoise website.
Thomas & Corentin
PS: if you use MSNoise for your research and prepare publications, please consider citing it:
Lecocq, T., C. Caudron, et F. Brenguier (2014), MSNoise, a Python Package for Monitoring Seismic Velocity Changes Using Ambient Seismic Noise, Seismological Research Letters, 85(3), 715‑726, doi:10.1785/0220130073.
For this release, we have replaced the Configurator by a more intuitive web- based configuration interface. All fields present in the Configurator are present, and more !
easier to customise
less bandwidth when working remotely
removes dependency for traits/traitui
allows to customise “Views” to provide more information
allows the validation of fields before saving to database
“home” page shows the status of the Data Availability and of the Jobs
will allow interactive plotting in the future
adds dependency to flask & flask-admin
See MSNoise Admin (Web Interface) for more details !
MSNoise supports Plugins, this means the default workflow “from archive to dv/v” can be branched at any step (well, currently at 3 steps)!
A plugin is a python package, properly structured, that can be imported from msnoise, i.e. it has to be “installed” like any other python package.
After installing a plugin, its package name must be declared in the
plugins parameter in the configuration. This must be done PER PROJECT.
Pythonly speaking, plugins declare entry points that MSNoise uses to define
commands and job types. This way, the
msnoise plugin command would populate
with the plugin commands and plugin’s custom Job Types will be declared at
different steps of the code. Currently, custom job types can be defined at three
moments of the workflow: “when a new file is detected in the archive”,
“when a new pair-job has to be done” and “when a new REF stack has been
computed”. More will be added in the future releases.
See Extending MSNoise with Plugins to learn how to create this Amazing plugin:
$ msnoise plugin amazing Usage: msnoise-script.py plugin amazing [OPTIONS] COMMAND [ARGS]... Example Amazing Plugin for MSNoise Options: --help Show this message and exit. Commands: sayhi A Very Polite Command
and its command too:
$ msnoise plugin amazing sayhi Hi
Largely untested, not cross-validated. It looks good, but that doesn’t mean a lot, does it? Use with Caution! And if you cross-validate it, please let us know!!
The Phase Weighted Stack has been implemented. When configured, it is applied
on both the “N minutes windows to daily stacks”,on the “moving window stacks”
and on the “reference stacks”. It is thus applied at the
stack_method is ‘linear’, then a simple mean CFF is done.
On the other hand, if
stack_method is ‘pws’, then all the Phase Weighted
Stack (PWS) is computed and saved as the mov or ref CCF.
The PWS is done in two steps: first the mean coherence between the instataneous
phases of all windows is calculated, and eventually serves a weighting factor
on the mean. The smoothness of this weighting array is defined using the
pws_timegate parameter in the configuration. The weighting array is the
power of the mean coherence array. If
pws_power is equal to 0, a linear
stack is done (then it’s faster to do set
stack_method = ‘linear’). Usual
value is 2.
Schimmel, M. and Paulssen H., “Noise reduction and detection of weak, coherent signals through phase-weighted stacks”. Geophysical Journal International 130, 2 (1997): 497-505.
The following example has been provided by Carmelo Sammarco (thanks for testing/debugging!, but COME ONE, why SAC ??!!). There is maybe a too narrow time-gate on the final stack, but overall, the amplitude of the non-coherent phases are greatly diminished. Each of the three rows show the “REF” stack and its FTAN image. The three rows are:
A: Linear stack of “windows” to “daily” - Linear stack of “daily” to “ref”
B: PWS stack of “windows” to “daily” - Linear stack of “daily” to “ref”
C: PWS stack of “windows” to “daily” - PWS stack of “daily” to “ref”
To obtain an LIN-LIN or PWS-PWS stack, simply set
stack_method = ‘linear’ or
‘pws’, respectively when running the
And for mixed cases LIN-PWS or PWS-LIN, edit the config between the two steps!
During pre-processing of the waveforms (at
compute_cc step), the instrument
response can be corrected, based on those parameters:
remove_response: Remove instrument response Y/[N] (default=N)
response_format: Remove instrument file format [dataless]/inventory/paz/resp (default=dataless)
response_path: Instrument correction file(s) location (path relative to db.ini), defaults to ‘./inventory’, i.e. a subfolder in the current project folder. | All files in that folder will be parsed. (default=inventory)
response_prefilt: Remove instrument correction pre-filter (0.005, 0.006, 30.0, 35.0) (default=(0.005, 0.006, 30.0, 35.0))
Currently, only dataless seed and inventory XML are supported.
Removing the instrument response is a computationally very expensive task and not useful for dv/v iff your instruments didn’t change during the analysed period. It is also not needed for tomography iff all instruments are the same, or at least have an identical phase response in the frequency band of interest.
msnoise admin: new command to start the web interface
msnoise config: accepts a
--set name=valueoption, to rapidly change a configuration parameter in the database.
msnoisecommand accepts a
-coption that triggers the “custom” mode, currently only for plots. See below.
msnoise compute_mwcs: support the
-tthreading flag and shoud be able to work multiple threads. Example calls:
msnoise -t 4 compute_ccor
msnoise -t 16 compute_mwcs. Don’t start more threads than the actual number of real cores on your machine, and take into account that if each instance loads a lot of data (stations), you shoud have have enough RAM to store it.
msnoise infonow returns the location where the MSNoise package is installed (useful when developping / hacking the code). It also returns the list of configured Filters and Stations in the database.
All commands are now documented: Help on the msnoise commands.
All plots commands can be overriden using a -c agument in front of the plot command !!
msnoise -c plot distance
msnoise -c plot ccftime YA.UV02 YA.UV06 -m 5
To make this work, one has to copy the plot script from the msnoise install directory to the project directory (where your db.ini file is located), then edit it to one’s desires. The first thing to edit in the code is the import of the MSNoise API:
from ..api import *
from msnoise.api import *
and it should work.
Improvements in terms of performances have also been done for MSNoise 1.4:
keep_all: if set to
Y(=True) in the config, all CCF are now stored in a single HDF5 file, which makes it much nicer to backup/transfer/delete. – XXX not used actually !!!
compute_cc: reversed the change done in 1.3, the pre-whitening of the traces is now disabled, it led to very high memory usage and needs a fresh rewrite. This doens’t mean whitening is no longer done, but just some sort of caching of the pre-whitened traces.
Some users will want to keep their current project without recomputing everything. This requires adding a few configuration parameters to the database
Running the following command will take care of the upgrade from 1.3 to 1.4:
There was a bug, mainly present in MySQL, with too sharp rounding of station
coordinates. The bugfix change is done automatically for MySQL databases.
It is a little different if you are using SQLite as it
can’t be done automatically. This is because SQLite doesn’t support “ALTER”
commands. Ultimately we want the
station.Y to be be of
double. You will have to do this operation manually:
Do the following at your own risk. It might not be needed! From the tests we ran, it seems the coordinates rounding error was present only for MySQL databases!
Open SQLite database browser (SQLiteManager extension for Firefox, for example)
Open the msnoise.sqlite file
Select the station table
Xfield and change its type to
Yfield and change its type to
Ignore the warnings (it should work, although it could fail!)
Close the database
MSNoise team is
1 developper (Thomas)
1 dedicated debugger (Corentin)
less than 5 really active users, providing feedback and/or lines of codes (Esteban, Raphaël, Aurélien, Carmelo, …)
All software engineering ideas are coming from too infrequent beerstormings between Thomas & others
The web-interface and the plugin support were developed during Thomas’ holidays
If you need help, please ask your question on the mailing list. Don’t be afraid to ask. If you have ideas, please share them. If you develop codes to supplement MSNoise, please share them, even if very small, even if you don’t master gitHub. If you have complaints, post them too, but remember that the package you are using has been coded by 1 person, and that it’s not his full time job. So MSNoise is provided “as-is”, carefully written and tested, but there will be bugs, issues, incompatibility with certain python installations, OS or module versions. If you want or need developments made, contact Thomas via email directly. If these developments are within the focus of the developers’ research, then a collaboration, i.e. resulting in a co-authored peer reviewed publication, can be an option. Otherwise, you can contract us for paid-developments.