MSNoise 1.3

Release date: 20 March 2015

Release type: major

Release notes:

Introduction 

8 months after the last bugfix release (MSNoise 1.2.5), and 17 months after the last major release (MSNoise 1.2) we are proud to announce the new MSNoise 1.3. It is a major release, with a massive amount of work since the last release: in GitHub numbers , it’s over 100 commits and about 3500 new lines of code and documentation added ! MSNoise 1.3 introduces a brand new way of executing the workflow. The workflow in itself doesn’t change, so experienced users as well as new users reading the SRL publication will find their way easily!

MSNoise is now a Python Package, allowing a single (and easy) install for all your projects and/or all users using pip. The new top-level msnoise command contains all the steps of the workflow, plus new additions, as the very useful reset command to easily mark all jobs “T”odo. The msnoise plot command group which includes seven plots, all directly callable from the command line, without needing to hack/edit the source codes. About hacking: MSNoise has now a proper documented API which allows pythonistas to write their own plots, computation steps, …, while interacting with the database and the data archive! The “dynamic time lag” allows to use parts of the coda that is dependent from the interstation distance (provided station coordinates are defined). Finally, MSNoise is now tested and automatically checked by Travis-CI!

This version has benefited from outputs/ideas/pull requests/questions from several users:

Rebecca Kramer
Carmelo Sammarco
Oscar Alberto Castro Artola
Kasper van Wijk
Kohtaro R. Araragi
Esteban Chaves
Adrian Shelley
Weston Thelen
Robert Abbott
Jean Battaglia
Sébastien Carniato
Xiao Wang
Lion Krisher
Tobias Megies
all participants to the 2014 Pre-AGU MSNoise workshop
all others (don’t be mad :-) )

Thanks to all for using MSNoise, and please, let us know why/how you use it (and please cite it!)!

Thomas Lecocq & Corentin Caudron

PS: if you use MSNoise for your research and prepare publications, please consider citing it:

Lecocq, T., C. Caudron, et F. Brenguier (2014), MSNoise, a Python Package for Monitoring Seismic Velocity Changes Using Ambient Seismic Noise, Seismological Research Letters, 85(3), 715‑726, doi:10.1785/0220130073.

MSNoise is a real Python Package 

This is probably the most important change since the original release of MSNoise 1.0 (August 2013), it represents a massive amount of work since the last release (1.2.5 in June 2014), and is probably the most needed by users! In GitHub numbers , it’s over 100 commits and about 3500 new lines of code (and of documentation!) added !

In practice, what does change ?

MSNoise is installable using pip or easy_install, soon using conda
MSNoise is installed in the common “site-package/” folder of one’s python install.
Once installed, it is available for all users, all projects.
It allows updating MSNoise for all projects at once.
It removes all python files from project folders, which is much cleaner.

MSNoise being installed in the standard lib directories means it shouldn’t write or output anything in those folders. To facilitate the launch of commnands a new top level msnoise command has been created, and should be available right after installing.

`msnoise` is now a top-level command 

Users of MSNoise will have to change the way they call the steps, i.e.:

python s000_installer.py becomes msnoise install
python s001_configurator.py becomes msnoise config
python s002_populate_station_table.py becomes msnoise populate
python s01_scan_archive.py becomes msnoise scan_archive
python s02_new_jobs.py becomes msnoise new_jobs
python s03_compute_cc.py becomes msnoise compute_cc
python s04_stack.py becomes msnoise stack
python s05_compute_mwcs.py becomes msnoise compute_mwcs
python s06_compute_dtt.py becomes msnoise compute_dtt

All the commands are visible using the --help argument:

msnoise --help

Usage: msnoise-script.py [OPTIONS] COMMAND [ARGS]...

Options:
  -t, --threads INTEGER  Number of threads to use (only affects modules that
                         are designed to do parallel processing)
  --help                 Show this message and exit.

Commands:
  bugreport           This command launches the Bug Report script.
  compute_cc          Computes the CC jobs (based on the "New Jobs"...
  compute_dtt         Computes the dt/t jobs based on the new MWCS...
  compute_mwcs        Computes the MWCS based on the new stacked...
  compute_stretching  [experimental] Computes the stretching based...
  config              This command launches the Configurator.
  info                Outputs general information about the current...
  install             This command launches the installer.
  ipython             Launches an ipython notebook in the current...
  new_jobs            Determines if new CC jobs are to be defined
  plot                Top level command to trigger different plots
  populate            Rapidly scan the archive filenames and find...
  reset               Resets the job to "T"odo.
  scan_archive        Scan the archive and insert into the Data...
  stack               Stacks the [REF] and/or [MOV] windows
  test                Runs the test suite, should be executed in an...
  upgrade_db          Upgrade the database from pre-1.3 to MSNoise...

The parameters/arguments of each command are explained using its own --help , for example:

msnoise reset --help

Usage: msnoise-script.py reset [OPTIONS] JOBTYPE

  Resets the job to "T"odo. ARG is [CC] or [DTT]. By default only resets
  jobs "I"n progress. --all resets all jobs, whatever the flag value

Options:
  -a, --all  Reset all jobs
  --help     Show this message and exit.

The description of each step has been updated in the documentation.

`msnoise plot`: Plotting made easy 

As explained above, msnoise is a top level command available in your command prompt. MSNoise 1.3 includes several plots which are available using the msnoise plot command. See Plotting to view all plots!

All the available plots are listed using the --help argument:

msnoise plot --help

Usage: msnoise-script.py plot [OPTIONS] COMMAND [ARGS]...

  Top level command to trigger different plots

Options:
  --help  Show this message and exit.

Commands:
  ccftime            Plots the dv/v (parses the dt/t results)
  data_availability  Plots the Data Availability vs time
  distance           Plots the REFs of all pairs vs distance
  dvv                Plots the dv/v (parses the dt/t results)
  interferogram      Plots the interferogram between sta1 and sta2...
  mwcs               Plots the mwcs results between sta1 and sta2...
  station_map        Plots the station map (very basic)

Same as above, sub-commands have their own --help:

msnoise cc plot interferogram --help

Usage:  [OPTIONS] STA1 STA2 [EXTRA_ARGS]...

  Plots the interferogram between sta1 and sta2 (parses the CCFs) STA1 and
  STA2 must be provided with this format: NET.STA !

Options:
  -f, --filterid INTEGER   Filter ID
  -c, --comp TEXT          Components (ZZ, ZE, NZ, 1E,...). Defaults to ZZ
  -m, --mov_stack INTEGER  Mov Stack to read from disk. Defaults to 1.
  -s, --show BOOLEAN       Show interactively?
  -o, --outfile TEXT       Output filename (?=auto). Defaults to PNG format,
                           but can be anything matplotlib outputs, e.g. ?.pdf
                           will save to PDF with an automatic file naming.
  -r, --refilter TEXT      Refilter CCFs before plotting (e.g. 4:8 for
                           filtering CCFs between 4.0 and 8.0 Hz. This will
                           update the plot title.
  --help                   Show this message and exit.

New functionality: Dynamic time lags 

As before, the dt/t is determined as the slope of the delays vs time lags. the slope is calculated a weighted linear regression (WLS) through selected points. The selection of points is first based on the time lag criteria. The minimum time lag can either be defined absolutely or dynamically. When dtt_lag is set to “dynamic” in the database, the inter-station distance is used to determine the minimum time lag. This lag is calculated from the distance and a velocity configured (dtt_v). The velocity is determined by the user so that the minlag doesn’t include the ballistic waves. For example if ballistic waves are visible with a velocity of 2 km/s, one could configure dtt_v=1.0. This way, if stations are located 15 km apart, the minimum lag time will be set to 15 s. The dtt_width determines the width of the lag window used. A value of 30.0 means the process will use time lags between 15 and 45 s in the example above, on both sides if configured (dtt_sides), or only causal or acausal parts of the CCF.

Note

It seems obvious that these parameters are frequency-dependent, but they are currently common for all filters !

New parameters have been added to the configuration:

dtt_lag : How is the lag window defined (default=static)
dtt_v : If dtt_lag =dynamic: what velocity to use to avoid ballistic wave (in km/s) (default=1)
dtt_minlag : If dtt_lag =static: min lag time (in seconds) (default=5)
dtt_width : Width of the time lag window (in seconds) (default=30)
dtt_sides : Which sides to use (default=both)
dtt_mincoh : Minimum coherence on dt measurement, MWCS points with values lower than that will not be used in the WLS, [0:1] (default=0.65)
dtt_maxerr : Maximum error on dt measurement, MWCS points with values larger than that will not be used in the WLS [0:1] (default=0.1)
dtt_maxdt : Maximum dt values, MWCS points with values larger than that will not be used in the WLS (in seconds) (default=0.1)

Math updates & bugfixes 

Some improvements to the maths have been done for MSNoise 1.3:

whiten: the symmetric hermitian was not properly defined and could lead to a 1 sample shift in the negative frequencies.
compute_cc: it is now possible to define an overlap of the windows
compute_cc: setting winsorizing to -1 now computes the 1-bit normalization of the trace. Reminder: 0: no normalization, N: N*rms clipping.
mwcs: the tapering of the windowed CCF has been improved in order to optimize the calculation for the center of the window.

Performance improvements 

Improvements in terms of performances have also been done for MSNoise 1.3:

new_jobs: the procedure has been completely rewritten and should be a lot faster, certainly for large (to very-large) archives and/or number of days.
keep_all: if set to Y (=True) in the config, all CCF are now stored in a single HDF5 file, which makes it much nicer to backup/transfer/delete.
compute_cc: if only ZZ components are to be computed, the whitened windows are pre-computed, which makes the process faster. This could lead to memory issues if the job contains a lot of stations, a lot of filters are configured and a large number of windows.
compute_mwcs: The procedure updates the jobs all at once, which brings a big gain in transaction time.

MSNoise has a proper API: Hacking MSNoise 

The former database_tools.py has been renamed to api.py and all the functions are now documented (in MSNoise API) so they can be used from the console or from custom scripts.

Using the msnoise ipython command, one triggers the start of an IPython notebook in the current project folder. Once in a new notebook, one could

from msnoise.api import connect, get_config
session = connect()
print get_config(session, "mov_stack")

to get the current configuration of the mov_stack parameter ! Enjoy Hacking!

MSNoise is tested 

UnitTests are now defined for some (not all, yet) MSNoise functions, and most of the workflow is tested automatically by launching msnoise test in a new folder.

This will allow contributors to easily know the code they want to be merged in the next versions of MSNoise doesn’t break anything. Travis-CI runs automatically on every push or pull request made on GitHub.

Upgrading an existing project to MSNoise 1.3 

Some users will want to keep their current project without recomputing everything. This requires:

adding a few configuration parameters to the database
modifying the structure of the jobs table.

Running the following command will do both parts for MySQL and only the first part for SQLite:

msnoise upgrade_db

The second part is a little different if you are using SQLite as it can’t be done automatically. This is because SQLite doesn’t support “ALTER” commands. Ultimately we want the jobs.type to be renamed to jobs.jobtype. You will have to do this operation manually:

Open SQLite database browser (SQLiteManager extension for Firefox, for example)
Open the msnoise.sqlite file
Select the jobs table
Edit the type field and rename it to jobtype
Ignore the warnings (it should work, although it could fail!)
Close the database

A note on parallel processing 

Although the msnoise command accepts the -t INTEGER argument to launch a number of threads in parallel, it currently only works with scan_archive: msnoise -t 4 scan_archive will run the scan on four folders in parallel. For the other steps, one has still to run multiple commands in a console. This should change in the future.