eqcorrscan/EQcorrscan: EQcorrscan 0.5.0

This release represents a significant increase in efficiency in large-scale matched-filters in EQcorrscan. Lots of work has gone in to reducing memory usage in the non-correlation components of the matched-filter workflow, streamlining the code, making better use of shared memory multi-threaded parallelism and increasing CPU loads. In our testing we can now achieve and maintain >190% CPU efficiency (e.g. >95% hyperthreaded performance). We can also better load GPUs by making use of concurrent CPU and GPU processing of workflow steps. You should not need to change your code to make use of most of these speed-ups. Hopefully you will notice that you can run larger datasets faster than even!

Changelog

• core.match_filter.tribe
  • Significant re-write of detect logic to take advantage of parallel steps (see #544)
  • Significant re-structure of hidden functions.
• core.match_filter.matched_filter
  • 5x speed up for MAD threshold calculation with parallel (threaded) MAD calculation (#531).
• core.match_filter.detect
  • 1000x speedup for retrieving unique detections for all templates.
  • 30x speedup in handling detections (50x speedup in selecting detections, 4x speedup in adding prepick time)
• core.match_filter.template
  • new quick_group_templates function for 50x quicker template grouping.
  • Templates with nan channels will be considered equal to other templates with shared nan channels.
  • New grouping strategy to minimise nan-channels - templates are grouped by similar seed-ids. This should speed up both correlations and prep_data_for_correlation. See PR #457.
• utils.pre_processing
  • _prep_data_for_correlation: 3x speedup for filling NaN-traces in templates
  • New function ``quick_trace_select` for a very efficient selection of trace by seed ID without wildcards (4x speedup).
  • process, dayproc and shortproc replaced by multi_process. Deprecation warning added.
  • multi_process implements multithreaded GIL-releasing parallelism of slow sections (detrending, resampling and filtering) of the processing workflow. Multiprocessing is no longer supported or needed for processing. See PR #540 for benchmarks. New approach is slightly faster overall, and significantly more memory efficeint (uses c. 6x less memory than old multiprocessing approach on a 12 core machine)
• utils.correlate
  • 25 % speedup for _get_array_dicts with quicker access to properties.
• utils.catalog_to_dd
  • _prepare_stream
    • Now more consistently slices templates to length = extract_len * samp_rate so that user receives less warnings about insufficient data.
  • write_correlations
    • New option use_shared_memory to speed up correlation of many events by ca. 20 % by moving trace data into shared memory.
    • Add ability to weight correlations by raw correlation rather than just correlation squared.
• utils.cluster.decluster_distance_time
  • Bug-fix: fix segmentation fault when declustering more than 46340 detections with hypocentral_separation.