There is a newer version of the record available.

Published June 18, 2018 | Version v5.2.0
Software Open

ropensci/drake: Parallel computing improvements

Creators

  • 1. Eli Lilly and Company @EliLillyCo
  • 2. Indiana Commission for Higher Education
  • 3. University of California at Berkeley Agricultural and Resource Economics
  • 4. University of Zürich
  • 5. MIT
  • 6. Generali China AMC @GCAMC
  • 7. LECA Grenoble

Description

  • Sequester staged parallelism in backends "mclapply_staged" and "parLapply_staged". For the other lapply-like backends, drake uses persistent workers and a master process. In the case of "future_lapply" parallelism, the master process is a separate background process called by Rscript.
  • Remove the appearance of staged parallelism from single-job make()'s. (Previously, there were "check" messages and a call to staged_parallelism().)
  • Remove uncontained remnants of staged parallelism internals.
  • Allow different parallel backends for imports vs targets. For example, make(parallelism = c(imports = "mclapply_staged", targets = "mclapply").
  • Fix a bug in environment pruning. Previously, dependencies of downstream targets were being dropped from memory in make(jobs = 1). Now, they are kept in memory until no downstream target needs them (for make(jobs = 1)).
  • Improve predict_runtime(). It is a more sensible way to go about predicting runtimes with multiple jobs. Likely to be more accurate.
  • Calls to make() no longer leave targets in the user's environment.
  • Attempt to fix a Solaris CRAN check error. The test at https://github.com/ropensci/drake/blob/b4dbddb840d2549621b76bcaa46c344b0fd2eccc/tests/testthat/test-edge-cases.R#L3 was previously failing on CRAN's Solaris machine (R 3.5.0). In the test, one of the threads deliberately quits in error, and the R/Solaris installation did not handle this properly. The test should work now because it no longer uses any parallelism.
  • Deprecate the imports_only argument to make() and drake_config() in favor of skip_targets.
  • Deprecate migrate_drake_project().
  • Deprecate max_useful_jobs().
  • For non-distributed parallel backends, stop waiting for all the imports to finish before the targets begin.
  • Add an upstream_only argument to failed() so users can list failed targets that do not have any failed dependencies. Naturally accompanies make(keep_going = TRUE).
  • Add an RStudio R Markdown template compatible with https://krlmlr.github.io/drake-pitch/.
  • Remove plyr as a dependency.
  • Handle duplicated targets better in drake_plan() and bind_plans().
  • Add a true function target() to help create drake plans with custom columns.
  • In drake_gc(), clean out disruptive files in storrs with mangled keys (re: https://github.com/ropensci/drake/issues/198).
  • Move all the vignettes to the up and coming user manual: https://ropenscilabs.github.io/drake-manual/
  • Rename the "basic example" to the "mtcars example".
  • Deprecate load_basic_example() in favor of load_mtcars_example().
  • Refocus the README.md file on the main example rather than the mtcars example.
  • Use a README.Rmd file to generate README.md.
  • Add function deps_targets().
  • Deprecate function deps() in favor of deps_code()
  • Add a pruning_strategy argument to make() and drake_config() so the user can decide how drake keeps non-import dependencies in memory when it builds a target.
  • Add optional custom (experimental) "workers" and "priorities" columns to the drake plans to help users customize scheduling.
  • Add a makefile_path argument to make() and drake_config() to avoid potential conflicts between user-side custom Makefiles and the one written by make(parallelism = "Makefile").
  • Document batch mode for long workflows in the HPC guide.
  • Add a console argument to make() and drake_config() so users can redirect console output to a file.
  • Make it easier for the user to find out where a target in the cache came from: show_source(), readd(show_source = TRUE), loadd(show_source = TRUE).

Files

ropensci/drake-v5.2.0.zip

Files (2.4 MB)

Name Size Download all
md5:9195e60a435d89e300d3b9fd43a95c2b
2.4 MB Preview Download

Additional details

Related works

Is supplement to
https://github.com/ropensci/drake/tree/v5.2.0 (URL)