Published December 4, 2021
| Version 0.9.0
Software
Open
ropensci/targets: Data version control in the cloud
Authors/Creators
- 1. Eli Lilly and Company @EliLillyCo
- 2. University of Toronto
- 3. @wildlifeevoeco
- 4. Manaaki Whenua
- 5. University of Cambridge
- 6. Human Predictions LLC
- 7. @cynkra
- 8. University of Leeds
- 9. @jumpingrivers
Description
targets 0.9.0
Highlights
- Track the version ID of AWS S3-backed targets if the bucket is version-enabled (#711). If you put your targets in AWS and the metadata and code under version control, you can
git checkouta different branch of your code and all you targets will stay up to date. - Refactor the AWS path format internally. It now consists of arbitrarily extensible key-value pairs so more AWS S3 functionality may be added more seamlessly going forward (#711).
- Switch the AWS S3 backend to
paws(#711).
- Add a
regionargument totar_resources_aws()to allow the user to explicitly declare a region for each AWS S3 buckets (@caewok, #681). Different buckets can now have different regions. This feature required modifying the metadata path for AWS storage formats. Before, the first element of the path was simply the bucket name. Now, it is internally formatted like"bucket=BUCKET:region=REGION", whereBUCKETis the user-supplied bucket name andREGIONis the user-supplied region name. The newtargetsis back-compatible with the old metadata format, but if you run the pipeline withtargets>= 0.8.1.9000 and then downgrade totargets<= 0.8.1, any AWS targets will break. - Add new reporters
timestamp_positives"and"verbose_positives"that omit messages for skipped targets (@psanker, #683). - Implement
tar_assert_file(). - Implement
tar_reprex()for creating easier reproducible examples of pipelines. - Implement
tar_store()to get the path to the store of the currently running pipeline (#714, @MilesMcBain). - Automatically write a
_targets/user/folder to encouragegittargetsusers to put custom files there for data version control.
- Make sure
tar_path()uses the current store path of the currently running pipeline instead oftar_config_get("store")(#714, @MilesMcBain).
- Refactor the automatic
.gitignorefile inside the data store to allow the metadata to be committed to version control more easily (#685, #711). - Document target name requirements in
tar_target()andtar_target_raw()(@tjmahr, #679). - Catch and relay any the error if a target cannot be checked in
target_should_run.tar_builder(). These kinds of errors sometimes come up with AWS storage. - Fix the documentation of the reporters.
- Only write
_targets/.gitignorefor new data stores so the user can delete the.gitignorefile without it mysteriously reappearing (#685).
Files
ropensci/targets-0.9.0.zip
Additional details
Related works
- Is supplement to
- https://github.com/ropensci/targets/tree/0.9.0 (URL)