ENCODE-DCC/caper: v0.8.0

Deprecated parameters:

• --use-netrc: Autouri defaults to use ~/.netrc.
• --http-user and --http-password: Use ~/.netrc to access private URLs

Change of parameters:

• --use-gsutil-over-aws-s3 -> --use-gsutil-for-s3: Autouri uses gsutil CLI only for direct transfer between S3 and GCS buckets. Otherwise, it always use Python libraries like google-cloud-storage and boto3.

Added parameters:

• --debug and --verbose: For better logging.

Localization and preventing repetitive file transfer

• When a new localization module makes a copy of source on destination cache directory, it compares md5 hash of source and destination if a file already exists on destination. All bucket URIs (s3://, gs://) and most of URLs provide md5 hash information in their headers. If md5 hash of those match, Caper skips unnecessary file transfer. For local paths, Caper calculate md5 hash of them and store md5 hash string in .md5 file since md5 hash calculation is expensive. This happens only when Caper writes on a local storage (i.e. when localizing files on local cache). .md5 file is not valid if its modification time (mtime) is older than the file itself.
• If md5sum comparison fails, Caper compares file sizes and mtimes instead. If file sizes match and mtime is newer for destination then Caper skips file transfer.

File locking

• Caper uses a stable file locking tested up to multiple threads (50 for local, 10 for cloud URIs) competing to write on the same file.

Automatic subworkflow zipping

• Fixed bugs in old auto-zipping module.
• Caper can automatically zip subworkflow WDLs imported in the main WDL. It can also be manullay defined by users in command line arguments --imports. Caper will skip auto-zipping if --imports is defined.
• Enabled for caper submit only. i.e. caper run does not use any automatic subworkflow zipping since it is assumed that all sub-WDLs are already localized for caper run.

Womtool validation

• If --imports is defined or there is an auto-zipped subworkflow WDLs, then Caper creates a temporary directory and put the main WDL and unpack the zip file there. And then Caper runs Womtool to validate those WDLs.
• You can still skip Womtool validation with --ignore-womtool.