Create reproducible digests of objects in R

Not all aspects of R objects are captured by current hashing tools in R (e.g. digest::digest, fastdigest::fastdigest, knitr caching, archivist::cache). This is mostly because many objects have "transient" (e.g., functions have environments), or "disk-backed" features. This function allows for these accommodations to be made and uses fastdigest internally. Since the goal of using reproducibility is to have tools that are not session specific, this function attempts to strip all session specific information so that the fastdigest works between sessions and operating systems. It is tested under many conditions and object types, there are bound to be others that don't work correctly.

.robustDigest(object, objects, length = 1e+06, algo = "xxhash64",
  quick = getOption("reproducible.quick", FALSE), classOptions = list())

# S4 method for ANY
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for cluster
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for `function`
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for expression
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for character
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for Path
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for environment
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for list
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for data.frame
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for Raster
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

# S4 method for Spatial
.robustDigest(object, objects, length = 1e+06,
  algo = "xxhash64", quick = getOption("reproducible.quick", FALSE),
  classOptions = list())

Arguments

object	an object to digest.
objects	Optional character vector indicating which objects are to be considered while making digestible. This argument is not used in the default cases; the only known method that uses this in the default cases; the only known method that uses this argument is the simList class from SpaDES.core.
length	Numeric. If the element passed to Cache is a Path class object (from e.g., asPath(filename)) or it is a Raster with file-backing, then this will be passed to digest::digest, essentially limiting the number of bytes to digest (for speed). This will only be used if quick = FALSE.
algo	The algorithms to be used; currently available choices are md5, which is also the default, sha1, crc32, sha256, sha512, xxhash32, xxhash64 and murmur32.
quick	Logical. If TRUE, little or no disk-based information will be assessed, i.e., mostly its memory content. This is relevant for objects of class Path and Raster currently. For class Path objects, the file's metadata (i.e., filename and file size) will be hashed instead of the file contents. If set to FALSE (default), the contents of the file(s) are hashed. If quick = TRUE, length is ignored. NOTE: this argument is experimental and may change in future releases.
classOptions	Optional list. This will pass into .robustDigest for specific classes. Should be options that the .robustDigest knows what to do with.

Value

A hash i.e., digest of the object passed in.

Classes

Raster* objects have the potential for disk-backed storage. If the object in the R session is cached using archivist::cache, only the header component will be assessed for caching. Thus, objects like this require more work. Also, because Raster* can have a built-in representation for having their data content located on disk, this format will be maintained if the raster already is file-backed, i.e., to create .tif or .grd backed rasters, use writeRaster first, then Cache. The .tif or .grd will be copied to the "raster" subdirectory of the cacheRepo. Their RAM representation (as an R object) will still be in the usual gallery/ directory. For inMemory raster objects, they will remain as binary .RData files.

Functions (which are contained within environments) are converted to a text representation via a call to format(FUN).

Objects contained within a list or environment are recursively hashed using fastdigest, while removing all references to environments.

Character strings are first assessed with dir.exists and file.exists to check for paths. If they are found to be paths, then the path is hashed with only its filename via basename(filename). If it is actually a path, we suggest using asPath(thePath)

See also

cache.

fastdigest.

Examples

a <- 2
tmpfile1 <- tempfile()
tmpfile2 <- tempfile()
save(a, file = tmpfile1)
save(a, file = tmpfile2)

# treats as character string, so 2 filenames are different
fastdigest::fastdigest(tmpfile1)
#> [1] "5cb44abc5f5a24029d0361c023cdc657"
fastdigest::fastdigest(tmpfile2)
#> [1] "1e4de8e672b759da47a007172b5c49c4"

# tests to see whether character string is representing a file
.robustDigest(tmpfile1)
#> [1] "d608ee53f4355b64"
.robustDigest(tmpfile2) # same
#> [1] "d608ee53f4355b64"

# if you tell it that it is a path, then you can decide if you want it to be
#  treated as a character string or as a file path
.robustDigest(asPath(tmpfile1), quick = TRUE)
#> [1] "611b3ab7e3e918165aba8dd80334d03a"
.robustDigest(asPath(tmpfile2), quick = TRUE) # different because using file info
#> [1] "325cecdd64e8e8841673e2cc8672c441"

.robustDigest(asPath(tmpfile1), quick = FALSE)
#> [[1]]
#> [1] "d608ee53f4355b64"
#> 
.robustDigest(asPath(tmpfile2), quick = FALSE) # same because using file content
#> [[1]]
#> [1] "d608ee53f4355b64"
#> 

# Rasters are interesting because it is not know a priori if it
#   it has a file name associated with it.
library(raster)
r <- raster(extent(0,10,0,10), vals = 1:100)

# write to disk
r1 <- writeRaster(r, file = tmpfile1)
r2 <- writeRaster(r, file = tmpfile2)

digest::digest(r1)
#> [1] "64ddf6f9bb1b2085fcb9b909ee751649"
digest::digest(r2) # different
#> [1] "cb591ca99e6fc42ec7647ddb76a90c95"
fastdigest::fastdigest(r1)
#> [1] "c47f1cc4b70125b3a3cbb8d963fa9a6e"
fastdigest::fastdigest(r2) # different
#> [1] "da7e24a412747d2dcf54b6ffe87c1484"
.robustDigest(r1)
#> [1] "ad6655dc935a001207ba8fd3fa9d5b15"
.robustDigest(r2) # same... data are the same in the file
#> [1] "ad6655dc935a001207ba8fd3fa9d5b15"

# note, this is not true for comparing memory and file-backed rasters
.robustDigest(r)
#> [1] "f6dddb6b1d2896b041a37cef3a9b3d1e"
.robustDigest(r1) # different
#> [1] "ad6655dc935a001207ba8fd3fa9d5b15"