Download requests in a queue
occ_download_queue(..., .list = list(), status_ping = 10)
... | any number of |
---|---|
.list | any number of |
status_ping | (integer) seconds between pings checking status of the download request. generally larger numbers for larger requests. default: 10 (i.e., 10 seconds). must be 10 or greater |
a list of occ_download
class objects, see occ_download_get()
to fetch data
This function is a convenience wrapper around occ_download()
,
allowing the user to kick off any number of requests, while abiding by
GBIF rules of 3 concurrent requests per user.
see downloads for an overview of GBIF downloads methods
It works by using lazy evaluation to collect your requests into a queue. Then it kicks of the first 3 requests. Then in a while loop, we check status of those requests, and when any request finishes, we kick off the next, and so on. So in theory, there may not always strictly be 3 running concurrently, but the function will usually provide for 3 running concurrently.
This function is still in development. There's a lot of complexity to this problem. We'll be rolling out fixes and improvements in future versions of the package, so expect to have to adjust your code with new versions.
# NOT RUN { # passing occ_download() requests via ... out <- occ_download_queue( occ_download('taxonKey = 3119195', "year = 1976"), occ_download('taxonKey = 3119195', "year = 2001"), occ_download('taxonKey = 3119195', "year = 2001", "month <= 8"), occ_download('taxonKey = 5229208', "year = 2011"), occ_download('taxonKey = 2480946', "year = 2015"), occ_download("country = NZ", "year = 1999", "month = 3"), occ_download("catalogNumber = Bird.27847588", "year = 1998", "month = 2") ) # supports <= 3 requests too out <- occ_download_queue( occ_download("country = NZ", "year = 1999", "month = 3"), occ_download("catalogNumber = Bird.27847588", "year = 1998", "month = 2") ) # using pre-prepared requests via .list keys <- c(7905507, 5384395, 8911082) queries <- list() for (i in seq_along(keys)) { queries[[i]] <- occ_download_prep( paste0("taxonKey = ", keys[i]), "basisOfRecord = HUMAN_OBSERVATION,OBSERVATION", "hasCoordinate = true", "hasGeospatialIssue = false", "year = 1993" ) } out <- occ_download_queue(.list = queries) out # another pre-prepared example yrs <- 1930:1934 length(yrs) queries <- list() for (i in seq_along(yrs)) { queries[[i]] <- occ_download_prep( "taxonKey = 2877951", "basisOfRecord = HUMAN_OBSERVATION,OBSERVATION", "hasCoordinate = true", "hasGeospatialIssue = false", paste0("year = ", yrs[i]) ) } out <- occ_download_queue(.list = queries) out # }