bety: BETYdb 4.6

Summary of Changes

New API and Use of Proper Datatypes

This release adds a new versioned API for adding new trait data and for querying most all data. It updates measurement and statistical data to use the proper datatypes and adds and modifies many features to make maintenance of BETYdb easier.

Administrators need to do a database migration.

See "Database Changes" below.

New API

A new API (version v0) has been added for obtaining data in both JSON and XML formats. The new API also supports insertion of new trait data (provided in either JSON or XML format), something not provided by previous APIs. Documentation of this API is forthcoming. (A brief introduction is given in some of the comments to GitHub issues https://github.com/PecanProject/bety/issues/381 and https://github.com/terraref/computing-pipeline/issues/52.)

New Features Docker files for application packaging

Docker files have been added.

Travis

A Travis configuration file has been added in order to hook BETYdb into the Travis CI continuous integration service. This service will automatically run all of the non-Javascript based RSpec tests each time a new commit is pushed to GitHub.

New and Modified Rake Tasks

• A new Rake task bety:db:populate has been added to make it easier for developers to get a new BETYdb instance up and running or to get a fresh copy of the BETYdb database. This is a wrapper for Pecan's load.bety.sh script and replaces the wrapper script update-betydb.sh for most use-cases relevant to the BETYdb developer.

• db:test:load_structure has been modified to behave correctly when using the postgis adapter. It will now drop the test database if it exists and then recreate it and load the database schema defined by the *_structure.sql file corresponding to the current environment. This makes it easier to set up the test database for testing, even if the other databases have not been set up.

• A description has been added for the Rake task db:test:clone_structure.

• The task db:fixtures:load has been disabled for all but the test environment. This will help prevent data loss caused by inadvertently running it in the wrong environment.

• The db:create task now only creates the database corresponding to the current environment. (When Rails.env was "development", it used to also create the test database. Though perhaps useful, this seemed counterintuitive.)

Bug Fixes

• Some missing model associations have been added. This should facilitate getting useful information from related tables via an API call.

• The script script/update-betydb.sh has been updated to make it consistend with the load.bety.sh script from the Pecan repository, for which it serves as a wrapper.

• Some fixes have been made to the Add-SQL-Geometry Python script (script/buildSQLGeom.py).

• A bug has been fixed in the header-normalization routine of the Bulk Upload wizard whereby any header (e.g. a trait name) containing the consecutive letters 'se' was getting capitalized. Now only the string 'se' itself is normalized (to 'SE'—Standard Error).

Steps Needed for Upgrade Database Changes

Administrators need to do database migrations!

The migration ChangeNumericsToFloats changes the datatype of all columns that were previously arbitrary precision numbers of type NUMERIC(16, 4) into double precision floating-point types. The columns in question are those used to represent values of trait, covariate, and yield measurements and the values of associated statistics (and also the parameter columns in the priors table). This helps to make the number of significant digits recordable for a particular measurement independent of the unit used for recording the measurement. Moreover, as the PostgreSQL documentation points out, "arithmetic on numeric values is very slow compared ... to the floating-point types", so using floating-point types represents an improvement if computational features of SQL are to be used.

The database version for this release is 20160303221049.

Gem Installation

Administrators need to run the bundler to install new Ruby Gems related to the new API.

Status of RSpec Tests

• All tests continue to pass when run in the default environment and can be run using the command

  bundle exec rspec

• Pass/failure status of the latest commits to GitHub are now shown at https://travis-ci.org/PecanProject/bety. Only the non-JavaScript based tests are run by Travis. To check the others, run

  bundle exec rspec -t js

Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Automated-Tests