Open XDMoD Release Notes
Below is a list of Open XDMoD releases with major features and bug fixes listed.
Version 8.0.0 (2018-10-30)
Features
- General
- Added a beta version of the Cloud realm to provide metrics relevant to
cloud computing resources.
- Added a beta version of the Storage realm to provide metrics relevant to
storage systems installed at a center.
- Federated XDMoD has been released for production. Federated XDMoD allows
individual, locally managed, XDMoD instances to report all or a subset of their
accounting data to a central Hub which provides a global view of the federation.
- All XDMoD user profiles are now associated with an organization. Previously, this
was only required for Campus Champions.
- Added support for automatically detecting / assigning a new SSO User's
organization.
- Added support for automatically detecting if a user's organization has changed
and updating their accounts accordingly. This may include, but is not limited to,
the removal of elevated privileges.
- Hardened the login and password reset process as a result of a security audit by
University of Cambridge.
- Improved support for resource manager job arrays.
- Many improvements to the documentation.
- ETL
- Reorganized several ETL pipelines.
- Improved data sanitization for tighter checks present in MySQL 5.7.
- Refactored Jobs realm ingestion to utilize ETLv2.
- Standardize action names to follow the format module.pipeline.action. For
example, xdmod.acls.manage-tables.
- Added character set and coalition to table definitions.
- Added support for foreign key constraints.
- Added support for the definition of ETL variables on the command line using -d
variable=value.
- Add ingestion of node hostname data from SGE logs.
- Various ETL performance improvements.
Bug Fixes
- User Interface
- Deep linking when logged in using SSO has been restored.
- Update the logrotate configuration to use the su and create options.
- ETL
- Add primary keys to select ETL source queries.
- When modifying an existing table, preserve the order of the columns in the
definition file.
- Ensure that file handles are flushed before inserting the final chunk of
data.
- Misc
- Fixed several exceptions that were outside of a namespace.
- Fixed an issue where ACLs were not properly created on upgrade.
- Several minor bugfixes
Version 7.5.1 (2018-05-23)
Bug Fixes
- Properly implement data access for non-feature ACLs (e.g., ACLs that provide access to
data but not a feature such as a tab) that are not part of the hierarchy. For example,
the Value Analytics ACL.
Version 7.5.0 (2018-03-01)
Features
- General
- Improve performance of Utilization statistic
- Do not embed JavaScript in chart objects returned by the back end, instead
include this code directly in the user interface code
- General improvements to the performance of the REST stack
- Updated Google Captcha to v2 (v1 is now discontinued)
- Added numerous component and unit tests
- Removed unused code paths
- Update greenlion/PHP-SQL-Parser to newer release that does not autoload itself
- User Interface
- Legend item edit box is now displayed next to the legend item that is being
edited and is larger to accommodate longer strings
- Added a "Select All" button to the filter dialog
- Added PDF export capability, which greatly improves the quality of
images included in LATEX documents
- ETL
- When ingesting Slurm data, return duplicate jobs such as those that were
resubmitted due to node fail conditions
- Improve performance of filter list generation
- Application Kernels
- Several minor bug fixes and UI improvements
Bug Fixes
- General
- Fix the Show Guidelines button in the Usage tab
- Fixed an issue with data filtering when an unprivileged user tries to view
timeseries data for a restricted realm or statistic
- User Interface
- Fix the "TypeError: element is undefined" error when plotting Pie chart in
metric explorer
Version 7.1.0 (2018-02-09)
Features
- General
- Added support for Globus as a federated authentication provider
- Improvements to the user login dialog and matching of users to institutions
- Added the ability to use fine-grained ACLs for controlling access to features such
as tabs and realms as well as data
- Code pertaining to creating/sending emails has been moved to a central location.
- Improved online documentation for installations and upgrades
- Internal Admin Dashboard
- Updated user management functionality in the administrative dashboard to support
fine-grained ACLs for individual users
- Many stability improvements to the administrator dashboard, especially for
managing users
- Storage Realm
- Added alpha version of the Storage realm to track resource storage utilization
- ETL
- Added an ETL pipeline for ingesting log files generated by Eucalyptus clouds
- Added support for PBS/Torque logs where the host of a job is not included in the job's
ID string
- Support references into complex source records such as JSON objects
Bug Fixes
- General
- Improved error reporting
- Fixed several issues where JSON was not properly encoded
- For end dates that fall on the current date or in the future, do not
automatically adjust the date to the end of the aggregation period that it falls
into.
- Report Generator
- Make timeframes editable for individual charts
- ETL
- Ignore duplicate hosts found in LSF accounting log files
- Don't verify data endpoints associated with disbaled actions
- Always regenerate source data queries prior to execution to ensure that any
modified ETL variables are properly applied
- Update host list parser to ensure that empty host names are not returned
- Remove PHP memory limit when running ETL pipelines
Version 7.0.1 (2017-09-27)
Bug Fixes
- General
- Fixed compatibility with PHP 5.3.3
Version 7.0.0 (2017-09-21)
Features
- General
- Improved login prompt to automatically appear when an
unauthenticated user attempts to access any private tab
- Improved design of Metric Explorer Load Chart menu
- Re-enabled aggregate mode for "Wall Hours: Per Job" metric
- Added quarterly report template for center directors
- Job Viewer
- Added Gantt chart view of job peers
- Added ability to link directly to jobs
- SUPReMM
- Enabled dataflow diagram for Open XDMoD administrators
Bug Fixes
- General
- Fixed handling of "%" characters in Metric Explorer options when
using Firefox
- Improved handling of invalid start and end dates
- Fixed validation of length of names in contact forms
- Improved handling of jobs with "0" start or end time
- Fixed case where About tab would display a blank page when loaded
using Chrome
- Job Viewer
- Fixed directions not always appearing when all jobs are
closed
- Fixed duplicate search nodes being created when opening jobs from
Metric Explorer charts with "#" in the title
- Usage
- Fixed error bars option being enabled when error bars are not
available
- Fixed metrics appearing to be available to users that do not have
access
Miscellaneous
- Performed work in anticipation of federated instances
Version 6.6.0 (2017-05-11)
Features
- General
- Replaced realms with categories.
- Categories behave similarly to realms but, unlike realms, can
contain metrics based on different underlying data sets.
- Enhanced new PHP-based ETL system with a variety of features.
- SUPReMM
- Improved application classification code.
Bug Fixes
- General
- Fixed warning that could appear when using federated authentication.
- Fixed unnecessary rounding in processor count metrics.
- Fixed errors when attempting to preview an unsaved report.
- Fixed aggregators excluding data on certain time boundaries.
- Fixed non-aggregate metrics being allowed in aggregate datasets.
- Fixed display error with password reset form that occurred when
using federated authentication.
- Fixed "Show chart title" option in Usage tab not always working
as expected.
- Metric Explorer
- Fixed Y-axis context menu disappearing after
changing between linear and logarithmic scales.
- Fixed chart last modified time not updating.
- Fixed typing certain characters causing chart options menu to close
when using Firefox.
- Fixed chart errors not displaying if help graphic is active.
Miscellaneous
- Performed work in anticipation of allocations, accounts, and cloud
data in Open XDMoD.
Version 6.5.0 (2017-01-10)
Features
- General
- Modified the Summary tab to reload automatically after its charts
are modified in Metric Explorer.
- Added a new PHP-based ETL system to Open XDMoD.
- This ETL provides many predefined actions to support flexible
ingestion of data from multiple sources and is configured via JSON
files to support flexibility at customer installations with minimal
changes to source code.
- SUPReMM
- Added peak memory usage metric.
- Improved application identification data.
- Added a count column to the Job Viewer's detailed metrics pane to
show how many data points were used to calculate the metrics.
Bug Fixes
- General
- Fixed stacked area charts not handling empty data points correctly.
- Stopped inaccurate warning about Adobe Flash content in browsers
that block Flash. (XDMoD does not make use of Flash.)
- Fixed incorrectly-oriented subpanel titles in Internet Explorer 10.
- Fixed inconsistency between labels used for users and PIs on a job.
- Fixed automatic aggregation unit selection for charts not working
correctly for some installations of Open XDMoD.
- Fixed UGE shredder
- Fixed Slurm job array index parsing.
- SUPReMM
- Job Viewer
- Fixed "Show Raw Data" button in Metric Explorer not filtering
results correctly when using some combinations of drilldowns and
filters.
- Fixed single-point datasets not appearing in exported charts.
- Fixed Search History tree sorting nodes that should not be
sorted.
- Fixed "Show Raw Data" window in Metric Explorer staying active
after the chart underneath it changes (for example, when a new chart
is imported from Usage).
- Improved handling of raw data specified in kilobytes and
megabytes in Detailed Metrics pane.
- Fixed handling of Search History entries that have numeric
names.
Miscellaneous
- General
- Added documentation for using LDAP for federated authentication.
- Moved development of Open XDMoD into public GitHub repositories.
Version 6.0.0 (2016-09-21)
IMPORTANT NOTE: Highcharts has been updated from
v3.0.9 to v4.2.5. If you are a commercial user as defined by the
Highcharts license terms and your Highcharts license does not cover the
new version, you will need to acquire a new license.
Features
- Added demo user type.
- Added roadmap link to feature request dialog.
- Shredders
- Fixed calculation of missing end times from the start time and wall
time.
- LSF
- Now storing the node list, exit code and exit status for SUPReMM
support.
- Slurm
- Now allowing null eligible times.
- Failed parsing of datetimes are no longer a fatal error.
- SGE
- Now supporting older versions of SGE that contain only 43 fields
in their accounting logs.
- SUPReMM
- Job Viewer
- Added sort options to search history panel.
- Organized advanced search filters into categories.
- Modified timeseries charts to use the timezone of a job's
resource instead of the timezone used by the web browser.
- Modified analytics pane to always be present and explain why
missing data is missing.
- Modified byte units to use IEC prefixes instead of SI ones.
- Allowed some metrics to be displayed in multiple tabs.
- Added tooltips to advanced search filters.
- Added tooltips to detailed metrics.
- Added help sections to tabs that didn't have any previously.
- Added a help button to the at-a-glance analytics.
- Added a loading message to charts that are loading.
Bug Fixes
- General
- Fixed some charts not rendering if more than 1000 points were
plotted.
- Improved consistency of chart aesthetics across tabs.
- Fixed report generator not cleaning up old report files.
- Metric Explorer
- Fixed Add Filter list only including filter types applicable to the
realm of the first data series on a chart.
- Shredders
- PBS
- Fixed parsing of newer
exec_host
formats
(e.g. host/0-3
, host/0*8
).
xdmod-import
- Fixed importing of hierarchy and group-to-hierarchy files.
- SUPReMM
- Job Viewer
- Fixed a number of cases where editing a search did not work as
expected.
- Fixed various edge cases where timeseries chart drilldowns
did not work as expected.
- Fixed case where a top-level timeseries chart was exported
instead of the current, drilled-down chart.
- Fixed case where timeseries chart drilldowns performed on a
chart were not reflected in the navigation tree.
- Fixed case where selecting a low-level timeseries chart in the
navigation tree opened a top-level chart instead.
- Fixed basic search resource list loading immediately on page
load.
Under the Hood
- Reorganized code to enable easier sharing of development efforts across
XDMoD and Open XDMoD.
- Updated various third-party components used by Open XDMoD.
- Made numerous other internal improvements to prepare for future features
and ease development efforts.
Version 5.6.0 (2016-05-24)
New Features
- Federated Authentication
- Roadmap
- Added interactive roadmap for XDMoD development to About tab.
- Contact Us
- Added option to contact developers for technical support.
- Data Warehouse
- Improved speed of filter (dimension value) lists by precomputing
available values.
- Metric Explorer
- Shredders
- Added support for PBS Pro.
- Added additional input validation to the Slurm helper script.
- Upgrade Process
- Added warning to upgrade addon modules before migrating.
- Database
- Changed column types and added indexes to the logging table.
- Documentation
- User manual now available.
- Improved installation and configuration documentation.
- Application Kernels
- Added ability to select the 29th-31st of a month as a monthly report
delivery day.
- For months that don't have these days, reports scheduled for
those days will be delivered on the last day of the month.
- SUPReMM
- Configuration
- Switched to URL-based method for specifying Mongo databases.
- This adds support for Mongo databases that require
authentication.
- Improved setup process to be more user-friendly.
- The interactive setup script now generates the required
configuration files.
- Improved configuration file structure.
- Data Processing
- Added ability to transfer ingested/aggregated data between
databases.
- This allows SUPReMM data to be reprocessed in a secondary
database before deploying a new version of the SUPReMM ingestor
in the main XDMoD instance.
- Improved logging for ingestion and aggregation scripts.
- Job Viewer
- Added ability to edit searches.
- Improved layout of search window.
- Added ability to export timeseries plots as images or CSV
data.
Bug Fixes
- General
- Enabled toolbar scrolling when window is too small.
- Fixed trend line R2 values sometimes not being
calculated.
- Removed redundant warnings about role-based restrictions.
- Fixed the About tab not being able to be the default tab.
- Improved chart loading speed by removing most animations.
- Fixed incorrect trend line calculation and display for
discontinuous data series.
- Fixed security vulnerability in Report Generator's image
generator.
- Fixed extra slashes in URLs in password reset emails (including
account creation emails).
- Fixed email template paths.
- The About tab can now be set as the default module.
- Data Warehouse
- Fixed cases where current time period would not be aggregated.
- Now that this is resolved, code that can work by scanning any
aggregate table now uses the year tables, as those are smallest.
- Improved consistency of time period tables and removed unused ones.
- Metric Explorer
- Fixed handling of HTML characters in chart titles.
- Fixed issues with axis renames not persisting.
- Fixed Load Chart search not working.
- Modified chart options menu label to be a fixed length.
- This prevents the button from being placed in the toolbar's
overflow area when it shouldn't be.
- Fixed changes to y-axes not being undoable.
- Modified "Show Trend Line" button in chart options to be disabled
when trend lines aren't available.
- Fixed case where duration selectors in main toolbar and on x-axis
popup could desynchronize.
- Shredders
- Fixed handling of "NONE" in SGE's pe_taskid field.
- Now allowing all values that were previously resulting in fatal
errors in Slurm fields that represent lengths of time.
- Upgrade Process
- Fixed broken error for when portal_settings.ini is not writable.
- Usage
- Fixed bad default settings for dimensions that default to pie
charts.
- Fixed long initial load time by starting on a single-chart page.
- Application Kernels
- Fixed Reports tab not appearing for some authorized users.
- Reduced user authorization requirements for showing certain data.
- Removed "." from tab labels.
- Removed non-functional x-y swap from tabs.
- SUPReMM
- Job Viewer
- Added error dialog box for if Quick Job Lookup's resource list
fails to load instead of silently failing.
- Fixed existing searches breaking after performing a re-ingest of
SUPReMM data.
- Fixed charts sometimes not resizing properly.
- Fixed memory leak in search history right-click menu.
Known Issues
- Some text boxes will not allow ampersands to be typed into them in
Firefox.
- To work around this, ampersands can be copied and pasted into these
boxes.
Version 5.5.0 (2015-12-18)
New Features
- SUPReMM support
- Metric Explorer Changes
- Improved management of charts
- Charts are no longer automatically saved after each
operation. A manual Save button has been added that will
indicate whether or not a chart needs to be saved.
- When displaying pie charts, only a single dataset is
allowed.
- Added ability to rename saved charts.
- Added undo capability.
- Improved UI for data filtering
- Added Quick Filters for quickly restricting data. This
replaces the Role selector.
- Improved data access controls
- Non-public metrics and dimensions are now restricted
according to users' roles.
- Improved filter lists to only include values for which
the user can view the underlying data.
- Added the ability to include a remainder data series.
- Migrated more server calls to the new REST stack
- Simplified the chart/data retrieval infrastructure used by the
Metric Explorer and Usage tabs
- Improved performance of certain queries
- Updated PBS parsing to support time fields that are formatted in
seconds and not HH:MM:SS
Bug Fixes
- Fixed report generator document formatting
- Fixed inconsistent data series ordering between timeseries charts
when sorted by value in Usage and Metric Explorer tabs
- Fixed Metric Explorer sometimes loading the same chart twice
- Fixed the "Hide Tooltip" option not always working as expected in
Usage tab
- Fixed filter lists occasionally getting stuck under a loading
mask
- Fixed predefined date durations not being saved correctly in the
Metric Explorer
- Removed auto-loading of the last open chart in Metric Explorer, a
new chart is now opened when visiting the Metric Explorer
- Improved stability, consistency, and performance of numerous other
components
- Prevent warning messages produced by LSF jobs that don't have a
host list
Version 5.0.0 (2015-08-19)
New Features
- Utilization metric may now be normalized to a specified percent
allocated
- Removed javac dependency
- Add host filter shredder feature
- Improved stability of Usage tab
- Improved error handling for internal dashboard errors
- Added meaningful HTTP status codes to failed login responses
- Resolved frequent warnings in Apache logs about implicitly
converting arrays to strings
- PBS memory value parsing errors are no longer fatal
- Removed Apache requirement for listening on 127.0.0.1
- Added support for Univa Grid Engine 8.2+
- Shredders now skip duplicate data
- Added support for MySQL 5.6+
- Added support for PhantomJS 2.0+
- Added helper script for updating resource specs
- Improved documentation
Bug Fixes
- Adjust LSF unique key constraint
- Added PBS unique key constraint to prevent duplicate job data
Version 4.5.2 (2015-02-17)
New Features
- Remove excessive "wall time" error messages
- Add more debug logging
- SGE memory value parsing errors are no longer fatal
Bug Fixes
- Fix SGE parsing regression
Version 4.5.1 (2015-02-05)
New Features
- Support for Torque's new "exec_host" syntax
- Updated SGE parsing to support Univa's new "job_class" field and
requests for "INFINITY" memory
Bug Fixes
- Backslash characters in LSF log files now parsed correctly
- Report generator now handles double quotes in portal_settings.ini
properly
- Fixed optional Node Utilization statistic
Version 4.5.0 (2014-12-04)
New Features
- Major re-write of the Metric Explorer user interface including a
metric catalog and context-sensitive menus
- Report generator performance enhancements
- General user interface updates
- Improved image export dialog
- Added metric description pane in Metric Explorer
- Improved session management
- Added additional error checking and reporting to shredders
- Added LSF support
- Improved utilization metric to support resources that don't have a
constant cpu count over their lifetime
- Added suport for alternate PI data sources
- Improved logging
- Added weighted job size statistic
- Improved IE compatibility
- Added xdmod-admin script
- Added automatic update checker
- Added support for slurm job arrays
- Improved documentation
Bug Fixes
- PBS and SGE job arrays were not ingested correctly
- sacct configuration option was not recognized
- Arrow keys did not work in axis range boxes
- Parentheses not available in Metric Explorer chart titles
- Proper handling of apostrophes in search dialogs
- On some charts, data points on the last day of the specified date
range were not displayed correctly
Version 3.5.0 (2013-11-18)