Link liveliness assessment tool (SoilWise-HE)

Metadata is commonly linked to the resource it describes via a web link. But records may also include links to related materials. Not all of these URIs are persistent, so over time they can degrade. The link liveliness assessment component runs over the available links in metadata records and checks their status. The component currently build against the OGCAPI - Records protocol, a standardised protocol on metadata exchange in the spatial domain. In future the component may be extended to interact with other catalogue standards, such as oai-pmh. Test results are stored in a database or can be exported to a file.

The link liveliness privides the following functions:

1. OGC API Catalogue Integration
   • Designed to work specifically with OGC API - Records System
   • Extracts and evaluates URLs from catalogue items
2. Link Validation
   • Evaluates the validity of links to external sources and within the repository
3. Support for OGC service links
   • Identifies and properly handles OGC service links (WMS, WFS, CSW, WCS etc.) and captures and verified connection details of the remote service
4. Health Status Tracking
   • Provides up-to-date status history for every assessed link
   • Maintains a history of link health over time
5. Flexible Evaluation
   • Supports single resource evaluation on demand
   • Performs periodic tests to provide availability history
6. Broken link management
   • Identifies and categorizes broken links based on their status code ( 401 Unauthorized, 404 Not Found, 500 Server Error)
   • Flags deprecated links after consecutive failed tests and excludes them from future check
7. Timeout management
   • Identifies resources exceeding specified timeout thresholds

A javascript widget is further used to display the link status directly in a website.

Technology

• Python Used for the linkchecker integration, API development, and database interactions
• OWSlib for interaction with remote services
• PostgreSQL Primary database for storing and managing link information
• FastAPI Employed to create and expose REST API endpoints. Utilizes FastAPI's efficiency and auto-generated Swagger documentation
• Docker Used for containerizing the application, ensuring consistent deployment across environments
• CI/CD Automated pipeline for continuous integration and deployment, with scheduled weekly runs for link liveliness assessment