Maritime Linked Data for Situational Awareness Heterogeneous Sensor Networks

—Maritime heterogeneous sensor networks (HSN), which are widely used in support of marine monitoring, maritime security and safety applications, combine diversiﬁed sensing systems and platforms observing the state of environmental features of interest or the status of anthropogenic objects to achieve awareness on the situation. To model the information characteristics of HSN observations for maritime situational awareness (MSA), we developed the MSA-HSN ontology, an integrated semantic model for maritime linked data, speciﬁcally observations suiting the requirements of maritime information fusion systems. MSA-HSN integrates established ontologies and models for sensors, observations, measurements, quantities, and occurrents, tailored to MSA applications and requirements. To support the interoperability with existing maritime data models, MSA-HSN is aligned with the relevant aspects of the EUCISE/e-CISE (Common Information Sharing Environment) data models. As a validation use case, in this paper the MSA-HSN ontology is applied to model the information elements of the maritime surveillance system developed within the Interactive Extreme-Scale Analytics and Forecasting (INFORE) project, where different situational views offered by a variegate suite of sensors and platforms are fused using big data analytics to achieve maritime situational awareness for maritime security. The paper describes the design of the MSA-HSN ontology, illustrating its application through examples taken from the INFORE use case and from relevant datasets developed by recent European projects involving


I. INTRODUCTION
Maritime Situational Awareness (MSA) is defined as the human mental model of a situation concerning the maritime environment. MSA is constructed on the basis of the information available on the situation, often acquired though heterogeneous sensor networks (HSNs) that may combine in-situ and remote sensing devices, collaborative and noncooperating systems, fixed sources as well as autonomous (uncrewed) multi-domain vehicles (AxVs/UxVs). The variety and the spatial distribution of sensing devices produce multimodal signals and a multiplicity of observations, potentially augmenting the completeness and the veracity of the resulting situational picture and supporting different MSA applications.
The authors acknowledge the support of the Horizon 2020 project "Interactive Extreme-Scale Analytics and Forecasting" (INFORE) GA No. 825070. (to appear in OCEANS21 San Diego -Porto) Maritime security systems, for example, fuse multiple sensor feeds, which are cleaned, combined, correlated and analysed to facilitate the detection of threatening situations.
Maritime information fusion is primarily obtained on the basis of the spatial, temporal, and statistical characteristics of the signals. In addition, the correlation of the sensor streams can leverage semantic associations, primarily based on common data features like object identifiers and shared attributes or relying on advanced information similarity functions that leverage the semantic distance among information instances. Overall, the information semantics can be exploited to augment the informativeness of the fused stream to facilitate the interpretation, contextualisation and consequently the understanding of the sensor signals and the alerts produced by the information fusion system.
To model the information requirements of maritime information systems and MSA applications, we developed a semantic model for maritime linked data, namely the maritime situational awareness -heterogeneous sensor network (MSA-HSN) ontology [1]. MSA-HSN is defined in OWL (Web Ontology Language) 1 and can be used to semantically annotate and enrich the sensor information flowing into and produced by a typical maritime information fusion system. MSA-HSN maritime linked data offer an effective support towards MSA data interoperability and for semantically exploiting MSA information to enhance data analysis. The MSA-HSN ontology is directly applicable in a variety of scenarios addressing maritime surveillance for safety and security, like the detection of maritime threats and illegal activities, maritime traffic monitoring, critical infrastructures protection, but the proposed model suites all maritime scenarios requiring the fusion of information produced by sources of various reliability and quality.
MSA-HSN extends established sensor and observations [2], units and measurements [3] and event [4] semantic patterns to model MSA information elements. Specific characteristics of maritime information are mutuated from a reference maritime data interoperability model developed for the European Common Information Sharing Environment (CISE) [5]. MSA-HSN seamlessly models sensor signals, observations and derived information like detection contacts or complex analytics, detected and forecasted maritime events or anomalies and maritime activities, addressing information aspects like provenance and information quality which are crucial for MSA and need to be evaluated by the information fusion system.
The motivating use case of the MSA-HSN ontology is the MSA workflow proposed for the Horizon 2020 project "Interactive Extreme-Scale Analytics and Forecasting" (IN-FORE) 2 [6], where the situational views offered by an heterogeneous suite of sensors and platforms are coupled with big data analytics to achieve situational awareness for maritime security. The INFORE MSA platform is developed to fuse global, regional and local sensor information produced by the Automatic Information System (AIS), radars, multispectral satellite imaging sensors, passive acoustic sensors and thermal cameras, and feed distributed analytics to detect and forecast events of interests for different surveillance tasks.
The paper is organised as follows. Section II presents related work, in particular the semantic models exteded by MSA-HSN. Section III describes and motivates the conceptual design of the MSA-HSN ontology and formalises the information components of a typical information fusion systems. The modelling of the most relevant elements of the ontology design, i.e., maritime sensors and observations, observation quantity and values, information quality and maritime events, will be illustrated using data from the INFORE maritime use case, in particular the maritime robotic network presented in [6], and open maritime data from real heterogeneous sensors and events from the same project [7], [8]. Finally, Section IV concludes the papers and outlines the future developments of MSA-HSN.

II. RELATED WORK
The MSA-HSN ontology extends several existing ontologies and data models that have been recently proposed for the modelling of sensors and observations [2], values and measures, units and quantities [3], and data models defined to facilitate the interoperability of institutional maritime safety and security systems [5].
The Semantic Sensor Network/Sensor Observation Sampling Actuator (SSN/SOSA) ontology 3 is a lightweight semantic model for sensors, observations, samples and actuators that revises some of the design choices proposed for the Semantic Sensor Network ontology. MSA-HSN adopts the SSN/SOSA sensor pattern and extends it to model MSA information sources. As for SOSA, in MSA-HSN all information sources relevant for the modelled use case are represented as sensors, encompassing both physical sensors and virtual sensors like software or human beings. Differently from SSN/SOSA, maritime sensors and observations in MSA-HSN are modelled as subclasses of features of interests. This extension enables to model the situation in which they are observed by some sensor in the network, introducing a pattern for modelling information quality [9], which is needed by the information fusion system to appropriately combine the sensor signals.
Furthermore, MSA observations are complemented with additional properties to model data observation provenance and specific MSA information aspects (e.g., the qualitative nature and the qualitative temporal nature of the information), mostly mutuated from the EUCISE model. To facilitate the alignment with the EUCISE model, the MSA-HSN class that models the feature of interest inherits from the EUCISE abstract model for physical entities, including vessels. In addition to SSN/SOSA, MSA-HSN endorses the modelling of observation streams and stream analytics proposed as extension of the SSN/SOSA ontology to support the annotation of Internet of Things (IoT) data streams 4 [10].
Differently from SOSA, which relies on the QUDT Catalog of Quantities, Units, Dimensions and Data Types Ontologies 5 , the MSA-HSN ontology aligns and extends the 2.0 version of the Ontology of units of measures (OM) 6 , originally presented in [3] to represent sensor values, quantity kinds and units. OM is compatible with QUDT as both are based on [11], but is evaluated being more complete [12]. In particular, differently from other models for quantities and values, OM models scales, which are widely adopted in maritime security to model information uncertainty and quality.
The OM's pattern for quantities and measures has been extended in MSA-HSN to model multi-granular spatio-temporal information and sensor and processing result values of any kind, going beyond quantitative measures, to model all the pieces of information relevant for MSA, which include qualitative, or categorical, values. This extension enables a straightforward modelling of the descriptive enumeration values and scales defined in the EUCISE model. In addition, MSA-HSN allows to include uncertainty annotations as advised by the guide to the expression of uncertainty in measurement defined in [13].
The modelling of maritime related concepts relies on the data model defined for the Common Information Sharing Environment [14], proposed on the basis of the EUCISE data model, which has been recently converted in an OWL-2 ontology EUCISE-OWL [5] to facilitate its adoption in a border security European project 7 . In particular, the modelling of maritime events proposed in [5] is leveraged and extended in MSA-HSN. This event model relies on the Simple Event Model Ontology [4], which is widely adopted in the literature (e.g., for the representation of marine data and events [15]).
In the MSA-HSN ontology, this event pattern has been extended to include the representation of the event context, constructed on the basis of the profiles of the entities observed by sensors and specifically modelled for supporting situation awareness and threat assessment in surveillance applications [16].
MSA-HSN adopts taxonomies for maritime anomalies and situational indicators defined in the literature, adapted to model the facts, activities and events of interest for the INFORE maritime use case. The maritime events of interest for INFORE include simple and complex events. Simple events correspond to maritime situational facts as defined in [17] and simple maritime situational indicators as defined in [18]. Complex events represent maritime activities, anomalies and complex situational indicators, according to [17], [18], [19], [20] III. THE MSA-HSN ONTOLOGY The MSA-HSN ontology is designed to represent and enrich maritime fusion system information to facilitate maritime situation awareness. Typical scenarios are offered by the maritime security use case addressed by the INFORE MSA platform: fighting illegal fishing and protection of critical infrastructures. However, the proposed high-level model is also applicable to maritime safety and monitoring applications with minimum modifications like the creation of subclasses and individuals to model the specific entities.
The MSA-HSN ontology may be decoupled in three parts, corresponding to three main information spaces in situational awareness: sensors and observations, observation values, and events. The modelling of these information entities is mostly addressed separately in the literature, and MSA-HSN extends and adapts prominent information models which have been specifically proposed for each information aspect. MSA-HSN integrates them with a standard data model for maritime information and develops a comprehensive model for MSA applications.
The ontology is rooted in the representation of the MSA sensors and observations and the associated entities, in particular the features of interest for MSA and their properties, whose status is discovered through sensing devices. The classes modelling MSA sensors, observations, observable features of interest and their properties, and the procedures to follow to produce the observations are described in Section III-A.
Linked to observations are observation values. In MSA-HSN, numerical quantities and measures which are well codified in the literature are extended and integrated to include other values types frequently used in MSA and maritime security. MSA-HSN specifically refers to the categorical values defined in the CISE/EUCISE data model [5], [14] but the proposed model can be extended to support other such values. The resulting quantities & values model is further extended to support multi-granularity and measurement uncertainty and described in Section III-B.
The third important information space supported by the MSA-HSN ontology addresses the modelling of occurrents, or events, which are crucial information entities in MSA, treated extensively by the domain literature [17], [21], [20]. Maritime events are seamlessly integrated in the observation model, and extended to better support the specific use case scenarios. The modelling of maritime events is described in Section III-C.  [2], CISE/EUCISE [5], [14] and IoT-Stream [10] that MSA-HSN imports and extends.

A. Sensor and observations
The MSA-HSN information chain starts from the concept MSASensor that extends sosa:Sensor and captures all information sources relevant for MSA. Figure 2  Akka-based distributed system to efficiently fuse different kinds of maritime data."@en.

1) Types of sensors:
An individual of type MSASensor may be a physical sensor, either in-situ or remote, which observes a feature of interest and its properties, like global positioning system (GPS) devices estimating the position on Earth of vehicles and Conductivity-Temperature-Depth (CTD) devices estimating the conductivity (and indirectly, the salinity), temperature and pressure of the water column. MSA physical sensors include GPS, AIS, radar, synthetic aperture radar (SAR), remote sensing devices, electro-optical and infrared (EO/IR) cameras, acoustic sensors.
In addition, MSASensor individuals may be virtual, like software, e.g., an ocean condition model forecasting seawater temperature in the next hours, or the INFORE MSA big data analytics. As a consequence of the proposed extension of the concept of sensor, the second order information produced by data analytics is modelled as a particular type of observation. This design choice, which models seamlessly sensor data and analytics results, allows recursion and enables a straightforward representation of data processing workflows.
Other important MSASensor are soft sensor sources like human being (for example, the coast guard operators directly observing the situation in a port, and reporting on the vessels behaviour), institutions publishing official bulletins and reports, newspapers and social media. All are examples of sources of information that, connected in a HSN, may contribute to MSA and, as MSASensor individuals, are seamlessly modelled in the MSA-HSN ontology.
The use of a taxonomy for sensors might be useful to support the guided population of the ontology, as well as    to enable reasoning on the characteristics of the different sensor types (e.g., to infer information on aspects that are not explicitly modelled). However, it is not strictly required and in most of the cases the class MSASensor is sufficient to effectively represent all the sensors for an application.
2) Observations, procedures and observable properties: Sensors generate observations on the status of the observable properties of some feature of interests. Information resulting from the analysis of sensor data includes analytics results like target detection and classifications, maritime events, anomalies and maritime activities detected from data. All these concepts are represented in the ontology as subclasses of MSAObservation ( Figure 3 shows the INFORE subclasses).
Observations may be gathered in streams, using the IoT-Stream concept IoT-stream:IoTStream. The stream can be analysed using stream analytics (individuals of type ioT-stream:Analytics). Through the MSA-HSN ontol-    . ogy, observations are combined in an integrated maritime knowledge graph which gives a comprehensive representation of the maritime situation.
The act of observation is carried out according to codified procedures, i.e., a physical sensor plan or an algorithm. In line with SOSA, MSA-HSN procedures can refer to shared vocabularies, e.g., defined by the Earth Science communities, and modelled as instances of classes sosa:Procedure of its subclasses (a procedure taxonomy is introduced here to facilitate the instance retrieval). Figure 4 shows some of the algorithmic procedures used in INFORE.
Similarly to procedures, observable properties are defined as owl:NamedIndividual reusing shared domain vocabularies. For example, position, speedOverGround, heading, and bearing are observable properties for a broad class of vehicles, including maritime ones (e.g., surface vessels, rubber boats, autonomous surface and underwater robots). As such, properties are usually associated to multiple features of interest and observations in one-to-many relationships. Like procedures, properties may be organised in taxonomies, facilitating the alignment with existing vocabularies. As shown in Figure 5, in MSA-HSN properties are organised into sensor and vehicle properties, and quality dimensions.
To give an example, the code snippet III-A. 2 shows the specification of an observation generated from the dataset [8] that reveals the position of vessel 3) Quality of Information and Sources: An important feature of the MSA-HSN ontology is the support it offers to model information quality, which integrates the base pattern for sensor and observation, while preserving the sensor and observation interoperability with other models derived from SSN/SOSA. The MSA-HSN quality support is composite and layered, with three complementary representations for quality that can be adopted.
To qualitatively describe the nature and the temporal validity of an observations, which offer a preliminary indication of its quality, MSA-HSN introduces two properties, eucise: hasQualitativeNature and hasQualitativeTemporalValidity, expressed according to the CISE/EUCISE vocabulary. These are included in the observation defined in Example III-A. 3, whose declared quality nature is <observed> (in alternative, <estimated>, <declared>, <simulated>, and <validated> quality natures are modelled). The qualitative temporal validity of the observation is <nowcast> (alternatively, the observation could be a <forecast>, or a <reanalysis>).
The second component of the information quality model is given by quality observations, which are produced by MSA-HSN sensors to specifically reveal the quality of sensors and MSA observations. Differently from SSN/SOSA, the fact that sensor are designed to observe sensors and information is explicitly modelled in the MSA-HSN ontology: classes MSAObservation and MSASensor are defined as subclasses of the class sosa:FeatureOfInterest, therefore they are observable by sensing devices (see Figure 1). Quality assessments are represented by individuals of the class QualityObservation, subclass of MSAObservation. Quality observations are specified according to quality-related properties. These are defined like the other properties as a vocabulary of named entities representing quality dimensions, for example those defined for data, information and information sources [22], [23] that inspired MSA-HSN quality properties. The snippet III-A. 3 shows the <reliability> assessment of the observation in Example III-A. 2. As illustrated in this example, sensors can also auto-observe their quality or the quality of the information they produce. <AIS/a10fab-f282-4455-a3c6-2020-02-29T22-00-11Z> hasQualityObservation <QAIS/2020-02-29T22-00-11Z>.
In addition, note in Figure 1 that individuals of MSAobservation and MSASensor can be directly associated to their quality observations through the relationships sosa:madeBySensor and sosa:isFeatureOfInterestBy, facilitating the quality evaluation of the fusion system.
The last quality specification enabled by MSA-HSN is the direct modelling of uncertainty in the observation value specification, which is illustrated in Section III-B by Example III-B. 3 where an estimated position is returned with the uncertainty measure produced by the sensor. 4) Provenance of Information: MSA-HSN models also information provenance. Observations may be annotated with the originating data and sensor (relationships isDerivedFrom and sosa:madeBySensor in Figure 1). The provenance of information, together with quality modelling, are needed by the information fusion system to assess the reliability and the credibility of information and to correctly evaluate and combine the sensor signals. In addition, the MSA-HSN modelling of provenance enables the representation of observation processing workflows. For example, analytics results may be associated with raw data from physical sensors, or other preprocessed data.

B. Observation values
Given a set of features of interest in a scenarios, observations described in the previous section are semantically annotated measures and estimations of the status of their properties, collected through sensing devices to achieve MSA. In MSA, observations are variegate and their value depends on the nature of the properties. To improve the expressivity of the MSA-HSN model, the base model for representing quantities and values [3] , is expanded to support sensor and processing results of any kind, including categorical values and multi-granular spatial and temporal values. We also adopt a richer base model for quantities and measures. The MSA-HSN quantity-value model is illustrated in Figure 6.
MSA-HSN observation values are represented by individuals of the class Value, which extends sosa:Result and om:Measure. As illustrated in Figure 7, the class Value is extended by four subclasses, representing the four value types supported by the ontology.
The qualitative values represented in the ontology refers in particular to the enumerations defined in the CISE/EUCISE data model, like the quality and qualitative temporal nature values defined in Example III-A. 3. Other categorical values defined in the ontology are those describing the vessel status, type, and, as we will see in the next Section, the values describing the participation roles of objects and agents in events, and the specific events and threats. To extend the ontology with other such values is sufficient to create the corresponding named individuals. Both quantitative and qualitative values can be characterised with respect to the granularity used for expressing them.
1) Quantity kinds and dimensions: In MSA-HSN, an observation value is semantically annotated by its quantity kind, following the original value-quantity pattern defined in the Ontology of units of measures (OM) 2.0 [3]. All generic quantities and qualities that characterise the observation values in the ontology are described by named individuals of the class QuantityKind, defined as the union of the individuals in the  The individuals in class om:Quantity are defined in the base mode OM and represent the base and derived quantities like time, mass, length, speed, acceleration defined in the systems of units of reference. The corresponding quantity dimension is given by a dimension vector as specified in [11] and represented by individuals of the class om:Dimension. Quantitative values are also associated to unit as scales as defined in the unit system of reference, by importing from OM the concepts om:Unit and om:Scale.
The same modelling pattern applies also to qualities, which generically describe the type of qualitative values. In absence of a shared vocabulary for MSA qualities,MSA-HSN qualities and the corresponding dimensions are customised to describe the broader information dimensions of MSA, mainly referring to CISE/EUCISE enumerative values, but also using domaindriven terms which are of common use (e.g., AIS-related and platform-related terminology) . Qualities in the ontology describe for example the status of the features, the role of the entities involved in a situation, the qualitative nature of the information, etc. The value dimension of a quality is formalised as a named individual of the class QualityValueDimension that links to all the possible values defined in the ontology for that quality. ValueDimension is the overall class that encompasses all value domains (i.e., ValueDomain ≡ om:Domain  2) ValueDomain: ValueDomain is a new concept that is introduced in MSA-HSN to connect a property to the subset of values in the value dimension that are applicable to that property. For example, in III-B. 2, the named entity <vesselStatusDomain> represents the domain of the property <vesselStatus>, defined to semantically annotate the status of vessels in the MSA-HSN as individual of the class <VesselProperty> subclass of <VehicleProperty>. <vesselStatusDomain> is associated to the value dimension <status> defined above, and associated to the subset of status values that pertain to vessels through the relations hasDomainValue. 3) Spatio-temporal entities: The MSA-HSN ontology offers a complete support for the representation of spatiotemporal information at multiple granularities. The MSA-HSN support for temporal values includes the representation of temporal entities like intervals and time instants offered by the Time OWL ontology 11 , of temporal elements [24], i.e., set of disjoint instants and intervals, and of temporal series, ordered sequence of pairs (temporal entity, value), useful to defined series of observations. The MSA-HSN support for temporal values includes the representation of spatial entities, including geographical points and regions (represented by exterior and internal boundaries expressed as ordered sequence of points), of spatial elements, i.e., set of disjoint spatial entities; of spatial sets, set of spatially disjoint pairs (spatial entity, value), and of spatial series, which can be used to represent spatio-temporal series of observations. Spatial and temporal elements are introduced in MSA-HSN to enable a flexible representation of the temporal and spatial validity of observations. Temporal validity complements the execution time of an observation, expressing the period of time during which the observation must be considered by the fusion engine. Beyond that time, the observation must be discarded by the fusion system or penalised in the evaluation. The temporal validity complements temporal quality dimensions like timeliness and relevance. Similarly, the spatial validity expresses where the observation is of interest for the fusion engine. 11 Time Ontology in OWL www.w3.org/TR/owl-time/ 4) Value Uncertainty: As introduced in the previous section, observation values can also be directly annotated with respect to their uncertainty. MSA-HSN uses a simple uncertainty representation as propopsed by the guide to the expression of uncertainty in measurement [13], directly connecting the values with an uncertainty measure, complemented by information on the measurement (e.g., the type of evaluation applied to assess the confidence on the value, like the uncertainty function applied, as illustrated in Figure 6). The annotation can be specified according to existing vocabularies for quality measures and uncertainty (e.g., the quality measures defined for geographical data in ISO 19157:2013(E)).
In Example III-B. 3, the INFORE target detection sensor <iCADME> estimates the position of an unknown target by processing the signals of acoustic vector sensors as described in [6]. The estimated position is returned with an uncertainty measure produced by the sensor, directly modelled in the observation value.

C. Maritime events
The Figure 8 illustrates how the MSA-HSN ontology models detected, estimated or forecasted maritime events. In the ontology, maritime events are a particular type of observations (i.e., the class Maritime Event is subclass of MSAObservations), generated by software sensors like the INFORE Event Detector. The class Maritime Event also extends eucise:Event, aligning with the CISE/EUCISE model and the event pattern it adopts. In MSA-HSN maritime events group simple events, complex events and maritime activities, supporting also the representation of EUCISE/CISE events like anomalies, incidents, and actions.