hyperstream package¶
Subpackages¶
Submodules¶
hyperstream.channel_manager module¶
Channel manager module. Defines the ChannelManager - a container for channels, that can load in plugins
-
class
hyperstream.channel_manager.
ChannelManager
(plugins, **kwargs)[source]¶ Bases:
dict
,hyperstream.utils.utils.Printable
Container for channels.
-
database_channels
¶ The database channels as a list
-
get_channel
(channel_id)[source]¶ Get the channel by id
Parameters: channel_id – The channel id Returns: The channel object
-
get_tool
(name, parameters)[source]¶ Gets the tool object from the tool channel(s), and instantiates it using the tool parameters
Parameters: - name – The name or stream id for the tool in the tool channel
- parameters – The parameters for the tool
Returns: The instantiated tool object
-
get_tool_class
(tool)[source]¶ Gets the actual class which cna then be instantiated with its parameters
Parameters: tool (str | unicode | StreamId) – The tool name or id Returns: The tool class :rtype Tool | MultiOutputTool
-
memory_channels
¶ The memory channels as a list
-
tool_channels
¶ The tool channels as a list
-
hyperstream.client module¶
The main hyperstream client connection that is used for storing runtime information. Note that this is also used by the default database channel, although other database channels (connecting to different database types) can also be used.
-
class
hyperstream.client.
Client
(server_config, auto_connect=True)[source]¶ Bases:
object
The main mongo client
-
client
= None¶
-
connect
(server_config)[source]¶ Connect using the configuration given
Parameters: server_config – The server configuration
-
db
= None¶
-
get_config_value
(key, default=None)[source]¶ Get a specific value from the configuration
Parameters: - key – The of the item
- default – A default value if not found
Returns: The found value or the default
-
session
= None¶
-
hyperstream.config module¶
HyperStream configuration module.
-
class
hyperstream.config.
HyperStreamConfig
[source]¶ Bases:
hyperstream.utils.utils.Printable
Wrapper around the hyperstream configuration files (hyperstream_config.json and meta_data.json)
hyperstream.hyperstream module¶
Main HyperStream class
-
class
hyperstream.hyperstream.
HyperStream
[source]¶ Bases:
object
HyperStream class: can be instantiated simply with hyperstream = HyperStream() for default operation
-
create_workflow
(workflow_id, name, owner, description)[source]¶ Create a new workflow. Simple wrapper for creating a workflow and adding it to the workflow manager.
Parameters: - workflow_id – The workflow id
- name – The workflow name
- owner – The owner/creator of the workflow
- description – A human readable description
Returns: The workflow
-
hyperstream.online_engine module¶
Online Engine module. This will be used in the online execution mode.
hyperstream.plugin_manager module¶
Plugin manager module for additional user added channels and tools.
-
class
hyperstream.plugin_manager.
Plugin
[source]¶ Bases:
hyperstream.plugin_manager.PluginBase
,hyperstream.utils.utils.Printable
Plugin class - simple wrapper over namedtuple
hyperstream.time_interval module¶
Module for dealing with time intervals containing TimeInterval, TimeIntervals, and RelativeTimeInterval
-
class
hyperstream.time_interval.
RelativeTimeInterval
(start, end)[source]¶ Bases:
hyperstream.time_interval.TimeInterval
Relative time interval object. Thin wrapper around a (start, end) tuple of timedelta objects that provides some validation
-
class
hyperstream.time_interval.
TimeInterval
(start, end)[source]¶ Bases:
object
Time interval object. Thin wrapper around a (start, end) tuple of datetime objects that provides some validation
-
end
¶
-
start
¶
-
width
¶
-
-
class
hyperstream.time_interval.
TimeIntervals
(intervals=None)[source]¶ Bases:
hyperstream.utils.utils.Printable
Container class for time intervals, that manages splitting and joining Example object: (t1,t2] U (t3,t4] U ...
-
end
¶
-
is_empty
¶
-
span
¶
-
start
¶
-
-
hyperstream.time_interval.
parse_time_tuple
(start, end)[source]¶ - Parse a time tuple. These can be:
- relative in seconds, e.g. (-4, 0) relative in timedelta, e.g. (timedelta(seconds=-4), timedelta(0)) absolute in date/datetime, e.g. (datetime(2016, 4, 28, 20, 0, 0, 0, UTC), datetime(2016, 4, 28, 21, 0, 0, 0, UTC)) absolute in iso strings, e.g. (“2016-04-28T20:00:00.000Z”, “2016-04-28T20:01:00.000Z”)
Mixtures of relative and absolute are not allowed :param start: Start time :param end: End time :type start: int | timedelta | datetime | str :type end: int | timedelta | datetime | str :return: TimeInterval or RelativeTimeInterval object
hyperstream.tool module¶
Tool module. Defines Tool and MultiOutputTool base classes.
-
class
hyperstream.tool.
AggregateTool
(aggregation_meta_data, **kwargs)[source]¶ Bases:
hyperstream.tool.Tool
This type of tool aggregates over a given plate. For example, if the input is all the streams in a node on plate A.B, and the aggregation is over plate B, the results will live on plate A alone. This can also be thought of as marginalising one dimension of a tensor over the plates
-
class
hyperstream.tool.
MultiOutputTool
(**kwargs)[source]¶ Bases:
hyperstream.utils.utils.Printable
,hyperstream.utils.utils.Hashable
Special type of tool that outputs multiple streams on a new plate rather than a single stream. There are in this case multiple sinks rather than a single sink, and a single source rather than multiple sources. Note that no alignment stream is required here. Also note that we don’t subclass Tool due to different calling signatures
-
execute
(source, sinks, interval, input_plate_value, output_plate)[source]¶ Execute the tool over the given time interval.
Parameters: - source (Stream) – The source stream
- sinks (list[Stream] | tuple[Stream]) – The sink streams
- interval (TimeInterval) – The time interval
- input_plate_value (tuple[tuple[str, str]] | None) – The value of the plate where data comes from (can be None)
- output_plate (Plate) – The plate where data is put onto
Returns: None
-
name
¶
-
-
class
hyperstream.tool.
Tool
(**kwargs)[source]¶ Bases:
hyperstream.utils.utils.Printable
,hyperstream.utils.utils.Hashable
Base class for tools. Tools are the unit of computation, operating on input streams to produce an output stream
-
execute
(sources, sink, alignment_stream, interval)[source]¶ Execute the tool over the given time interval. If an alignment stream is given, the output instances will be aligned to this stream
Parameters: - sources (list[Stream] | tuple[Stream] | None) – The source streams (possibly None)
- sink (Stream) – The sink stream
- alignment_stream (Stream | None) – The alignment stream
- interval (TimeInterval) – The time interval
Returns: None
-
name
¶
-