hyperstream package

Submodules

hyperstream.channel_manager module

Channel manager module. Defines the ChannelManager - a container for channels, that can load in plugins

class hyperstream.channel_manager.ChannelManager(plugins, **kwargs)[source]

Bases: dict, hyperstream.utils.utils.Printable

Container for channels.

database_channels

The database channels as a list

get_channel(channel_id)[source]

Get the channel by id

Parameters:channel_id – The channel id
Returns:The channel object
get_tool(name, parameters)[source]

Gets the tool object from the tool channel(s), and instantiates it using the tool parameters

Parameters:
  • name – The name or stream id for the tool in the tool channel
  • parameters – The parameters for the tool
Returns:

The instantiated tool object

get_tool_class(tool)[source]

Gets the actual class which cna then be instantiated with its parameters

Parameters:tool (str | unicode | StreamId) – The tool name or id
Returns:The tool class

:rtype Tool | MultiOutputTool

memory_channels

The memory channels as a list

tool_channels

The tool channels as a list

update_channels()[source]

Pulls out all of the stream definitions from the database, and populates the channels with stream references

hyperstream.client module

The main hyperstream client connection that is used for storing runtime information. Note that this is also used by the default database channel, although other database channels (connecting to different database types) can also be used.

class hyperstream.client.Client(server_config, auto_connect=True)[source]

Bases: object

The main mongo client

client = None
connect(server_config)[source]

Connect using the configuration given

Parameters:server_config – The server configuration
db = None
get_config_value(key, default=None)[source]

Get a specific value from the configuration

Parameters:
  • key – The of the item
  • default – A default value if not found
Returns:

The found value or the default

session = None

hyperstream.config module

HyperStream configuration module.

class hyperstream.config.HyperStreamConfig[source]

Bases: hyperstream.utils.utils.Printable

Wrapper around the hyperstream configuration files (hyperstream_config.json and meta_data.json)

hyperstream.hyperstream module

Main HyperStream class

class hyperstream.hyperstream.HyperStream[source]

Bases: object

HyperStream class: can be instantiated simply with hyperstream = HyperStream() for default operation

create_workflow(workflow_id, name, owner, description)[source]

Create a new workflow. Simple wrapper for creating a workflow and adding it to the workflow manager.

Parameters:
  • workflow_id – The workflow id
  • name – The workflow name
  • owner – The owner/creator of the workflow
  • description – A human readable description
Returns:

The workflow

hyperstream.online_engine module

Online Engine module. This will be used in the online execution mode.

class hyperstream.online_engine.OnlineEngine(hyperstream)[source]

Bases: object

OnlineEngine class.

execute()[source]

Execute the engine - currently simple executes all workflows.

hyperstream.plugin_manager module

Plugin manager module for additional user added channels and tools.

class hyperstream.plugin_manager.Plugin[source]

Bases: hyperstream.plugin_manager.PluginBase, hyperstream.utils.utils.Printable

Plugin class - simple wrapper over namedtuple

load()[source]

Loads the channels and tools given the plugin path specified

Returns:The loaded channels, including a tool channel, for the tools found.

hyperstream.time_interval module

Module for dealing with time intervals containing TimeInterval, TimeIntervals, and RelativeTimeInterval

class hyperstream.time_interval.RelativeTimeInterval(start, end)[source]

Bases: hyperstream.time_interval.TimeInterval

Relative time interval object. Thin wrapper around a (start, end) tuple of timedelta objects that provides some validation

validate_types(val)[source]
class hyperstream.time_interval.TimeInterval(start, end)[source]

Bases: object

Time interval object. Thin wrapper around a (start, end) tuple of datetime objects that provides some validation

end
start
to_tuple()[source]
validate_types(val)[source]
width
class hyperstream.time_interval.TimeIntervals(intervals=None)[source]

Bases: hyperstream.utils.utils.Printable

Container class for time intervals, that manages splitting and joining Example object: (t1,t2] U (t3,t4] U ...

compress()[source]
end
is_empty
span
split(points)[source]
start
hyperstream.time_interval.parse_time_tuple(start, end)[source]
Parse a time tuple. These can be:
relative in seconds, e.g. (-4, 0) relative in timedelta, e.g. (timedelta(seconds=-4), timedelta(0)) absolute in date/datetime, e.g. (datetime(2016, 4, 28, 20, 0, 0, 0, UTC), datetime(2016, 4, 28, 21, 0, 0, 0, UTC)) absolute in iso strings, e.g. (“2016-04-28T20:00:00.000Z”, “2016-04-28T20:01:00.000Z”)

Mixtures of relative and absolute are not allowed :param start: Start time :param end: End time :type start: int | timedelta | datetime | str :type end: int | timedelta | datetime | str :return: TimeInterval or RelativeTimeInterval object

hyperstream.tool module

Tool module. Defines Tool and MultiOutputTool base classes.

class hyperstream.tool.AggregateTool(aggregation_meta_data, **kwargs)[source]

Bases: hyperstream.tool.Tool

This type of tool aggregates over a given plate. For example, if the input is all the streams in a node on plate A.B, and the aggregation is over plate B, the results will live on plate A alone. This can also be thought of as marginalising one dimension of a tensor over the plates

class hyperstream.tool.MultiOutputTool(**kwargs)[source]

Bases: hyperstream.utils.utils.Printable, hyperstream.utils.utils.Hashable

Special type of tool that outputs multiple streams on a new plate rather than a single stream. There are in this case multiple sinks rather than a single sink, and a single source rather than multiple sources. Note that no alignment stream is required here. Also note that we don’t subclass Tool due to different calling signatures

execute(source, sinks, interval, input_plate_value, output_plate)[source]

Execute the tool over the given time interval.

Parameters:
  • source (Stream) – The source stream
  • sinks (list[Stream] | tuple[Stream]) – The sink streams
  • interval (TimeInterval) – The time interval
  • input_plate_value (tuple[tuple[str, str]] | None) – The value of the plate where data comes from (can be None)
  • output_plate (Plate) – The plate where data is put onto
Returns:

None

message(interval)[source]
name
class hyperstream.tool.Tool(**kwargs)[source]

Bases: hyperstream.utils.utils.Printable, hyperstream.utils.utils.Hashable

Base class for tools. Tools are the unit of computation, operating on input streams to produce an output stream

execute(sources, sink, alignment_stream, interval)[source]

Execute the tool over the given time interval. If an alignment stream is given, the output instances will be aligned to this stream

Parameters:
  • sources (list[Stream] | tuple[Stream] | None) – The source streams (possibly None)
  • sink (Stream) – The sink stream
  • alignment_stream (Stream | None) – The alignment stream
  • interval (TimeInterval) – The time interval
Returns:

None

message(interval)[source]
name
hyperstream.tool.check_input_stream_count(expected_number_of_streams)[source]

Decorator for Tool._execute that checks the number of input streams

Parameters:expected_number_of_streams – The expected number of streams
Returns:the decorator

hyperstream.version module

Module contents