hyperstream package

Subpackages

Submodules

hyperstream.channel_manager module

Channel manager module. Defines the ChannelManager - a container for channels, that can load in plugins

class hyperstream.channel_manager.ChannelManager(plugins, **kwargs)[source]

Bases: dict, hyperstream.utils.utils.Printable

Container for channels.

database_channels

The database channels as a list

get_channel(channel_id)[source]

Get the channel by id

Parameters:channel_id – The channel id
Returns:The channel object
get_tool(name, parameters, version=None)[source]

Gets the tool object from the tool channel(s), and instantiates it using the tool parameters

Parameters:
  • name – The name or stream id for the tool in the tool channel
  • parameters – The parameters for the tool
  • version – The string representation of the version
Returns:

The instantiated tool object
get_tool_class(tool)[source]

Gets the actual class which can then be instantiated with its parameters

Parameters:tool (str | unicode | StreamId) – The tool name or id
Return type:Tool | MultiOutputTool
Returns:The tool class
memory_channels

The memory channels as a list

populate_assets(tool_id)[source]

TODO: Unused?

tool_channels

The tool channels as a list

update_channels()[source]

Pulls out all of the stream definitions from the database, and populates the channels with stream references

hyperstream.client module

The main hyperstream client connection that is used for storing runtime information. Note that this is also used by the default database channel, although other database channels (connecting to different database types) can also be used.

class hyperstream.client.Client(server_config, auto_connect=True)[source]

Bases: object

The main mongo client

client = None
connect(server_config)[source]

Connect using the configuration given

Parameters:server_config – The server configuration
db = None
get_config_value(key, default=None)[source]

Get a specific value from the configuration

Parameters:
  • key – The of the item
  • default – A default value if not found
Returns:

The found value or the default
session = None

hyperstream.config module

HyperStream configuration module.

class hyperstream.config.HyperStreamConfig[source]

Bases: hyperstream.utils.utils.Printable

Wrapper around the hyperstream configuration files (hyperstream_config.json and meta_data.json)

class hyperstream.config.OnlineEngineConfig(interval, sleep=5, iterations=100, alarm=None)[source]

Bases: hyperstream.utils.utils.Printable

hyperstream.hyperstream module

Main HyperStream class

class hyperstream.hyperstream.HyperStream(loglevel=10, file_logger=True, console_logger=True)[source]

Bases: object

HyperStream class: can be instantiated simply with hyperstream = HyperStream() for default operation

create_workflow(workflow_id, name, owner, description, online=False)[source]

Create a new workflow. Simple wrapper for creating a workflow and adding it to the workflow manager.

Parameters:
  • workflow_id – The workflow id
  • name – The workflow name
  • owner – The owner/creator of the workflow
  • description – A human readable description
  • online – Whether this workflow should be executed by the online engine
Returns:

The workflow

hyperstream.online_engine module

Online Engine module. This will be used in the online execution mode.

class hyperstream.online_engine.OnlineEngine(hyperstream)[source]

Bases: object

OnlineEngine class.

execute(*args, **kwargs)[source]

Execute the engine - currently simple executes all workflows.

hyperstream.plugin_manager module

Plugin manager module for additional user added channels and tools.

class hyperstream.plugin_manager.Plugin[source]

Bases: hyperstream.plugin_manager.PluginBase, hyperstream.utils.utils.Printable

Plugin class - simple wrapper over namedtuple

load_channels()[source]

Loads the channels and tools given the plugin path specified

Returns:The loaded channels, including a tool channel, for the tools found.

hyperstream.time_interval module

Module for dealing with time intervals containing TimeInterval, TimeIntervals, and RelativeTimeInterval

class hyperstream.time_interval.RelativeTimeInterval(start, end)[source]

Bases: hyperstream.time_interval.TimeInterval

Relative time interval object. Thin wrapper around a (start, end) tuple of timedelta objects that provides some validation

absolute(dt)[source]
end
start
class hyperstream.time_interval.TimeInterval(start, end)[source]

Bases: hyperstream.time_interval.TimeInterval

Time interval object. Thin wrapper around a (start, end) tuple of datetime objects that provides some validation

classmethod all_time()[source]
end
humanized
classmethod now_minus(weeks=0, days=0, hours=0, minutes=0, seconds=0, milliseconds=0)[source]
start
to_tuple()[source]
classmethod up_to_now()[source]
width
class hyperstream.time_interval.TimeIntervals(intervals=None)[source]

Bases: hyperstream.utils.utils.Printable

Container class for time intervals, that manages splitting and joining Example object: (t1,t2] U (t3,t4] U ...

compress()[source]
end
humanized
is_empty
parse(intervals)[source]
span
split(points)[source]
start
hyperstream.time_interval.parse_time_tuple(start, end)[source]
Parse a time tuple. These can be:
relative in seconds, e.g. (-4, 0) relative in timedelta, e.g. (timedelta(seconds=-4), timedelta(0)) absolute in date/datetime, e.g. (datetime(2016, 4, 28, 20, 0, 0, 0, UTC), datetime(2016, 4, 28, 21, 0, 0, 0, UTC)) absolute in iso strings, e.g. (“2016-04-28T20:00:00.000Z”, “2016-04-28T20:01:00.000Z”) Mixtures of relative and absolute are not allowed
Parameters:
  • start (int | timedelta | datetime | str) – Start time
  • end (int | timedelta | datetime | str) – End time
Returns:

TimeInterval or RelativeTimeInterval object
hyperstream.time_interval.profile(ob)[source]

Comment out this function to be able to use the line_profiler module. e.g. call: kernprof -l scripts/deploy_summariser.py –loglevel=10 python -m line_profiler deploy_summariser.py.lprof > deploy_summariser.py.summary :param ob: object :return: object

hyperstream.version module

Module contents