hyperstream.workflow package

Submodules

hyperstream.workflow.factor module

class hyperstream.workflow.factor.Factor(tool, source_nodes, sink_node, alignment_node, plates)[source]

Bases: hyperstream.utils.utils.Printable

A factor in the graph. This defines the element of computation: the tool along with the source and sink nodes.

execute(time_interval)[source]

Execute the factor over the given time interval :param time_interval: :return:

get_alignment_stream(plate=None, plate_value=None)[source]

Gets the alignment stream for a particular plate value :param plate: The plate on which the alignment node lives :param plate_value: The plate value to select the stream from the node :return: The alignment stream

get_global_sources()[source]

Gets streams that live outside of the plates :return: Global streams

get_sources(plate, plate_value, sources=None)[source]

Gets the source streams for a given plate value on a plate. Also populates with source streams that are valid for the parent plates of this plate, with the appropriate meta-data for the parent plate. :param plate: The plate being operated on :param plate_value: The specific plate value of interest :param sources: The currently found sources (for recursion) :return: The appropriate source streams :type plate: Plate :type plate_value: tuple :type sources: list[Stream] | None

class hyperstream.workflow.factor.MultiOutputFactor(tool, source_node, sink_node, input_plate, output_plates)[source]

Bases: hyperstream.utils.utils.Printable

A multi-output factor in the graph. As with a factor, this links source nodes to sink nodes. However in this case the source node is being split onto multiple plates. Note there is no concept of an alignment node here.

execute(time_interval)[source]

Execute the factor over the given time interval. Note that this is normally done by the workspace, but can also be done on the factor directly :param time_interval: The time interval :return: self (for chaining)

get_alignment_stream(plate=None, plate_value=None)[source]

Gets the alignment stream for a particular plate value :param plate: The plate on which the alignment node lives :param plate_value: The plate value to select the stream from the node :return: The alignment stream

hyperstream.workflow.node module

Node module.

Nodes are a collection of streams defined by shared meta-data keys (plates), and are connected in the computational graph by factors.

class hyperstream.workflow.node.Node(node_id, streams, plate_ids, plate_values)[source]

Bases: hyperstream.utils.utils.Printable

A node in the graph. This consists of a set of streams defined over a set of plates

static combine_plate_values(parent_plate_value, plate_value)[source]

Combine the plate value(s) with the parent plate value(s) :param parent_plate_value: The parent plate value(s) :param plate_value: The plate value(s) :return: The combined plate values

factor
intersection(meta)[source]

Get the intersection between the meta data given and the meta data contained within the plates. Since all of the streams have the same meta data keys (but differing values) we only need to consider the first stream. :param meta: The meta data to compare :return: A stream id with the intersection between this node’s meta data and the given meta data :type meta: dict :rtype: StreamId

plate_values = None

When defining streams, it will be useful to be able to query node objects to determine the streams that have metadata of a particular value. Use Node.reverse_lookup as follows:

meta_data = {‘a’: 1, ‘b’: 1}
print_head(parent_plate_value, plate_values, interval, n=10, print_func=<function info>)[source]

Print the first n values from the streams in the given time interval. The parent plate value is the value of the parent plate, and then the plate values are the values for the plate that are to be printed.

e.g. print_head()
Parameters:
  • parent_plate_value – The (fixed) parent plate value
  • plate_values – The plate values over which to loop
  • interval – The time interval
  • n – The maximum number of elements to print
  • print_func – The function used for printing (e.g. logging.info() or print())
Returns:

None

hyperstream.workflow.plate module

Plate and plate manager definitions.

class hyperstream.workflow.plate.Plate(plate_id, meta_data_id, values, parent_plate=None)[source]

Bases: hyperstream.utils.utils.Printable

A plate in the execution graph. This can be thought of as a “for loop” over the streams in a node

ancestor_meta_data_ids
ancestor_plate_ids
ancestor_plates
is_root

hyperstream.workflow.plate_manager module

class hyperstream.workflow.plate_manager.PlateManager(global_meta_data)[source]

Bases: hyperstream.utils.utils.Printable

Plate manager. Manages the mapping between plates defined in the database with the global meta data definition.

add_plate(plate_definition)[source]

Add a plate using the plate definition :param plate_definition: The plate definition :return: None :type plate_definition: PlateDefinitionModel

create_plate(plate_id, description, meta_data_id, values, complement, parent_plate)[source]

Create a new plate, and commit it to the database :param plate_id: The plate id - required to be unique :param description: A human readable description :param meta_data_id: The meta data id, which should correspond to the tag in the global meta data :param values: Either a list of string values, or the empty list (for use with complement) :param complement: If complement is true, then the complement of the values list will be used when getting values from the global meta data :param parent_plate: The parent plate identifier :return: The newly created plate :type plate_id: str | unicode :type complement: bool :type values: list | tuple

static get_parent_data(tree, node, d)[source]

Recurse up the tree getting parent data :param tree: The tree :param node: The current node :param d: The initial dictionary :return: The hierarchical dictionary

get_parent_plate_value(tree, node, value=None)[source]

Recurse up the tree getting parent plate values :param tree: The tree :param node: The current node :param value: The initial plate value :return: The plate value as a list of tuples

get_plate_values(plate_definition)[source]

Gets the plate values from the global meta data according to the given plate definition :param plate_definition: The plate definition :return: The plate values :type plate_definition: PlateDefinitionModel

hyperstream.workflow.workflow module

Workflow and WorkflowManager definitions.

class hyperstream.workflow.workflow.Workflow(channels, plates, workflow_id, name, description, owner)[source]

Bases: hyperstream.utils.utils.Printable

Workflow. This defines the graph of operations through “nodes” and “factors”.

cartesian_product(plate_ids)[source]

Gets the plate values that are the cartesian product of the plate values of the lists of plates given

Parameters:plate_ids (list[str] | list[unicode] | set[str] | set[unicode]) – The list of plate ids
Returns:The plate values
static check_multi_output_plate_compatibility(source_plates, sink_plate)[source]
static check_plate_compatibility(tool, source_plate, sink_plate)[source]

Checks whether the source and sink plate are compatible given the tool

Parameters:
  • tool (Tool) – The tool
  • source_plate (Plate) – The source plate
  • sink_plate (Plate) – The sink plate
Returns:

Either an error, or None

Return type:

None | str

create_factor(tool, sources, sink, alignment_node=None)[source]

Creates a factor. Instantiates a single tool for all of the plates, and connects the source and sink nodes with that tool.

Note that the tool parameters these are currently fixed over a plate. For parameters that vary over a plate, an extra input stream should be used

Parameters:
  • alignment_node (Node | None) –
  • tool (Tool | dict) – The tool to use. This is either an instantiated Tool object or a dict with “name” and “parameters”
  • sources (list[Node] | tuple[Node] | None) – The source nodes
  • sink (Node) – The sink node
Returns:

The factor object

Return type:

Factor

create_multi_output_factor(tool, source, sink)[source]

Creates a multi-output factor. This takes a single node, applies a MultiOutputTool to create multiple nodes on a new plate Instantiates a single tool for all of the input plate values, and connects the source and sink nodes with that tool.

Note that the tool parameters these are currently fixed over a plate. For parameters that vary over a plate, an extra input stream should be used

Parameters:
  • tool (MultiOutputTool | dict) – The tool to use. This is either an instantiated Tool object or a dict with “name” and “parameters”
  • source (Node) – The source node
  • sink (Node) – The sink node
Returns:

The factor object

Return type:

Factor

create_node(stream_name, channel, plate_ids)[source]

Create a node in the graph. Note: assumes that the streams already exist

Parameters:
  • stream_name – The name of the stream
  • channel – The channel where this stream lives
  • plate_ids – The plate ids. The stream meta-data will be auto-generated from these
Returns:

The streams associated with this node

execute(time_interval)[source]

Here we execute the factors over the streams in the workflow

Returns:None
get_overlapping_plate_values(plate_ids)[source]

Need to find where in the tree the two plates intersect, e.g.

We are given as input plates D, E, whose positions in the tree are:

root -> A -> B -> C -> D root -> A -> B -> E

The results should then be the cartesian product between C, D, E looped over A and B

If there’s a shared plate in the hierarchy, we need to join on this shared plate, e.g.:

[self.plates[p].values for p in plate_ids][0] =
[((‘house’, ‘1’), (‘location’, ‘hallway’), (‘wearable’, ‘A’)),
((‘house’, ‘1’), (‘location’, ‘kitchen’), (‘wearable’, ‘A’))]
[self.plates[p].values for p in plate_ids][1] =
[((‘house’, ‘1’), (‘scripted’, ‘15’)),
((‘house’, ‘1’), (‘scripted’, ‘13’))]
Result should be one stream for each of:
[((‘house’, ‘1’), (‘location’, ‘hallway’), (‘wearable’, ‘A’), (‘scripted’, ‘15)),
((‘house’, ‘1’), (‘location’, ‘hallway’), (‘wearable’, ‘A’), (‘scripted’, ‘13)), ((‘house’, ‘1’), (‘location’, ‘kitchen’), (‘wearable’, ‘A’), (‘scripted’, ‘15)), ((‘house’, ‘1’), (‘location’, ‘kitchen’), (‘wearable’, ‘A’), (‘scripted’, ‘13))]
Parameters:plate_ids (list[str] | list[unicode]) –
Returns:The plate values

hyperstream.workflow.workflow_manager module

class hyperstream.workflow.workflow_manager.WorkflowManager(channel_manager, plates)[source]

Bases: hyperstream.utils.utils.Printable

Workflow manager. Responsible for reading and writing workflows to the database, and can execute all of the workflows

add_workflow(workflow, commit=False)[source]

Add a new workflow and optionally commit it to the database :param workflow: The workflow :param commit: Whether to commit the workflow to the database :type workflow: Workflow :type commit: bool :return: None

commit_all()[source]

Commit all workflows to the database :return: None

commit_workflow(workflow_id)[source]

Commit the workflow to the database :param workflow_id: The workflow id :return: None

execute_all()[source]

Execute all workflows

Module contents