hyperstream.workflow package¶
Submodules¶
hyperstream.workflow.factor module¶
-
class
hyperstream.workflow.factor.
Factor
(tool, source_nodes, sink_node, alignment_node, plates)[source]¶ Bases:
hyperstream.utils.utils.Printable
A factor in the graph. This defines the element of computation: the tool along with the source and sink nodes.
-
execute
(time_interval)[source]¶ Execute the factor over the given time interval :param time_interval: :return:
-
get_alignment_stream
(plate=None, plate_value=None)[source]¶ Gets the alignment stream for a particular plate value :param plate: The plate on which the alignment node lives :param plate_value: The plate value to select the stream from the node :return: The alignment stream
-
get_sources
(plate, plate_value, sources=None)[source]¶ Gets the source streams for a given plate value on a plate. Also populates with source streams that are valid for the parent plates of this plate, with the appropriate meta-data for the parent plate. :param plate: The plate being operated on :param plate_value: The specific plate value of interest :param sources: The currently found sources (for recursion) :return: The appropriate source streams :type plate: Plate :type plate_value: tuple :type sources: list[Stream] | None
-
-
class
hyperstream.workflow.factor.
MultiOutputFactor
(tool, source_node, sink_node, input_plate, output_plates)[source]¶ Bases:
hyperstream.utils.utils.Printable
A multi-output factor in the graph. As with a factor, this links source nodes to sink nodes. However in this case the source node is being split onto multiple plates. Note there is no concept of an alignment node here.
hyperstream.workflow.node module¶
Node module.
Nodes are a collection of streams defined by shared meta-data keys (plates), and are connected in the computational graph by factors.
-
class
hyperstream.workflow.node.
Node
(node_id, streams, plate_ids, plate_values)[source]¶ Bases:
hyperstream.utils.utils.Printable
A node in the graph. This consists of a set of streams defined over a set of plates
-
static
combine_plate_values
(parent_plate_value, plate_value)[source]¶ Combine the plate value(s) with the parent plate value(s) :param parent_plate_value: The parent plate value(s) :param plate_value: The plate value(s) :return: The combined plate values
-
factor
¶
-
intersection
(meta)[source]¶ Get the intersection between the meta data given and the meta data contained within the plates. Since all of the streams have the same meta data keys (but differing values) we only need to consider the first stream. :param meta: The meta data to compare :return: A stream id with the intersection between this node’s meta data and the given meta data :type meta: dict :rtype: StreamId
-
plate_values
= None¶ When defining streams, it will be useful to be able to query node objects to determine the streams that have metadata of a particular value. Use Node.reverse_lookup as follows:
meta_data = {‘a’: 1, ‘b’: 1}
-
print_head
(parent_plate_value, plate_values, interval, n=10, print_func=<function info>)[source]¶ Print the first n values from the streams in the given time interval. The parent plate value is the value of the parent plate, and then the plate values are the values for the plate that are to be printed.
e.g. print_head()Parameters: - parent_plate_value – The (fixed) parent plate value
- plate_values – The plate values over which to loop
- interval – The time interval
- n – The maximum number of elements to print
- print_func – The function used for printing (e.g. logging.info() or print())
Returns: None
-
static
hyperstream.workflow.plate module¶
Plate and plate manager definitions.
-
class
hyperstream.workflow.plate.
Plate
(plate_id, meta_data_id, values, parent_plate=None)[source]¶ Bases:
hyperstream.utils.utils.Printable
A plate in the execution graph. This can be thought of as a “for loop” over the streams in a node
-
ancestor_meta_data_ids
¶
-
ancestor_plate_ids
¶
-
ancestor_plates
¶
-
is_root
¶
-
hyperstream.workflow.plate_manager module¶
-
class
hyperstream.workflow.plate_manager.
PlateManager
(global_meta_data)[source]¶ Bases:
hyperstream.utils.utils.Printable
Plate manager. Manages the mapping between plates defined in the database with the global meta data definition.
-
add_plate
(plate_definition)[source]¶ Add a plate using the plate definition :param plate_definition: The plate definition :return: None :type plate_definition: PlateDefinitionModel
-
create_plate
(plate_id, description, meta_data_id, values, complement, parent_plate)[source]¶ Create a new plate, and commit it to the database :param plate_id: The plate id - required to be unique :param description: A human readable description :param meta_data_id: The meta data id, which should correspond to the tag in the global meta data :param values: Either a list of string values, or the empty list (for use with complement) :param complement: If complement is true, then the complement of the values list will be used when getting values from the global meta data :param parent_plate: The parent plate identifier :return: The newly created plate :type plate_id: str | unicode :type complement: bool :type values: list | tuple
-
static
get_parent_data
(tree, node, d)[source]¶ Recurse up the tree getting parent data :param tree: The tree :param node: The current node :param d: The initial dictionary :return: The hierarchical dictionary
-
hyperstream.workflow.workflow module¶
Workflow and WorkflowManager definitions.
-
class
hyperstream.workflow.workflow.
Workflow
(channels, plates, workflow_id, name, description, owner)[source]¶ Bases:
hyperstream.utils.utils.Printable
Workflow. This defines the graph of operations through “nodes” and “factors”.
-
cartesian_product
(plate_ids)[source]¶ Gets the plate values that are the cartesian product of the plate values of the lists of plates given
Parameters: plate_ids (list[str] | list[unicode] | set[str] | set[unicode]) – The list of plate ids Returns: The plate values
-
static
check_plate_compatibility
(tool, source_plate, sink_plate)[source]¶ Checks whether the source and sink plate are compatible given the tool
Parameters: Returns: Either an error, or None
Return type: None | str
-
create_factor
(tool, sources, sink, alignment_node=None)[source]¶ Creates a factor. Instantiates a single tool for all of the plates, and connects the source and sink nodes with that tool.
Note that the tool parameters these are currently fixed over a plate. For parameters that vary over a plate, an extra input stream should be used
Parameters: - alignment_node (Node | None) –
- tool (Tool | dict) – The tool to use. This is either an instantiated Tool object or a dict with “name” and “parameters”
- sources (list[Node] | tuple[Node] | None) – The source nodes
- sink (Node) – The sink node
Returns: The factor object
Return type:
-
create_multi_output_factor
(tool, source, sink)[source]¶ Creates a multi-output factor. This takes a single node, applies a MultiOutputTool to create multiple nodes on a new plate Instantiates a single tool for all of the input plate values, and connects the source and sink nodes with that tool.
Note that the tool parameters these are currently fixed over a plate. For parameters that vary over a plate, an extra input stream should be used
Parameters: Returns: The factor object
Return type:
-
create_node
(stream_name, channel, plate_ids)[source]¶ Create a node in the graph. Note: assumes that the streams already exist
Parameters: - stream_name – The name of the stream
- channel – The channel where this stream lives
- plate_ids – The plate ids. The stream meta-data will be auto-generated from these
Returns: The streams associated with this node
-
execute
(time_interval)[source]¶ Here we execute the factors over the streams in the workflow
Returns: None
-
get_overlapping_plate_values
(plate_ids)[source]¶ Need to find where in the tree the two plates intersect, e.g.
We are given as input plates D, E, whose positions in the tree are:
root -> A -> B -> C -> D root -> A -> B -> E
The results should then be the cartesian product between C, D, E looped over A and B
If there’s a shared plate in the hierarchy, we need to join on this shared plate, e.g.:
- [self.plates[p].values for p in plate_ids][0] =
- [((‘house’, ‘1’), (‘location’, ‘hallway’), (‘wearable’, ‘A’)),
- ((‘house’, ‘1’), (‘location’, ‘kitchen’), (‘wearable’, ‘A’))]
- [self.plates[p].values for p in plate_ids][1] =
- [((‘house’, ‘1’), (‘scripted’, ‘15’)),
- ((‘house’, ‘1’), (‘scripted’, ‘13’))]
- Result should be one stream for each of:
- [((‘house’, ‘1’), (‘location’, ‘hallway’), (‘wearable’, ‘A’), (‘scripted’, ‘15)),
- ((‘house’, ‘1’), (‘location’, ‘hallway’), (‘wearable’, ‘A’), (‘scripted’, ‘13)), ((‘house’, ‘1’), (‘location’, ‘kitchen’), (‘wearable’, ‘A’), (‘scripted’, ‘15)), ((‘house’, ‘1’), (‘location’, ‘kitchen’), (‘wearable’, ‘A’), (‘scripted’, ‘13))]
Parameters: plate_ids (list[str] | list[unicode]) – Returns: The plate values
-
hyperstream.workflow.workflow_manager module¶
-
class
hyperstream.workflow.workflow_manager.
WorkflowManager
(channel_manager, plates)[source]¶ Bases:
hyperstream.utils.utils.Printable
Workflow manager. Responsible for reading and writing workflows to the database, and can execute all of the workflows
-
add_workflow
(workflow, commit=False)[source]¶ Add a new workflow and optionally commit it to the database :param workflow: The workflow :param commit: Whether to commit the workflow to the database :type workflow: Workflow :type commit: bool :return: None
-