.. figure:: https://cdn.rawgit.com/IRC-SPHERE/HyperStream/dfbac332/hyperstream.svg
   :alt: HyperStream logo

   HyperStream logo

HyperStream
===========

|DOI| |Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby|
|Build Status| |Dependency Status| |Test Coverage| |Issue Count|
|Documentation Status|

Hyperstream is a large-scale, flexible and robust software package for
processing streaming data.

-  HyperStream `homepage <https://irc-sphere.github.io/HyperStream/>`__
-  Tutorial
   `notebooks <http://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/>`__
-  Gitter `chat room <https://gitter.im/IRC-SPHERE-HyperStream/Lobby>`__
-  Developer
   `documentation <http://hyperstream.readthedocs.io/en/latest/>`__

Hyperstream overcomes the limitations of other computational engines and
provides high-level interfaces to execute complex nesting, fusion, and
prediction both in online and offline forms in streaming environments.
Although developed specifically for SPHERE, Hyperstream is a general
purpose tool that is well-suited for the design, development, and
deployment of algorithms and predictive models in a wide space of
sequential predictive problems.

This software has been designed from the outset to be
domain-independent, in order to provide maximum value to the wider
community. Key aspects of the software include the capability to create
complex interlinked workflows, and a computational engine that is
designed to be "compute-on-request", meaning that no unnecessary
resources are used.

Installation
============

Docker images
-------------

If you do not want to install all the packages separately you can use
our Docker bundle available
`here <https://github.com/IRC-SPHERE/Hyperstream-Dockerfiles>`__.

Local machine
-------------

Install via pip

::

    pip install hyperstream
    python -c 'from hyperstream import HyperStream'

To get the latest version

::

    pip install -U git+git://github.com/IRC-SPHERE/HyperStream.git#egg=hyperstream
    pip install -r requirements.txt

Or clone the repository

.. code:: bash

    git clone git@github.com:IRC-SPHERE/HyperStream.git
    cd HyperStream
    virtualenv venv
    . venv/bin/activate
    pip install -r requirements.txt
    python -c 'from hyperstream import HyperStream'

Additionally, one of the requirements to run Hyperstream is a MongoDB
server. By default, Hyperstream tries to connect to the port 27017 on
the localhost.

To see the installation steps of MongoDB go to the `official
documentation <https://docs.mongodb.com/manual/installation/>`__. E.g.
in a Debian OS it is possible to install with the following command

.. code:: bash

    sudo apt-get install mongodb

Once the MongoDB server is installed, it can be started with the
following command

.. code:: bash

    service mongod start

Running tests
=============

Run the following command

::

    nosetests

Note that for the MQTT logging test to succeed, you will need to have an
MQTT broker running (e.g. Mosquitto). For example:

::

    docker run -ti -p 1883:1883 -p 9001:9001 toke/mosquitto

or on OSX you will need pidof and mosquitto:

::

    brew install pidof
    brew install mosquitto
    brew services start mosquitto

Tutorials
=========

The following tutorials show how to use HyperStream in a step-by-step
guide.

-  `Tutorial 1:
   Introduction <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_01.ipynb>`__
-  `Tutorial 2: Creating
   tools <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_02.ipynb>`__
-  `Tutorial 3: Stream
   composition <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_03.ipynb>`__
-  `Tutorial 4: Real-time
   streams <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_04.ipynb>`__
-  `Tutorial 5:
   Workflows <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_05.ipynb>`__

.. code:: python

        from hyperstream import HyperStream, StreamId, TimeInterval
        from hyperstream.utils import utcnow, UTC
        from datetime import timedelta

        hs = HyperStream(loglevel=20)
        M = hs.channel_manager.memory
        T = hs.channel_manager.tools
        clock = StreamId(name="clock")
        clock_tool = T[clock].window().last().value()
        ticker = M.get_or_create_stream(stream_id=StreamId(name="ticker"))
        now = utcnow()
        before = (now - timedelta(seconds=30)).replace(tzinfo=UTC)
        ti = TimeInterval(before, now)
        clock_tool.execute(sources=[], sink=ticker, interval=ti, alignment_stream=None)
        print(list(ticker.window().tail(5)))

        [StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=<UTC>))]

HyperStream Viewer
==================

The `HyperStream
Viewer <https://github.com/IRC-SPHERE/HyperStreamViewer>`__ is a
python/Flask web-app for interacting with HyperStream. In order to keep
HyperStream to a minimum, this web-app is released as a separate
repository that takes the core as a dependency.

License
=======

This code is released under the `MIT
license <https://github.com/IRC-SPHERE/HyperStream/blob/master/LICENSE>`__.

Acknowledgements
================

This work has been funded by the UK Engineering and Physical Sciences
Research Council (EPSRC) under Grant
`EP/K031910/1 <http://gow.epsrc.ac.uk/NGBOViewGrant.aspx?GrantRef=EP/K031910/1>`__
- "SPHERE Interdisciplinary Research Collaboration".

.. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.242227.svg
   :target: https://doi.org/10.5281/zenodo.242227
.. |Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby| image:: https://badges.gitter.im/IRC-SPHERE-HyperStream/Lobby.svg
   :target: https://gitter.im/IRC-SPHERE-HyperStream/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
.. |Build Status| image:: https://travis-ci.org/IRC-SPHERE/HyperStream.svg?branch=master
   :target: https://travis-ci.org/IRC-SPHERE/HyperStream
.. |Dependency Status| image:: https://www.versioneye.com/user/projects/58e423cb26a5bb005220301e/badge.svg?style=flat-square
   :target: https://www.versioneye.com/user/projects/58e423cb26a5bb005220301e
.. |Test Coverage| image:: https://codeclimate.com/github/IRC-SPHERE/HyperStream/badges/coverage.svg
   :target: https://codeclimate.com/github/IRC-SPHERE/HyperStream/coverage
.. |Issue Count| image:: https://codeclimate.com/github/IRC-SPHERE/HyperStream/badges/issue_count.svg
   :target: https://codeclimate.com/github/IRC-SPHERE/HyperStream
.. |Documentation Status| image:: https://readthedocs.org/projects/hyperstream/badge/?version=latest
   :target: http://hyperstream.readthedocs.io/en/latest/?badge=latest
