PyDaddy
Discovering stochastic dynamical equations from ecological time series data, together with an easy to use Python package.
Citation to the manuscript
Nabeel, A., Karichannavar, A., Palathingal, S., Jhawar, J., Bruckner, B., Danny Raj, M., & Guttal, V., “Discovering stochastic dynamical equations from ecological time series data”, arXiv preprint arXiv:2205.02645, to appear in The American Naturalist.
Citation to the package
Nabeel, A., Karichannavar, A., Palathingal, S., Jhawar, J., Bruckner, David B., Danny Raj, M., & Guttal, V. (2024). PyDaddy: A Python Package for Discovering SDEs from Time Series Data (Version 1.1.1) [Computer software]. https://github.com/tee-lab/PyDaddy
Overview of the study and package
PyDaddy is an open source package which is a key contribution of the manuscript Nabeel et al, arXiv:2205.02645. The basic scientific premise for this package is to discover the nature of stochasticity in ecological time series datasets. It is well known that the stochasticity can affect the dynamics of ecological systems in counter-intuitive ways. Without understanding the equations (typically, in the form of stochastic differential equations or SDEs, in short) that govern the dynamics of populations or ecosystems, it’s challenging to determine the impact of randomness on real datasets. In this manuscript and accompanying package, we introduce a methodology for discovering equations (SDEs) that transforms time series data of state variables into stochastic differential equations. This approach merges traditional stochastic calculus with modern equation-discovery techniques. We showcase the generality of our method through various applications and discuss its limitations and potential pitfalls, offering diagnostic measures to address these challenges.
PyDaddy is a comprehensive and easy to use python package to discover data-derived stochastic differential equations from time series data. PyDaddy takes the time series of state variable \(x\), scalar or 2-dimensional vector, as input and discovers an SDE of the form:
\[ \frac{dx}{dt} = f(x) + g(x) \cdot \eta(t) \]
where \(\eta(t)\) is Gaussian white noise. The function \(f\) is called the drift, and governs the deterministic part of the dynamics. \(g^2\) is called the diffusion and governs the stochastic part of the dynamics.
 |
|---|
| An example summary plot generated by PyDaddy, for a vector time series dataset. |
PyDaddy also provides a range of functionality such as equation-learning for the drift and diffusion functions using sparse regresssion, a suite of diagnostic functions, etc.. For more details on how to use the package, check out the example notebooks and documentation.
Workflow and Documentation
The workflow of the package is summarised by the schematic given below - which is also the Fig 1 of the manuscript (https://arxiv.org/abs/2205.02645).
A detailed workflow of the package along with detailed instructions on various features are included as Supplementary Information Section S2 of the manuscript.
Detailed documentation of the PyDaddy package can be found at documentation.
 |
|---|
| Schematic illustration of PyDaddy functionality. |
Getting Started Without Package Installation
We provide a number of easy to use scripts for the ease of learning and using the package, and with an aim that our manuscript is easily reproducible.
First, we emphasise that PyDaddy can be executed online on Google Colab, without having to install it on your local machine. To run PyDaddy on Colab, open a notebook on Colab. Paste the following code on a notebook cell and run it:
%pip install git+https://github.com/tee-lab/PyDaddy.gitThis sets up PyDaddy in the notebook environment.
Example Scripts/Jupyter Notebooks
There are several example scripts / Jupyter notebooks provided, which can be used to familiarize yourself with various features and functionalities of PyDaddy. These can be executed on Colab. In the list below, we mention the path to location of each notebook as well as a link to the google colab notebook; the latter does not require installing either python or package on your system.
- Notebook 1
(notebooks/1_getting_started.ipynb): Getting
started with scalar data: Introduction to the basic functionalities
of PyDaddy, using a 1-dimensional dataset.
- Notebook 2 (notebooks/2_getting_started_vector.ipynb): Getting started with vector data: Introduction to the basic functionalities of PyDaddy on 2-dimensional datasets.
- Notebook 3 (notebooks/3_advanced_function_fitting.ipynb): Advanced function fitting: PyDaddy can discover analytical expressions for the drift and diffusion functions. This notebook describes how to customize the fitting procedure to obtain best results.
- Notebook 4 (notebooks/4_sdes_from_simulated_timeseries.ipynb): Recovering SDEs from synthetic time series: This notebook generates a simulated time series from a user-specified SDE, and uses PyDaddy to recover the drift and diffusion functions from the simulated time series.
- Notebook 5 (notebooks/5_exporting_data.ipynb): Exporting data: Demonstrates how to export the recovered drift and diffusion data as CSV files or Pandas data-frames.
- Notebook 6 (notebooks/6_non_poly_function_fitting.ipynb): Fitting non-polynomial functions: PyDaddy fits polynomial functions to drift and diffusion by default. This behaviour can be customized, this notebook illustrates how to do this.
- Notebook 9 (notebooks/9_higher_dimensions.ipynb): [Demonstration with a 3-dimensional system] An example to demonstrate that, in principle, the method of stochastic equation discovery can be extended to higher dimensions.
(See below for Notebooks 7 and 8).
Real datasets and Scripts/Jupyter Notebooks
There are also two notebooks that use PyDaddy to discover SDEs from real-world datasets.
- Fish Schooling Dataset
(pydaddy/data/fish_data/ectroplus.csv) : The fish
dataset contains the 2D polarisation vector time series of a fish school
(15 fish). Two columns in the csv file represent the x- and y-components
of the polarisation vector, respectively and each row corresponds to a
time stamp, with consecutive rows separated by a time frame of 0.04
seconds. The full dataset is available at a previously published
repository: https://zenodo.org/records/3632470. For more details
about the dataset, see the manuscript Jhawar et al - https://doi.org/10.1038/s41567-020-0787-y
- Notebook 7 (notebooks/7_example_fish_school.ipynb): Example analysis - fish schooling: An example analysis of a fish schooling dataset (Jhawar et. al., Nature Physics, 2020) using PyDaddy.
- Cell Migration Dataset
(pydaddy/data/cell_data/trajectories_x_pattern5.txt):
The confine cell migration dataset contains tracked trajectories of 149
cells, tracked for upto 300 time steps each, with one data point every
15 minutes. The data is provided as a plain text file. Each row
corresponds to the time series of one cell. For more details about the
dataset, see https://doi.org/10.1038/s41567-019-0445-4.
- Notebook 8 (notebooks/8_example_cell_migration.ipynb): Example analysis - cell migration: An example analysis of a confined cell migration dataset (Brückner et. al., Nature Physics, 2019) using PyDaddy.
Folder structure
The zipped folder of codes and data is structured as follows:
- parent folder has licence, citation, readme.md, etc files
- doc and its subfolders contain python codes and style files relevant to the package. Edit these only you are a developer and are proficient with python.
- notebooks contains nine well commented/documented jupyter-notebooks/scripts which help the readers to familiarise with the usage of the package.
- pydaddy/data folder contains three subfolders
containing key real and model datasets:
- cell_data
- fish_data
- model_data
- pydaddy and its subfolders contain various codes related to python package. Edit these only you are a developer and are proficient with python.
Package Installation
PyDaddy is available both on PyPI and Anaconda Cloud, and can be installed on any system with a Python 3 environment. If you don’t have Python 3 installed on your system, we recommend using Anaconda or Miniconda. See the PyDaddy package documentation for detailed installation instructions.
Using pip
To install the latest stable release version of PyDaddy, use:
pip install pydaddyTo install the latest development version of PyDaddy, use:
pip install git+https://github.com/tee-lab/PyDaddy.git
Detailed Package Documentation
For more information about PyDaddy, check out the package documentation.
Citation
If you are using this package in your research, please cite the repository and the associated paper as follows:
Nabeel, A., Karichannavar, A., Palathingal, S., Jhawar, J., Bruckner, David B., Danny Raj, M., & Guttal, V. (2024). PyDaddy: A Python Package for Discovering SDEs from Time Series Data (Version 1.1.1) [Computer software]. https://github.com/tee-lab/PyDaddy, DOI: To Do.
Nabeel, A., Karichannavar, A., Palathingal, S., Jhawar, J., Bruckner, B., Danny Raj, M., & Guttal, V., “Discovering stochastic dynamical equations from ecological time series data”, arXiv preprint arXiv:2205.02645, to appear in The American Naturalist.
Funding
This study was partially funded by Science and Engineering Research Board, Department of Science and Technology, Government of India to Vishwesha Guttal.
Licence
PyDaddy is distributed under the GNU General Public License v3.0.