Python bindings for LSDTopoTools: Users and dev guide

The first step is to create a python environment. As stated before, python is a "packaged" language: all the code take advantage of other bit of codes made by the community called packages. The (really) good thing with it is that a lot of algorithms are already coded and you can integrate it in your python environment easily.
The backside is that many packages relies on specific version of other packages and there can be some conflict when the environment becomes massif, the same way ArcGIS extentions are only compatible with some versions of the main software, or MATLAB scripts that absolutely require the specific version 2015rc4267-21f-march2016v2.5 to run.
On the bright side, python is free and open-source, you can therefore install as many versions of python and its packges as you want. We strongly advice to use miniconda (or anaconda) as environment manager: you can create and activate different environments with their own packages. Even better, we can provide templates of environments to make it easy-to-install.
A practical example: you can have one environment with lsdtopytools installed to run topographic analysis, a separate one with landlab to deals with modelling or even play with the packages to mix both tools in the same environment and run it together.

You can download miniconda from this website, and install it. It does not matter whether you have miniconda or anaconda: these are just light and full versions of the same tool. We advice miniconda to only install what you need and we advice python 3.6 or 3.7: we only tested our code on these versions. The code is theoretically compatible with 3.5, 3.4 and 2.7 but I would imagine that if you need such a specific version of python, you’ll know how to deal with custum installation. Also, you need to know if your computer is 32 (if old) or 64 bits (most of post 2010 computer).
You do not need admin right to install conda on your machine in case you are using institutions servers or computers.

As many (scientific) tools, the scripts need to be ran in a console/terminal/command prompt/whichever name you want. Many reasons for that: easier to manage, control and automate; quicker; actually easy to use. The only bad side is that it looks less attractive than a nice graphic interface. Basically, rather than double clic on a folder, you need to write cd name_of_the_folder to chagne the current directory for instance.

You are now within a conda environment that includes python. Let’s have a clean install, cleaner computing is easier to debug. The following command creates a ("sub")environment in you conda installation, where we will install the packages we need. To run the command, just copy-paste it in the terminal and hit enter.

Once created, the environment just need to be activate each time a new terminal is opened! You just have to run the following command:

You’ll know it is activated if (lsdtopytools) is displayed in your terminal. If you have an old version of conda, you may face a message error. You just have to use this command instead:

We’re basically done with environment stuff. You can create as many (or as few) environments as you need for your different programs (e.g., landlab, opencv).

let’s now install the required tools to run lsdtt_xtensor_python (the cpp-python mediator): we take advantage of carefully selected bits of other codes to help us.
The following command installs package. install means (surprise) install, and -c conda-forge means from the channel (= repository) "conda-forge" which is in plain language a big pile of easy-to-install python packages. + The following keywords are the name of the packages: (i) pip, it makes easier python packages installation, (ii) numpy, an EXTREMELY used and powerful tools for arrays and matrix calculations. It is written in fortran and c, therefore its built-in functions (e.g., sum, median, reordering, and millions of common tools) are fast. (iii) pybinds, requires to link cpp to python.

You environment is ready for lsdtt_xtensor_python:

lsdtt_xtensor_python is required for any use of the python tools, but only provide minimal interface with the cpp code and require knowledge of its structure. You probably now need lsdtopytools, the full, user-friendly interface that also provide (will provide actually as it is work in progress) easy raster loading/saving, ergonomic data management and visualisation routines. This however requires a bunch of other packages to be install: the simplest a program is, the bigger the amount of code behind is.
lsdtopytools, even if you won’t inteact with most of these, basically requires the following packages:

It shall run for few minuts to install these dependencies, and finally you are ready for the last bit:

This one should be universal (i.e., Compatible with with any OS as pure python stuffs).

And we are done with the installation!

There will be some, no worries.

python is a nice high-level programming language. It is part of the interpreted languages, meaning that each line of code is interpreted one by one by a software (here python), other well-know free examples are perl, R, Julia or the somehow popular not free and not open source MATLAB. These languages are useful to take abstraction of complex machine interaction such as memory management and control other more complex underlying code. There is no need to have advanced python or coding knowledge to use our tools, python is easy to write and read. Code academy offers a nice tutorial to get the basis (not mandatory but can be useful).
To run a custom python script: simply navigate to the right directory within your conda terminal (see installation instructions) and run python name_of_my_script.py.
If you don’t want to learn this useful tool, use the command-line interface.

lsdtopytools is coded in an object oriented logic: we have some precoded routines absed on a common template that you just need to construct with your custom parameters. Basically all our routines are based on an original DEM that you load first. Then each routines depends on others and the code deals internally on how to call these interaction. This sound probably fuzzy so let’s illustrate with an example:
Let’s imagine you have a geotif DEM of your area and you want to extract channel steepness. You need many information to do so: the code needs to know the flow direction of your DEM cells, it needs to know the drainage area, which pixels is a river and many different bits of data you probably don’t have yet. Thanks to the object orientation you just have to call the ksn extraction routine and the code will know what it needs to calculate beforehand, tells you what it is defaulting and what you could add to cusomise each steps and finally run the wanted routine.

The first step of each routines is to load the base raster: your dem from which the metrics will be calculated. You simply need to import the code (as any python code) and load the dem as follow:

+ THis is the simplest way to load a dem. It however implies that your dem is in the same folder than your python script. You can load a de with the following options:

, path = "./", file_name = "test.tif", already_preprocessed = False, verbose = True)

Before explaining each routines, all of them are build following the same pattern around the LSDDEM object. You call the routine ON the object the following way:

+

It either process and store internally new result, return a dataset to you or generate a figure depending on the routine called. In a single script, if a routine has already been ran, the results are usually stored internally.

Not available yet. But some routines can take a certain amount of time to run (even in c++), I am therefore implementing a save-load system based on h5py package. The aim is to load and save only what you need behind the scene to save memory and processing time. It will take a wee time to do properly, please bear with me. An easy solution would be to use pickle, but by experience it crashes and uses a LOT of memory when dealing with large dataset (e.g, a raster).

Most of the routines uses flow information to generates metrics, the most striking example being river extraction: the program obviously needs to know where the water will flow to calculate rivers. Preprocessing in our case means making sure that all our cells are draining to another one or to an edge. This is essential as DEM are inherantly noisy: this noise will produce artificial pits and depressions. Although minor in the digital topography, these will be fatal to a program as the program will end any flow into these pit therefore biasing drainage area, flow direction, basin outlet or many other routines.
The solution is to carve (sometimes refered as breaching) or to fill these depressions. Respectively, it either "dig" a way out of the depression to the nearest outreach or fill the depression until water can (numerically, there no real water involved) drain to another cell. Carving usually modify less cells than filling but is more computationnaly expensive, and filling is usually required after carving to ensure minimum slope on flat surfaces.

In the program, you can either feed the code with a dem externaly preprocessed, see Load your data into the system, or use one of our filing and carving algorithm:

It preprocesses your base raster and save it into the associated LSDDEM object. The options are the following:

This guide does not aim to discuss the filling and carving algorithm performances, relevant literature does that. So short story long, one might want to check the preprocessing result because (i) filling can generate "overfilled" areas, where for example a bridge or a dam creates an artificial obstacle; (ii) carving and filling alters the original data and therefore can generate unwanted artifact that one need to be aware to avoid misinterpretations.