FloodNet flood sensor data - October 2020 to October 2023

Before accessing our data, please review our Data Access License Agreement, which outlines the terms and conditions associated with the use of FloodNet data.

We also encourage you to thoroughly review this description, particularly the section regarding noise in sensor data. Floods are not the only events that show up in this dataset, and we want to reduce misinterpretation of data as much as possible. If you have questions about the dataset, please contact info@floodnet.nyc.

Data Summary

The contents of the CSVs are as follows:

deployment_id - This is the unique identifier assigned to the sensor deployment

time - The ISO-8601 date-time format in UTC. e.g. 2023-02-22T14:20:30.000Z

depth_proc_mm - Time-series filtered depth in mm, sampled every ~1min using a series of custom noise filters designed to remove common manifestations of sensor noise. This is the recommended depth field to use.

depth_filt_mm - Time-series raw depth in mm, sampled every ~1min, except that values below 10mm are set to zero. This data has more noise, but you can use this to verify that depth_proc_mm did not accidentally filter a flood.

depth_raw_mm - Time-series raw depth in mm, sampled every ~1min.

How do the sensors capture data?

The FloodNet sensors measure distance at a regular interval, currently once every 60 seconds. Certain older sensors were programmed to upload once every 5 minutes, so don't be alarmed if you see sensors with different upload rates.

The sensors work using ultrasound. They send out sound waves at a frequency outside the range of human hearing and then capture the echo when it bounces back off the closest reflective surface. That means that what the sensor is ultimately measuring is a difference in time. The sensor then uses the travel time to calculate the distance to that reflective surface (hopefully the ground!).

Sometimes the sensor will send out a ping, but will never receive an echo back. In these cases, the measurement is invalid and we record a null value. If you see a null measurement, this is likely the cause.

Distance is calculated using d=v∆t2 where d is the distance to the ground, v is the speed of sound, and ∆t/2 is the time that it takes for the sound to bounce back and return to the sensor (divided by two because it's traveling twice the distance we're measuring).

If you know a bit of physics, you might be saying "Wait a second, but the speed of sound isn't constant! It varies depending on the properties of the medium that it's traveling through!"  Good catch! The speed of sound depends on a few factors including the molecular composition and density of the medium, in this case, "air". Assuming dry air (low humidity), the speed of sound, in units of m/s, is represented as v=20.05TC where TC is the ambient temperature in Celsius. The FloodNet sensors have an internal temperature sensor that it uses to do these calculations.

In order to calculate flood depth, the measured distance between the sensor and the ground below must be known under non-flooded conditions. This value is calculated through a dynamic calibration procedure that occurs at 5 AM daily, in which the distance measurements z collected over the previous three nights (between 10 PM and 5 AM are analyzed to determine the median value (z_nighttime~median). Day-time measurements are excluded from the calibration because of their temperature-related variance caused by direct sunlight producing erroneously high measurements on the sensor's internal temperature sensors. If the standard deviation of the night-time distance measurements exceeds 5 mm signifying either a flood or erroneous high variance), the previous day's z_nighttime~median calculation is used. To calculate flood depth D_t at time point t, the sensor's distance measurement at that time z_t) is subtracted from z_nighttime~median: Dt =  z_nighttime~median - z_t

This dynamic calibration approach allows data collection to adapt to changes in sensor height, caused for example by a shift in signpost position if there is a vehicle impact, or seasonal variation in baseline z_nighttime~median  readings.

Why is there noise in the data?

It may have occurred to you "How do you know that you're measuring the ground, and not a pigeon?" Well, the short answer is, we don't! That's why you may see some values that are not zero but are not actually floods.

The issue of noise is a challenge that our team is focused on and is actively researching. We have developed some methods designed to remove common manifestations of noise. The noise categories we have characterized are:

Blips: a momentary jump in the data, where it returns to the previous value a sample or two later. This is often caused by someone or something passing under the sensor while it is taking its measurement.

Boxes: a sudden and persistent jump in depth. This is commonly because something is placed beneath the sensor, such as garbage bags, bicycles, or loose trash.

Pulse Chain: a chaotic chain of pulse/box-like noise that can occur for an extended period of time. This can be caused by aberrant reflections on uneven, sloped, or complex surfaces such as the spokes on a bike wheel. 

The FloodNet data analysis pipeline uses a series of custom filters to address and filter some of this noise, and includes blip filters, detecting momentary jumps in the data, and box filters, detecting sustained jumps to a higher, near-constant value. These filters were optimized to minimize the risk of distorting the data and removing floods. We also employ a gradient filter that looks at the rate of change in depth to determine if the change observed is physically plausible. The gradient threshold used is 10 inches per minute, which is 7 times the maximum rate observed in Hurricane Ida in late summer of 2021.

Why isn't the sensor showing a flood?

As ultrasonic sensors need to be perpendicular and directly above the location that they are measuring, they are highly dependent on available mounting locations. To deploy these sensors, we are using existing street infrastructure such as signposts. This largely limits the locations on a street that we can use to install sensors, meaning that the sensors are not always located at the lowest point on the street, where floods would first develop. 

Streets have variable topography and therefore the depth measurements our sensors capture will not always reflect the depth at every part of the street. For example, if our sensor is 4 inches higher than the lowest point in the area, our sensor would be reading zero for any flooding below 4 inches, and a reading of 10 inches actually corresponds to 14 inches of depth at the lowest point. That offset calculation requires detailed elevation maps and is information that our data scientists are working to compile for any of our sensor deployments, but is not yet available.