Mains Frequency

Measurements of Mains Frequency (aka Utility Frequency)

This repository provides measurements of the mains frequency (duration of every fully wave), taken in southern Germany (Plochingen), starting on week 38 in 2022. New data is uploaded weekly.

The data has been captured using a microcontroller with clock frequency synchronized to a 1-pps GPS signal. More details about the measuring methodology can be found below.   

For each week, two data sets are uploaded: 

1. raw data: a stream of type-length-value (TLV) records as received from the measuring device (micro-controller). 
2. clean data: sanity-checked data produced from raw data and converted to user-friendly comma-separated values (CSV) format

More information about the data formats (TLV and CSV) and about the process to clean data (sanity check) can be found below.

Source Code

Source code including the microcontroller code, server code, and jupyter notebook to process the data is made available through Github: Github Repository

Measurement Methodology

The timer capture functionality of the microcontroller is used to capture the period of the mains sine waves and the 1-pps signal of a GPS device to calibrate the timer clock of the microcontroller. 

Transformer --(sine wave)--> half-wave rectifier --(half sine wave)--> Schmitt-Trigger --(square wave)--> Microcontroller --(samples)--> Computer --> raw file 
                                                                                                            ^ 
                                                                                                            |
GPS Device <-------------------------------1-pps signal------------------------------------------------------

RAW Data Records -- TLV Files

Raw data is recorded in binary format (Little Endian) as a stream of type-length-value (TLV) records. Type is a uint16 number; length is a uint16 number defining the length of the value(s ) in bytes.

The interpretation of the value(s) depends on the type. The following types are defined:

• SAMPLES record (type 0): a batch of n = length/4 uint32 values defining the number of clock ticks of n consecutive waves. The clock ticks at a nominal rate of 42 MHz. 
• ONEPPS record (type 1): a single uint32 value defining the number of clock ticks per second, calibrated by a 1-pps signal from a GPS device.
• WALLCLOCKTIME (type 2): a single uint64 value defining nanoseconds since the UNIX epoch (00:00:00 UTC, Jan. 1, 1970) referencing the stream roughly to wallclock (real) time. Note that this is just a rough reference to real-time. In particular, not every sample is timestamped, but wallclock timestamps are inserted into the stream every second.

C code to parse raw TLV files is provided in the Git.   

Clean Data -- Processing and Format of CSV Files

Clean data, published as comma-separated values (CSV) files, is created from raw data as follows:

A sanity check is performed on 1-pps records calibrating the microcontroller clock. In seldom cases, the GPS device might not output a 1-pps signal for a short period. In these cases, the 1-pps value deviated significantly from the nominal value of 42 MHz (the nominal clock frequency of the microcontroller). Such 1-pps records are removed.

No further checks have been performed on the raw data, in particular, samples are not filtered. Recommended further post-processing includes downsampling the data to remove (very seldom) outliers affecting only one or two consecutive waves. To this end, a median filter calculating the median of five consecutive samples has shown to be very effective.   

The clean CSV file contains the following fields:

• f_mains: mains frequency as captured by uncalibrated 42 MHz clock. The crystal oscillator can be expected to have an accuracy of 30 ppm over the full temperatur range (the calibration shows an accuracy better than 20 ppm, which is typical for for crystal oscillators at room temperature). 
• f_mains_syncd: mains frequency, calibrated to 1-pps signal of GPS device (using the value f_clk_synd as described next).
• f_clk_syncd: calibrated clock frequency of microcontroller using 1-pps signal (nominal 42 MHz).
• clk_accuracy_ppm: estimated clock accuracy in ppm (relevant when interpreting f_mains instead of f_mains_syncd).
• t_wallclock: wallclock time in nanoseconds since UNIX epoch (00:00:00 UTC, Jan 1, 1970). Note that this timestamp is only taken once per second to roughly reference the samples to wallclock time. Therefore, several samples have the same wallclock timestamp! Still, it is useful for selecting all samples of one day, hour, minute, etc.
• t_wallclock_str: string representation of t_wallclock value (in UTC). 

Gaps in Data

No or incomplete data is available during these periods due to technical maintenance:

• Nov 5-8, 2022
• Nov 23, 2022
• Dec 19, 2022