Quality-checked meteorological data from the Southern Ocean collected during the Antarctic Circumnavigation Expedition from December 2016 to April 2017. ***** Dataset abstract ***** This dataset contains quality-checked meteorological observations of air temperature, relative humidity, dew point, barometric pressure and observations of downwelling solar radiation and ultraviolet radiation. Further it contains the wind speed and direction relative to the ship but not corrected for air-flow distortion, and translated into the earth reference frame. For each of these variables observations are available from a portside and starboard side sensor. The dataset also contains, cloud base height and sky cover at three levels measured with a Ceilometer. As additional information the solar azimuth and altitude angle have been calculated for the ship’s position every five minutes and have been added as a one-minute time series using the nearest value. The ship’s position, heading, course and speed over ground are also provided. The wind speed measurements were made at a height of approximately 30.5 meters above sea level. The measurement height of the temperature and humidity probes is 23.7 meters above sea level. The barometric pressure was measured at 20 meters above sea level. The observations have been screened for implausible values and on some occasions despiking based on visual inspection and a rolling interquartile range filter have been applied. Solar radiation measurements are affected by shadowing of the ship, and the air temperature and humidity by the heating of air that passes over the ship. Masks are provided to flag affected observations. The wind speed readings are affected by airflow distortion and should be used with consideration until a dataset of corrected wind speeds is published. More details on airflow distortion can be requested from the contact person. ***** Original data collection ***** This dataset was created from the raw dataset which has been published separately, containing details of the data collection. ***** Data processing ***** The raw data files were read and processed with code for Python 3.6 written in a Jupyter notebook on a Laptop running a Windows 10 operating system. Major manipulation of the data was as follows: - For the data from the metdata_all_*.csv and metdata_wind_*.csv files the UTC time stamp was constructed by adding the content of the column TIMEDIFF (unit is in seconds) to the time stamp provided in the first column ("date_time"). Occasionally duplicates occur in the constructed UTC timestamp. These do however show independent measurements that fit well to neighbouring samples in the time series. A likely reason for this are small drifts and noise in the digital clocks. Therefore these samples were not removed, but included in the time averages. - For the data from the one-second resolution cruise track files the time stamp given in the first column ("date_time") was taken as a UTC timestamp. - For each variable a different set of filter methods was applied to replace samples that were considered as corrupted, by NaN. The details are described below. - The true wind speed and direction were calculated following best practice as described in Smith et al. 1999. - For the Ceilometer observations where SCn = 0 and CLn = NaN, the CLn value was changed to “inf” (infinity) since these combinations represent the clear sky condition. (Here n=1,2,3). - The time series were resampled to a uniform one-minute time resolution using an NaN-average (i.e. only valid samples within the one-minute window are considered for the average). Note that the digital sample resolution of the wind speed and the meteorological observations are 3 seconds and 30 seconds respectively. - The Ceilometer observations are updated only every five minutes hence the nearest value is used rather than the average. Most variables were measured by two sensors (three for solar radiation), of which one was located on the port and one the starboard side of the ship, respectively. The relation between sensor index and position (port or starboard) was concluded from clues found in the data itself. - For the wind speed readings a shadowing of the lee side sensor by the ship’s main mast was apparent at relative wind directions orthogonal to the ship’s main axis. - For the solar radiation, the angle between the solar azimuth and the ship’s heading was used, with sudden changes in the ratio of the radiation readings, which originated from shadowing of the sensor by the ship’s main mast. - For the air temperature and humidity probe it was found that one sensor showed higher air temperature than the other (especially during the day) when the relative wind direction was from port or starboard. It was assumed that the lee side sensor readings must be those that are higher as the ship's hull would heat the passing air flow. - For Cloud level (CL) and Sky Cover (SC) the numbering refers to the first, second and third cloud level detected by the ceilometer and sky cover estimate at the respective cloud level. In this updated version (v1.1), the range check for skycover (SC) and cloudlevel (CL) was added to the quality-checking routines. 53 data points violated the range check for these variables, these have now been marked as NaN. Table 1 below summarises the meaning of the variable indices. Some variables from the raw data files were not included in this data set: - The solar radiation sensor SR2 provided only a very small number of observations and is omitted in this dataset. - The variable VISCODE provides the same information as VIS but only at a resolution of 100 meters and with an upper limit of 9999. - The variables wawa, cond, CLOUDTEXT are all NaN (null values) - The readings about salinity and sea water temperature (salinity, TwTwTw) are incomplete and without calibration. The underway observations are published in a separate data set. Table 1: list of variables and units. The last two columns provide the lower and upper bound of the plausible range assumed for the observations. Note for relative humidity the range was extended to 103% since 3% of the readings are above 100%. A flag is provided to identify these data points. | Column name | Units | Description | Lower bound | Upper bound | |----------------------------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|-------------| | date_time | In UTC | Time stamp in ISO 8601 format YYYY-MM-DDThh:mm:ss+00:00 | - | - | | latitude | decimal degrees N | latitude of the ship at the position of the date_time | -90 | 90 | | longitude | decimal degrees E | longitude of the ship at the position of the date_time | -180 | 180 | | VIS | m | Horizontal visibility | 0 | 20000 | | CL1 | m | Height of first cloud level (if SC1==9: CL1 provides the vertical visibility). CL1=inf indicates clear sky (co occurrence with SC1=0) | 0 | 7620(inf) | | CL2 | m | Height of second cloud level. CL2=inf indicates clear sky or no clouds found above the first cloud level (co occurrence with SC2=0) | 0 | 7620(inf) | | CL3 | m | Height of third cloud level. CL3=inf indicates clear sky or no clouds found above the second cloud level (co occurrence with SC3=0) | 0 | 7620(inf) | | SC1 | Octants/9 | Sky cover of first cloud level. If SC1=9 CL1 provides the vertical visibility | 0 | 8(9) | | SC2 | Octants | Sky cover of second cloud level | 0 | 8 | | SC3 | Octants | Sky cover of third cloud level | 0 | 8 | | RH1 | % | Relative humidity measured by the port sensor | 20 | 103 | | TA1 | degrees C | Air temperature measured by the port sensor | -50 | 50 | | DP1 | degrees C | Dew point provided by the port sensor | -50 | 50 | | TW1 | degrees C | Wet bulb temperature calculated from RH1 and TA1 | -50 | 50 | | PA1 | mbar | Atmospheric pressure measured by the port sensor | 500 | 1100 | | RH2 | % | Relative humidity measured by the starboard sensor | 20 | 103 | | TA2 | degrees C | Air temperature measured by the starboard sensor | -50 | 50 | | DP2 | degrees C | Dew point provided by the starboard sensor | -50 | 50 | | TW2 | degrees C | Wet bulb temperature calculated from RH2 and TA2 | -50 | 50 | | PA1 | mbar | Atmospheric pressure measured by the starboard sensor | 500 | 1100 | | SR1 | W m^-2 | Solar radiation measured by the port sensor | 0 | 2000 | | SR3 | W m^-2 | Solar radiation measured by the starboard sensor | 0 | 2000 | | UV1 | W m^-2 | Likely this is Ultraviolet radiation (wavelength range unknown) measured by the port sensor. Readings are sparse. | 0 | 100 | | UV2 | W m^-2 | Likely this is Ultraviolet radiation (wavelength range unknown) measured by the starboard sensor. Readings are sparse and do sometimes deviate from UV1 observations, these values are potentially flawed. | 0 | 100 | | WSR1 | m s^-1 | Relative wind speed measured by the starboard sensor (affected by flow distortion!) | 0 | 100 | | WDR1 | degrees | Relative wind direction (from) measured by the starboard sensor (clockwise from the bow. E.g. +90 denotes wind coming from starboard) (affected by flow distortion!) | 0 | 360 | | WSR2 | m s^-1 | Relative wind speed measured by the port sensor (affected by flow distortion!) | 0 | 100 | | WDR2 | degrees | Relative wind direction (from) measured by the port sensor (clockwise from the bow. E.g. +90 denotes wind coming from starboard) (affected by flow distortion!) | 0 | 360 | | HEADING | degrees | Ship heading (clockwise from North) | 0 | 360 | | SOG | m s^-1 | Ship speed over ground | 0 | 12 | | COG | degrees | Ship course over ground (clockwise from North) | 0 | 360 | | WS1 | m s^-1 | Wind speed relative to ground computed from starboard sensor readings and the ships heading and velocity (results are affected by flow distortion!) | 0 | 100 | | WD1 | degrees | Wind direction (from) computed computed from starboard sensor readings and the ships heading and velocity (results are affected by flow distortion!) (clockwise from North E.g. +90 denotes wind coming from East) | 0 | 360 | | WS2 | m s^-1 | Wind speed relative to ground computed from port sensor readings and the ships heading and velocity (results are affected by flow distortion!) | 0 | 100 | | WD2 | degree | Wind direction (from) computed computed from port sensor readings and the ships heading and velocity (results are affected by flow distortion!) (clockwise from North E.g. +90 denotes wind coming from East) | 0 | 360 | | sol_altitude | degrees | Solar altitude, with zero at the horizon and positive when the sun is above the horizon. | -90 | 90 | | sol_azimuth | degrees | Azimuth is reckoned with zero corresponding to north. Positive azimuth estimates correspond to estimates east of north; negative estimates, or estimates larger than 180 are west of north. | 0 | 360 | | | | | | | | SR1_shadow_mask | Boolean | Flag indicating if SR1 (port side sensor) is likely shadowed by the ship’s structure (don’t use SR1 if the mask is True) | FALSE | TRUE | | SR3_shadow_mask | Boolean | Flag indicating if SR3 is likely shadowed by the ship’s structure (don’t use SR3 if the mask is True) | FALSE | TRUE | | TATWDPRH1_heat_island_mask | Boolean | Flag indicating if the port sensor readings (TA1, RH1, DP1, and TW1) are likely affected by heating of the air passing over the ship (the readings are likely affected if the mask is true). | FALSE | TRUE | | TATWDPRH2_heat_island_mask | Boolean | Flag indicating if the starboard sensor readings (TA2, RH2, DP2, and TW2) are likely affected by heating of the air passing over the ship (the readings are likely affected if the mask is true). | FALSE | TRUE | | TATWDPRH1_relhum_mask | Boolean | Flag indicating if the port sensor readings do not fulfill the conditions (DP1 <= TW1 <= TA1) and RH1<=100% (if the mask is true the readings are assumed to be physically not correct). | FALSE | TRUE | | TATWDPRH2_relhum_mask | Boolean | Flag indicating if the starboard sensor readings do not fulfill the conditions (DP2 <= TW2 <= TA2) and RH2<=100% (if the mask is true the readings are assumed to be physically not correct). | FALSE | TRUE | ***** Quality checking ***** The data where thoroughly quality checked using a variety of methods as described below. Observations outside of the ranges stated in table 1 were set to NaN and not included in the averaged dataset. For most cases only very few or none of the observations were removed by the range filter. Exceptions are: - The relative humidity readings of RH1 and RH2 were higher than 100 % for 3 % and 1 % of the samples respectively. The large and continuous number of RH values above 100% may indicate an inaccurate calibration of the unit. Therefore values up to RH=103% were not removed, but a flag was added to the dataset to allow masking of these data. A negligible fraction of data points with RH>103% have been removed prior to resampling. - WSR, WDR, for which less than 0.2 % of the data were outside of the plausible range for wind speed and direction. For VIS (horizontal visibility) 28 observations were above the upper limit of 20,000 meters, but matched the time series perfectly when divided by a factor of 10. Therefore these 28 observations were not removed but adjusted. Other values have been observed to differ from those nearby in the timeseries, by factors of 10 or 100, but these have not been corrected or flagged. The user should take care when using this parameter and take into account nearby datapoints. The following consistency checks between variables were used: - For the three cloud level readings, samples of CL1, CL2, and CL3, as well as SC1, SC2, and SC3 were removed if the condition CL1 < CL2 < CL3 was not satisfied or if any of the variables was outside the specified range (0.07 % of the observations). - Consistency between the port and starboard wind speed and direction readings was not used as criterion because readings from both sensors are affected by flow distortion. Accounting for variability with the relative wind direction, the observations from both sensors are consistent. The raw temperature readings show sudden events of elevated temperature (readings higher than long-term average by 10 to 30 degrees Kelvin) with a corresponding drop in the relative humidity (by several 10 %). This mostly happens for TA2/RH2 (without corresponding changes in the time series of TA1/RH1). In fewer cases, TA1/RH1 changed suddenly while TA2/RH2 varied smoothly in time. The events do not depend on the relative wind direction. These events are interpreted as artefacts, potentially venting of hot air from inside the ship that may have a very localised effect on the air temperature close to the temperature-humidity probes. A 60-minute rolling interquartile range (IQR) filter was used to automatically detect these events. Values that deviate from the median by more than 1.5*IQR (and more than twice the digital resolution of the signal) are considered as spikes. Based on this method spikes in TA, DP and RH were detected for 1% and 3% of the data for the port (1) and starboard (2) sensors respectively. The filtering was always done in the following order: 1. If a plausible range was used to remove data points this was applied first. 2. Consistency between linked observations was checked as described below. 3. For some variables an interquartile range filter was used to detect outliers in the time series. 4. For some variables data were removed (set to NaN) based on visual inspection of the time series. This was done only for a very limited number of occasions (see Table 2). No filtering was applied to variables UV1 and UV2. Both values are only sparsely populated (99.5% and 97.5% missing values) and have a resolution of 1.0 and show a variation between -2 and 31 that correlates strongly with the absolute latitude of the ship's position but have no obvious correlation with solar radiation or time of day (1% of the readings are lower than -2). The UV2 time series contains values that deviate from this pattern and range up to 100. Table 2: Periods where data was removed based on visual inspection. | Start of period | End of the period | Variables removed | Comment | |---------------------|---------------------|--------------------|---------| | 2016-12-20 20:02:00 | 2016-12-20 22:30:00 | PA1 | *p1 | | 2017-02-24 00:00:00 | 2017-02-24 00:05:00 | TA2, RH2, DP2, TW2 | *t1 | | 2017-03-19 02:32:00 | 2017-03-19 02:44:00 | TA2, RH2, DP2, TW2 | *t1 | | 2017-03-20 00:00:00 | 2017-03-20 00:10:00 | TA2, RH2, DP2, TW2 | *t1 | | 2017-04-02 17:24:00 | 2017-04-02 17:35:00 | TA2, RH2, DP2, TW2 | *t1 | | 2017-04-02 20:53:00 | 2017-04-02 21:04:00 | TA2, RH2, DP2, TW2 | *t1 | | 2017-04-05 06:00:00 | 2017-04-05 12:00:00 | WSR2, WDR2 | *w1 | Comments: *p1 = PA1 readings are suddenly 6 mbar higher and very variable, while PA2 readings are consistent with the readings of both sensors before and after this period. *t1 = TA2 suddenly reads much higher values but the interquartile range filter did not catch the event. *w1 = WSR2 readings are suddenly much lower for a six-hour period, while WSR1 time series does not show a corresponding variation (WDR is approximately 0 so both sensors should read roughly similar values). ***** Further information for interpreting the data and using the dataset ***** The relative wind speed and direction measurements are affected by air-flow distortion with errors in the relative wind speed ranging between -20% to +20% and errors in the relative wind direction between -10 degrees to +10 degrees (all values preliminary). The resulting relative error in true wind speed can be magnified or reduced depending on the orientation of the wind vector and the ship’s velocity. Therefore errors in the true wind speed can reach between -50% to +100% but are typically between -5% to +35% (all values preliminary). A paper to quantify and correct the bias is in the making and the corrected wind speeds will be published and these corrections will subsequently be applied to this dataset in an update. The solar radiation readings in this data set appear to be affected by shadowing of the radiation sensors due to the ship's structure (main mast and radio shack). Based on the data analysis the following simple filtering the authors recommend to exclude data, where the radiation sensor is shadowed by the ship: For solar_altitude < 50, the data from SR1 should not be used if the relative azimuth is between 0 to 150 degrees. For solar_altitude < 60, the data from SR1 should not be used if the relative azimuth is between 330 to 360 degrees. For solar_altitude < 75, the data from SR1 should not be used if the relative azimuth is between 60 to 120 degrees. For solar_altitude < 50, the data from SR3 should not be used if the relative azimuth is between 220 to 330 degrees. For solar_altitude < 75, the data from SR3 should not be used if the relative azimuth is between 240 to 320 degrees. where the relative azimuth denotes the difference (sol_azimuth - HEADING). The relative azimuth is zero degrees if the ship is heading into the sun and 90 degrees if the sun is on starboard. These criteria have been used to create the data masks SR1_shadow_mask and SR2_shadow_mask that can be used to filter likely shadowed observations (if mask is True). After the application of the masks to SR1 and SR3 some instances of large differences between the radiation readings remain. These occur sporadically and may be related to the passing of clouds. The air temperature and humidity measurements appear to be affected by the ship acting as a heat island, where the lee side sensor reads up to two-degree Centigrade higher air temperature than the windward side sensor. The largest differences occur during daytime and high solar insolation. Based on the observations the authors recommend using the windward-side values of RH, TA, TW, and DP, which should be much less affected by the presence of the ship than the lee-side values. The provided data masks (temprh1_mask, temprh2_mask) can be used to filter the data. Please note that for bow-on (and beam-on) wind direction both sensors might be affected by the heat-island effect in the same way. For each pair of air temperature, wet bulb temperature, relative humidity, and dew point (TA, TW, RH, and DP) the flag TATWDPRHn_relhum_mask was created and is set to True if the condition DP <= TW <= TA was not fulfilled or RH>100%. This is the case for approximately 5% of the data. It should be noted that potential inaccuracies in the calibration of the temperature and relative humidity readings may cause a false removal of data by this mask. The wet bulb temperature (Tw) was calculated from TA and RH following Stull 2011. Timescales - The timescales affecting the observations are very variable. Solar radiation and air temperature are affected by the diurnal cycle but also by cloud cover and passing frontal systems. Further there is a temperature gradient with latitude. Wind speed and thus air sea exchange varies on time scales between minutes and days. Small scale variations (turbulence) may still affect the one-minute observations. Averaging of the wind speed, radiation, temperature and humidity data over time scales between five minutes and three hours will likely produce meaningful results (the vector nature of the wind needs to be accounted for). Interpolation - Depending on the research question, an interpolation over gaps of up to several minutes may be adequate but should be approached carefully. The sky cover and cloud level data should be resampled with care since sudden jumps of the cloud levels occur when clouds form or disappear at one level. The cases with SC1=9 (where CL1 denotes the vertical visibility, rather than the cloud base) requires separate treatment. ***** Dataset contents ***** - ACE_filtered_meteorological_data_1min.csv, data file, comma-separated values - diff_TA1_TA3_WDR2_5min_1.png, metadata, portable network graphics - ratio_SR1_SR3_solangle_5min_1.png, metadata, portable network graphics - data_file_header.txt, metadata, text - README.txt, metadata, text - ace_filtered_meteorological_data_change_log.txt, metadata, text ***** Dataset contact ***** Sebastian Landwehr, Paul Scherrer Institute, Switzerland. ORCID: 0000-0003-2622-577X. Email: sebastian.landwehr@psi.ch Jenny Thomas, Swiss Polar Institute, Switzerland. ORCID: 0000-0002-5986-7026. Email: jenny.thomas@epfl.ch, jt.sciencedata@gmail.com ***** Data license***** This corrected meteorology dataset is made available under the Open Data Commons Attribution License (ODC-By) v1.0 whose full text can be found at https://www.opendatacommons.org/licenses/by/1.0/index.html ***** Dataset citation ***** Please cite this dataset as: Landwehr, S., Thomas, J., Gorodetskaya, I., Thurnherr, I., Robinson, C. and Schmale, J. 2019. Quality-checked meteorological data from the Southern Ocean collected during the Antarctic Circumnavigation Expedition from December 2016 to April 2017. (Version 1.1) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.3379590