Met Office Hadley Centre observations datasets
Home > HadIOD > 

HadIOD SST reference data (SIRDS)


Announcements:

None at this time

Introduction

The Sea Surface Temperature (SST) reference data extracted from HadIOD.1.2.0.0 (hereafter ‘reference data’) were initially created for the ESA SST CCI II project (European Space Agency Sea Surface Temperature Climate Change Initiative Phase 2, https://climate.esa.int/en/projects/sea-surface-temperature) for use in validating satellite SST data products. The reference data coverage was then extended for C3S (European Union Copernicus Climate Change Service, https://climate.copernicus.eu/, C3S_312a Lot 3). The data are officially referred to as the SIRDS (SST CCI Independent Reference Data Set) reference data.

SST reference data were initially made available to project users but have subsequently been released to users outside the project on request. These webpages provide information for those interested in this dataset. Please contact us if you are interested in using these data.

File description

Reference data files are in NetCDF format, the current version is v1.0. Data cover 1978-present with new data added with approximately a few months delay. Each NetCDF file contains monthly SST data from a particular platform type. Platforms selected for the reference dataset are: drifters, voluntary observing ships (ship), Global Tropical Moored Buoy Array (GTMBA), mooring data (excluding GTMBA), bottles, Conductivity-Temperature-Depth casts (CTDs), Mechanical BathyThermographs (MBTs), eXpendable BathyThermographs (XBTs), Argo floats and animal data (these are data from instrumented marine animals, e.g. elephant seals).

Files are named SSTCCI2_refdata_PLAT-TYPE_YYYYMM.nc, where YYYY=year, MM=month and PLAT-TYPE=platform type, e.g.:

Each file contains the observed data along with HadIOD metadata including bias corrections, measurement uncertainties and quality control decisions for each observation. The files contain several different bias correction and uncertainty variables, it is not possible to specify values for all of these variables for every observation and unspecified values are set to missing data. At a minimum we try to provide a quality flag and some estimate of uncertainty in each measurement.

For detailed information see 'File contents' below.

Note that files may not be available for every platform type each month because of changes in the usage of different platform types over time. Plots of the number of observations in the files per month for each platform type are shown in 'Figures' below.

User guide and supplementary files

Information about the reference data files is provided on these webpages. We would also strongly recommend reading the product user guide available for the HadIOD files which is of relevance here, in particular Section 2 which gives useful general information about HadIOD data and, importantly, describes the HadIOD observation error model and how to work with the bias corrections and uncertainties provided by HadIOD. Because the names used for the bias correction and uncertainty variables in the reference data files differ from the variable names used in the product user guide, the 'File Contents' section below includes a table showing which variable names are equivalent.

The bias corrections used in the reference data files are the same as those used for the HadIOD 'main' files. The HadIOD dataset also includes supplementary files that provide additional sets of bias corrections to help users explore the bias correction uncertainty. The component of uncertainty explored by these corrections is increasingly important on longer space and time scales (e.g. if concerned with decadal trends). The use of these additional corrections is encouraged and is described in the product user guide. These supplementary files could also be used with the reference data files and are linkable via the OB_ID variable.

Please contact us if you need help.

Data sources

Observations from several different sources have been ingested into the HadIOD.1.2.0.0 database. The following sources are used in the reference data files:

Terms & Conditions / Citing the data

The SST reference data were extracted from HadIOD.1.2.0.0 for the ESA SST CCI II project and later C3S (C3S_312a Lot 3) and are officially referred to as the SIRDS (SST CCI Independent Reference Data Set) reference data. If publishing work using SIRDS reference data, please cite the data as follows:

If applicable, for traceability, please describe which HadIOD bias corrections were used. This information can be found in the file global attribute ‘source’. This is particularly important if any additional corrections from supplementary files were used. See also 'how to cite the data' in the HadIOD product user guide for more information and references for bias corrections. For example:

HadIOD is subject to Crown copyright protection and is provided under the Open Government License v3.

Users must also acknowledge the following:

Contact us

Please contact us if you need any assistance.

File contents

File contents are summarised in the tables below.

Some platform type specific file information:

Reference Data files

All file variables are 1D arrays with dimension N_OBS.


NetCDF variable Description
Dimensions
N_OBS Number of observations in the file.
STRINGID Maximum length in characters of the PLAT_ID used to identify a platform.
Variables
OB_ID HadIOD database identifier for an observation. This is an integer that uniquely identifies an observation within its year and month. When combined with year and month the identifier is unique within the version of HadIOD used (e.g. HadIOD.1.2.0.0).
COLLECTION Code to identify the source of the data in HadIOD.
SUBCOL1 Code to identify sub-collections within collections.
SUBCOL2 Code to identify sub-collections within collections.
PROF_ID HadIOD database identifier for a profile. This is an integer that uniquely identifies a profile within its year and month. All observations from the same vertical profile receive the same profile ID (observations from surface-only platforms are treated as a profile with a single observation). When combined with year and month the identifier is unique within the version of HadIOD used (e.g. HadIOD.1.2.0.0)
PLAT_ID Identifier for the platform making the observations.
LONGITUDE Longitude of observation.
LATITUDE Latitude of observation.
YEAR Year of observation.
MONTH Month of observation.
DAY Day of observation.
HOUR Hour of observation.
MINUTE Minute of observation.
SECOND Second of observation.
DEPTH Depth of an observation below the surface (metres).
DEPTH_CORR Depth bias correction. Added to depth to help remove biases in reported depth.
SST Observed sea surface temperature (DegC).
SST_TYPE_CORR Temperature correction for observations made by the reference data platform type (e.g. ship, XBT). Added to observed temperature to help remove biases in the observed quantity.
SST_TYPE_CORR_UNC The uncertainty in the value of SST_TYPE_CORR (std. dev.). Once the correction is applied, this describes the spread of a probability density function from which the (unknown) residual bias is drawn. The error (residual bias) associated with this uncertainty can be correlated with errors in other observations of the same platform type.
SST_PLAT_CORR Temperature correction for observations made by an individual platform (e.g. an individual ship or float), see PLAT_ID. Added to observed temperature to help remove biases in the observed quantity.
SST_PLAT_CORR_UNC The uncertainty in the value of SST_PLAT_CORR (std. dev.). Once the correction is applied, this describes the spread of a probability density function from which the (unknown) residual bias is drawn. The error (residual bias) associated with this uncertainty can be correlated with errors in other observations from the same individual platform.
SST_RAND_UNC The random measurement error uncertainty (std. dev.) associated with the observed temperature. The error associated with this uncertainty is not correlated with errors in other observations.
SST_COMB_UNC The uncertainty obtained by combining all other temperature uncertainty terms in quadrature (std. dev.). Valid when corrections have been applied.
QC1 Observation quality flag 1.
QC2 Observation quality flag 2.
Global Attributes
description Description of file contents including reference data version number.
source Gives more details about the source of the data in the file, e.g. HadIOD version number and which bias correction schemes were used.
usage Gives any file-specific rules that must be followed when using the data (e.g. how bias corrections should be applied).
history States when the file was created.

Variables explained

Further useful information about some of the variables is given below.


'COLLECTION', 'SUBCOL1' and 'SUBCOL2' variable values

The content of the SUBCOL1 and SUBCOL2 variables varies dependent on the value of the COLLECTION variable. This is detailed in the table below.


Source COLLECTION SUBCOL1 SUBCOL2
ICOADS 1 ICOADS Deck ID* ICOADS Source ID*
EN.4.2.0,

EN.4.2.1,

EN.4.2.2

2, 4, 8 1=WOD13,

2=Argo GDAC,

3=GTSPP**

N/A
MetDB 3 N/A N/A
PMEL GTMBA 5 1=TAO/TRITON,

2=PIRATA,

3=RAMA

PMEL GTMBA SST source code***
CMEMS-Coriolis 7 N/A N/A

*See Table AII and AIII in [Woodruff et al., 2011]

**EN4 blends observations from multiple sources (see 'Data Sources').

***PMEL SST source codes:

  • 0 - No sensor, no data
  • 1 - Real time (telemetered mode)
  • 2 - Derived from real time
  • 3 - Temporally interpolated from real time
  • 4 - Source code inactive at present
  • 5 - Recovered from instrument RAM (delayed mode)
  • 6 - Derived from RAM
  • 7 - Temporally interpolated from RAM

QC Schemes

The content of the quality control variables QC1 and QC2 varies dependent on file platform type. This is detailed in the table below. It is recommended that users omit observations that fail QC. In the case of PMEL GTMBA data, users should filter based on which QC values seem most appropriate (see footnotes below).


Type QC1 QC2
drifter, ship Basic Met Office Hadley Centre QC*: pass=0, fail=1 SST tracking QC**: pass=0, fail=1
CMEMS-Coriolis drifter Improved Met Office Hadley Centre QC***: pass=0, fail=1 N/A
mooring Basic Met Office Hadley Centre QC*: pass=0, fail=1 N/A
CTD, XBT, MBT, bottle, animal Argo, Argo-surface Basic EN4 QC****: pass=0, fail>=1 N/A

GTMBA^* (COLLECTION=1 or 3) Basic Met Office Hadley Centre QC*: pass=0, fail=1 N/A
GTMBA^* (COLLECTION=5) PMEL SST flag^** PMEL position flag^***

*The Met Office Hadley Centre QC suite used for the ICOADS observations comprises basic sanity checks, position track check, climatology check and buddy check. If any of the individual checks fail, the QC flag is set to fail.

**The SST track check used for ICOADS is an updated version of the QC checks described in Atkinson et al. [2013]. SSTs from individual drifters and ships are tracked against a satellite-based reference and persistently poor quality observations are flagged. Position reports are used to flag buoys that have run aground or been picked-up. The coverage of the check is 1986-2015.

***An updated version of the Met Office Hadley Centre QC suite (MDS4) became available for the addition of CMEMS observations into HadIOD. This performs a similar suite of checks to the previous suite (MDS3) used for the ICOADS observations(*) but with improvements to the performance of checks such as the track, climatology and buddy checks. A repeated value check was also added.

****Profile observations have been QC'd by the EN4 system which has many individual checks (see EN4 webpages). Observations are flagged as failing if they have failed any of the EN4 checks. For profile reference data types, only observations passing QC are included in the files, with the exception of Argo-surface reference data which includes observations both passing and failing QC.

^*GTMBA files contain a blend of data from ICOADS (COLLECTION ID=1) and PMEL (COLLECTION ID=5). In November 2016, MetDB data (COLLECTION ID=3) are used as a substitute for ICOADS data.

^**PMEL SST flags:

  • 0 - datum missing
  • 1 - highest quality; pre/post-deployment calibrations agree to within sensor specifications. In most cases only pre-deployment calibrations have been applied
  • 2 - default quality; pre-deployment calibrations applied. Default value for sensors presently deployed and for sensors which were either not recovered or not calibratable when recovered.
  • 3 - adjusted data; pre/post calibrations differ, or original data do not agree with other data sources (e.g., other in situ data or climatology), or original data are noisy. Data have been adjusted in an attempt to reduce the error.
  • 4 - lower quality; pre/post calibrations differ, or data do not agree with other data sources (e.g. other in situ data or climatology), or data are noisy. Data could not be confidently adjusted to correct for error.
  • 5 - sensor or tube failed

^***PMEL position flags:

  • 0 - datum missing (nominal position added)
  • 1 - highest quality; Pre/post-deployment calibrations agree to within sensor specifications. In most cases only pre-deployment calibrations have been applied.
  • 2 - default quality; Pre-deployment calibrations applied. Default value for sensors presently deployed and for sensors which were either not recovered or not calibratable when recovered.
  • 3 - adjusted data; Pre/post calibrations differ, or original data do not agree with other data sources (e.g., other in situ data or climatology), or original data are noisy. Data have been adjusted in an attempt to reduce the error.
  • 4 - lower quality; Pre/post calibrations differ, or data do not agree with other data sources (e.g., other in situ data or climatology), or data are noisy. Data could not be confidently adjusted to correct for error.
  • 5 - sensor or tube failed
  • 8 - has moved from deployed location
  • 9 - daily drift speed exceeds .1 kts (0.05 m/s)
  • >=10 - missing value(s) in-filled by linear interpolation of neighbouring values (gaps up to and including 10 days); flag value is created by taking the lowest quality indicator of the neighbouring flag values used for interpolation and adding 10 (e.g. if the two neighbouring flag values are 2 [default quality] and 9 [drift speed exceeds .1 kts] then the flag for interpolated values will be 9 + 10 = 19.

Bias correction and uncertainty variables

Bias corrections should be added to the corresponding observations, e.g.:

  • SST_CORRECTED = SST + SST_TYPE_CORR + SST_PLAT_CORR
  • DEPTH_CORRECTED = DEPTH + DEPTH_CORR

Further rules on applying the correctons may be given in a file's 'usage' global attribute. Do read this. For example, the 'usage' may state that "DEPTH_CORR and SST_TYPE_CORR must be applied together (and not independently)" i.e. if applying SST_TYPE_CORR to SST, then DEPTH_CORR must also be applied to DEPTH (this might be because, for example, SST corrections are made relative to some reference level).

The reference data files use the same HadIOD observation error model as the HadIOD files. The HadIOD product user guide explains the HadIOD error model and how to work with the bias corrections and uncertainties provided by HadIOD. Reading Section 2 of the user guide is strongly recommended. Because the names used for the bias correction and uncertainty variables in the reference data files differ from the variable names used in the product user guide, the table below lists which variable names are equivalent.


Reference data variable HadIOD file variable
SST_TYPE_CORR temp_type_corr
SST_TYPE_CORR_UNC temp_type_corr_unc
SST_PLAT_CORR temp_plat_corr
SST_PLAT_CORR_UNC temp_plat_corr_unc

SST_RAND_UNC

temp_rand_unc

SST_COMB_UNC temp_comb_unc

Bias corrections and their uncertainties are not provided for all platform types (see the product user guide). For example, SST_TYPE_CORR corrections are only provided for ship, XBT and MBT instrument types and SST_TYPE_CORR_UNC is only provided for ships. At a minimum we try to provide some estimate of uncertainty in each measurement.

Figures

Missing
Missing

Monthly counts of the number of SST observations in the reference data files. Period shown is January 1978-June 2021. (Top) number of observations from surface-based platforms (these are not filtered for observations passing QC); for GTMBA, PMEL-only (gtmba_pmel) and PMEL+ICOADS (gtmba_all) are shown; for drifting buoys, ‘drifter’ and ‘drifter_cmems’ refer to ICOADS and CMEMS-Coriolis data sources respectively (note their divergence around mid-2016 - from 2018 onwards the patchy 'drifter' coverage is due to an internal change in HadIOD's processing). (Bottom) number of profiles from sub-surface profiling platforms (the shallowest observation passing QC in the depth range 4-6m for each profile is included in the file i.e. one observation per profile, except surface Argo where all observations in the depth range 0-10m are included). Note the difference in scale between the two plots. The drop in the number of mooring (excluding GTMBA) observations in November 2016 is due to a change of data source (from ICOADS to MetDB).


Commercial and media enquiries

You can access the Met Office Customer Centre, any time of the day or night by phone, fax or e-mail. Trained staff will help you find the information or products that are right for you.
Contact the Met Office Customer Centre