Converting MetObs Toolkit data to xarray Datasets#

This notebook demonstrates the to_xr() methods of the Station and Dataset classes from the metobs_toolkit package using the built-in demo dataset.

We show:

  1. Loading the demo dataset.

  2. Converting a single station to an xarray.Dataset.

  3. Inspecting the structure (dimensions, variables, attributes).

  4. Converting the full multi-station dataset to xarray.

  5. Exploring selections (e.g. picking observation values vs. labels).

What is xarray?#

xarray is a Python library that brings the labeled data concepts of pandas to N-dimensional arrays (NetCDF-style). It enables:

  • Named dimensions (e.g. datetime, kind, name)

  • Coordinate-based indexing and selection

  • Rich metadata via attributes

  • Easy export to formats like NetCDF / Zarr / GRIB (with plugins)

It is especially useful for structured time series, gridded data, or any multi-dimensional scientific data.

[2]:

# Imports
import metobs_toolkit
import xarray as xr

[3]:

# 1. Load the demo dataset into a Dataset object
dataset = metobs_toolkit.Dataset()
dataset.import_data_from_file(
    template_file=metobs_toolkit.demo_template,
    input_metadata_file=metobs_toolkit.demo_metadatafile,
    input_data_file=metobs_toolkit.demo_datafile,
)

print(f"Number of stations: {len(dataset.stations)}")
print("First 5 station names:", [s.name for s in dataset.stations[:5]])

Luchtdruk is present in the datafile, but not found in the template! This column will be ignored.
Neerslagintensiteit is present in the datafile, but not found in the template! This column will be ignored.
Neerslagsom is present in the datafile, but not found in the template! This column will be ignored.
Rukwind is present in the datafile, but not found in the template! This column will be ignored.
Luchtdruk_Zeeniveau is present in the datafile, but not found in the template! This column will be ignored.
Globe Temperatuur is present in the datafile, but not found in the template! This column will be ignored.
The following columns are present in the data file, but not in the template! They are skipped!
 ['Luchtdruk_Zeeniveau', 'Rukwind', 'Neerslagsom', 'Globe Temperatuur', 'Luchtdruk', 'Neerslagintensiteit']
The following columns are found in the metadata, but not in the template and are therefore ignored:
['benaming', 'sponsor', 'Network', 'stad']

Number of stations: 28
First 5 station names: ['vlinder01', 'vlinder02', 'vlinder03', 'vlinder04', 'vlinder05']

[4]:

# 2. Pick one station (e.g. 'vlinder05') and run a simple QC check to add labels
station = dataset.get_station('vlinder05')
station.repetitions_check(max_N_repetitions=200)

/home/thoverga/Documents/VLINDER_github/MetObs_toolkit/metobs_toolkit/qc_collection/repetitions_check.py:62: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.
  groups.get_group(

[6]:

# 3. Convert the single station to an xarray Dataset
ds_station = station.to_xr()

ds_station

[6]:

<xarray.Dataset> Size: 311kB
Dimensions:         (datetime: 4320, kind: 2)
Coordinates:
  * datetime        (datetime) datetime64[ns, UTC] 35kB 2022-09-01 00:00:00+0...
  * kind            (kind) <U5 40B 'obs' 'label'
    lat             float64 8B 51.05
    lon             float64 8B 3.675
    altitude        float64 8B nan
    LCZ             float64 8B nan
    school          <U12 48B 'Sint-Barbara'
Data variables:
    wind_direction  (kind, datetime) object 69kB 45.0 45.0 45.0 ... 'ok' 'ok'
    temp            (kind, datetime) object 69kB 21.100000381469727 ... 'repe...
    wind_speed      (kind, datetime) object 69kB 1.6111112833023071 ... 'ok'
    humidity        (kind, datetime) object 69kB 61.0 61.0 61.0 ... 'ok' 'ok'

Structure of the station-level Dataset#

For each observed variable (e.g. temp, humidity, etc.) a DataArray is created with:

  • Dimension kind: separates ‘obs’ (values) and ‘label’ (QC and gap-fill labels)

  • Dimension datetime: corresponding timestamp

Attributes on each variable include:

  • obstype_name, obstype_desc, obstype_unit

  • QC: dictionary of applied quality control checks

  • GF: dictionary of applied gap-fill

[7]:

# 4. Inspect one variable (e.g. temperature)
ds_station['temp']

[7]:

<xarray.DataArray 'temp' (kind: 2, datetime: 4320)> Size: 69kB
array([[21.100000381469727, 21.100000381469727, 21.100000381469727, ...,
        17.399999618530273, 17.399999618530273, 17.399999618530273],
       ['ok', 'ok', 'ok', ..., 'repetitions outlier',
        'repetitions outlier', 'repetitions outlier']],
      shape=(2, 4320), dtype=object)
Coordinates:
  * datetime  (datetime) datetime64[ns, UTC] 35kB 2022-09-01 00:00:00+00:00 ....
  * kind      (kind) <U5 40B 'obs' 'label'
    lat       float64 8B 51.05
    lon       float64 8B 3.675
    altitude  float64 8B nan
    LCZ       float64 8B nan
    school    <U12 48B 'Sint-Barbara'
Attributes:
    obstype_name:  temp
    obstype_desc:  2m - temperature
    obstype_unit:  degree_Celsius
    QC:            {'duplicated_timestamp': {'settings': {}}, 'repetitions': ...
    GF:            {}
[10]:

# 5. Inspect the QC labels (kind='label')
labels = ds_station['temp'].sel(kind='label')
labels

[10]:

<xarray.DataArray 'temp' (datetime: 4320)> Size: 35kB
array(['ok', 'ok', 'ok', ..., 'repetitions outlier',
       'repetitions outlier', 'repetitions outlier'],
      shape=(4320,), dtype=object)
Coordinates:
  * datetime  (datetime) datetime64[ns, UTC] 35kB 2022-09-01 00:00:00+00:00 ....
    kind      <U5 20B 'label'
    lat       float64 8B 51.05
    lon       float64 8B 3.675
    altitude  float64 8B nan
    LCZ       float64 8B nan
    school    <U12 48B 'Sint-Barbara'
Attributes:
    obstype_name:  temp
    obstype_desc:  2m - temperature
    obstype_unit:  degree_Celsius
    QC:            {'duplicated_timestamp': {'settings': {}}, 'repetitions': ...
    GF:            {}
[11]:

#or the observations
records = ds_station['temp'].sel(kind='obs')
records

[11]:

<xarray.DataArray 'temp' (datetime: 4320)> Size: 35kB
array([21.100000381469727, 21.100000381469727, 21.100000381469727, ...,
       17.399999618530273, 17.399999618530273, 17.399999618530273],
      shape=(4320,), dtype=object)
Coordinates:
  * datetime  (datetime) datetime64[ns, UTC] 35kB 2022-09-01 00:00:00+00:00 ....
    kind      <U5 20B 'obs'
    lat       float64 8B 51.05
    lon       float64 8B 3.675
    altitude  float64 8B nan
    LCZ       float64 8B nan
    school    <U12 48B 'Sint-Barbara'
Attributes:
    obstype_name:  temp
    obstype_desc:  2m - temperature
    obstype_unit:  degree_Celsius
    QC:            {'duplicated_timestamp': {'settings': {}}, 'repetitions': ...
    GF:            {}

Converting the full Dataset#

We can also use to_xr() on a Dataset object. Doing so, an extra dimension name is added in the xarray.Dataset.

[12]:

# 6. Convert the entire collection of stations
ds_all = dataset.to_xr()

ds_all

[12]:

<xarray.Dataset> Size: 8MB
Dimensions:         (name: 28, kind: 2, datetime: 4320)
Coordinates:
  * datetime        (datetime) datetime64[ns, UTC] 35kB 2022-09-01 00:00:00+0...
  * kind            (kind) <U5 40B 'obs' 'label'
    lat             (name) float64 224B 50.98 51.02 51.32 ... 51.16 51.06 51.04
    lon             (name) float64 224B 3.816 3.71 4.952 ... 4.998 3.728 3.77
    altitude        float64 8B nan
    LCZ             float64 8B nan
    school          (name) <U29 3kB 'UGent' 'UGent' ... 'GO! Ath.'
  * name            (name) <U9 1kB 'vlinder01' 'vlinder02' ... 'vlinder28'
Data variables:
    wind_direction  (name, kind, datetime) object 2MB 65.0 75.0 ... 'ok' 'ok'
    temp            (name, kind, datetime) object 2MB 18.799999237060547 ... ...
    wind_speed      (name, kind, datetime) object 2MB 1.5555555820465088 ... ...
    humidity        (name, kind, datetime) object 2MB 65.0 65.0 ... 'ok' 'ok'
[13]:

# 7. Selecting a single station from the multi-station Dataset
ds_one = ds_all.sel(name='vlinder05')
ds_one['temp']

[13]:

<xarray.DataArray 'temp' (kind: 2, datetime: 4320)> Size: 69kB
array([[21.100000381469727, 21.100000381469727, 21.100000381469727, ...,
        17.399999618530273, 17.399999618530273, 17.399999618530273],
       ['ok', 'ok', 'ok', ..., 'repetitions outlier',
        'repetitions outlier', 'repetitions outlier']],
      shape=(2, 4320), dtype=object)
Coordinates:
  * datetime  (datetime) datetime64[ns, UTC] 35kB 2022-09-01 00:00:00+00:00 ....
  * kind      (kind) <U5 40B 'obs' 'label'
    lat       float64 8B 51.05
    lon       float64 8B 3.675
    altitude  float64 8B nan
    LCZ       float64 8B nan
    school    <U29 116B 'Sint-Barbara'
    name      <U9 36B 'vlinder05'
Attributes:
    obstype_name:  temp
    obstype_desc:  2m - temperature
    obstype_unit:  degree_Celsius
    QC:            {'duplicated_timestamp': {'settings': {}}}
    GF:            {}

Dimension summary (multi-station)#

  • name: name of the station

  • kind: sub-type of the data (e.g. ‘obs’, ‘label’, possibly ‘model’ if model time series added)

  • datetime: consolidated time axis (union across stations)

If model time series (e.g. ERA5) are imported, an additional internal dimension (e.g. models) appears inside the model DataArrays (stacked under kind='model').

On this page
Show Source