Dataset

The Dataset holds a collection of Stations. All methods applied on a dataset are applied on all present stations (and a target observationtype).

Constructor

Dataset()

Dataset class for managing and processing meteorological observation data.

Data attributes

A summary of all the attributes that hold or return data.

Dataset.stations	Get the list of Stations present in the Dataset.
Dataset.obstypes	Get the dictionary of known Obstypes by the Dataset.
Dataset.template	Get the Template instance used when the data was imported.
Dataset.df	Dataset DataFrame constructor.
Dataset.metadf	Construct a DataFrame representation of metadata.
Dataset.outliersdf	Construct a DataFrame representation of all the outliers.
Dataset.gapsdf	Construct a DataFrame representation of all the gaps.
Dataset.modeldatadf	Construct a DataFrame representation of all the present model data.
Dataset.present_observations	Get a list of all the present observation types.
Dataset.start_datetime	Get the earliest start datetime from the observation data.
Dataset.end_datetime	Get the latest end datetime from the observation data.

General methods and attributes

Dataset.get_info([printout])	Retrieve and optionally print detailed information about the station.
Dataset.subset_by_stations(stationnames[, ...])	Create a subset of the dataset by selecting specific stations.
Dataset.get_station(stationname)	Retrieve a Station by name.
Dataset.rename_stations(renamedict)	Rename stations in the dataset.
Dataset.sync_records([obstype, ...])	Synchronize records of sensor data across stations.
Dataset.resample(target_freq[, obstype, ...])	Resample observation data to a specified frequency.
Dataset.add_new_observationtype(obstype)	Add a new observation type to the dataset known-obstypes.
Dataset.create_distancematrix()	Compute pairwise great-circle distances between all stations.

Exporting and converting methods

Dataset.import_data_from_file(template_file)	Import observational data and metadata from files.
Dataset.save_dataset_to_pkl([filepath, ...])	Save the dataset to a pickle (.pkl) file.
Dataset.to_csv([filepath, overwrite])	Save the dataset observations to a CSV file.
Dataset.to_parquet([filepath, overwrite])	Save the dataset observations to a parquet file.
Dataset.to_xr()	Concatenate multiple station Datasets into one along a new 'name' dimension.
Dataset.to_netcdf([filepath, overwrite])	Save the Dataset as a netCDF file.

GEE related methods

Dataset.modeldatadf	Construct a DataFrame representation of all the present model data.
Dataset.import_gee_data_from_file(filepath, ...)	Import Google Earth Engine (GEE) data from a CSV file
Dataset.get_static_gee_point_data(...[, ...])	Extract static data from GEE dataset at Stations locations.
Dataset.get_static_gee_buffer_fraction_data(...)	Extract circular buffer fractions of a GEE dataset at Stations locations.
Dataset.get_LCZ([update_metadata, ...])	Retrieve Local Climate Zone (LCZ) for the stations using Google Earth Engine (GEE).
Dataset.get_altitude([update_metadata, ...])	Retrieve altitude for the stations using Google Earth Engine (GEE).
Dataset.get_landcover_fractions([buffers, ...])	Get landcover fractions for a circular buffer at the stations using GEE.
Dataset.get_gee_timeseries_data(...[, ...])	Extract time series data from GEE.

Gaps related methods

Dataset.gapsdf	Construct a DataFrame representation of all the gaps.
Dataset.gap_overview_df()	Create gap status overview DataFrame with one row per gap period.
Dataset.convert_outliers_to_gaps([...])	Convert outlier values in the observation data to gaps.
Dataset.interpolate_gaps(obstype[, method, ...])	Fill the gap(s) using interpolation of SensorData.
Dataset.fill_gaps_with_raw_modeldata(obstype)	Fill the gap(s) using model data without correction.
Dataset.fill_gaps_with_debiased_modeldata(obstype)	Fill the gaps using model data corrected for the bias.
Dataset.fill_gaps_with_diurnal_debiased_modeldata(obstype)	Fill the gaps using model data corrected for the diurnal bias.
Dataset.fill_gaps_with_weighted_diurnal_debiased_modeldata(obstype)	Fill the gaps using a weighted sum of model data corrected for the diurnal bias and weights with respect to the start of the gap.

Quality control related methods

Dataset.outliersdf	Construct a DataFrame representation of all the outliers.
Dataset.gross_value_check([obstype, ...])	Identify outliers based on thresholds.
Dataset.persistence_check([obstype, ...])	Check if values are not constant in a moving time window.
Dataset.repetitions_check([obstype, ...])	Test if an observation changes after a number of repetitions.
Dataset.step_check([obstype, ...])	Check for 'spikes' and 'dips' in a time series.
Dataset.window_variation_check([obstype, ...])	Test if the increase/decrease in a time window exceeds a threshold.
Dataset.buddy_check([obstype, ...])	Spatial buddy check.
Dataset.buddy_check_with_safetynets([...])	Spatial buddy check with configurable safety nets.
Dataset.get_qc_stats([obstype, make_plot])	Summarize QC label frequencies across all stations for a given observation type.
Dataset.convert_outliers_to_gaps([...])	Convert outlier values in the observation data to gaps.

Visualisations

Dataset.make_plot_of_modeldata([obstype, ...])	Generate a timeseries plot of model data for a specific observation type.
Dataset.make_plot([obstype, colorby, ...])	Generate a time series plot for observational data.
Dataset.make_gee_plot([gee_manager, ...])	Create an interactive spatial plot of the GEE dataset and stations.

Dataset related functions

import_dataset_from_pkl(target_path)

Import a Dataset instance from a pickle file.

Special methods

The Dataset class implements several Python special methods for convenience:

• __add__: Combine two Dataset objects, merging stations and obstypes.

• __eq__: Test equality between two Dataset objects (compares stations).

• __str__ and __repr__: String representations for printing and debugging.

• copy: Create a (deep) copy of the Dataset.

Example usage:

from metobs_toolkit.dataset import Dataset

# Assume ds1 and ds2 are Dataset instances
ds3 = ds1 + ds2  # Merge datasets

# Equality check
if ds1 == ds2:
    print("Datasets are equal")

# Copying
ds4 = ds1.copy(deep=True)

# String representation
print(str(ds1))