metobs_toolkit.dataset.Dataset.buddy_check#

Dataset.buddy_check(obstype: str = 'temp', spatial_buddy_radius: int | float = 10000, min_sample_size: int = 4, max_sample_size: int | None = None, max_alt_diff: int | float | None = None, min_sample_spread: int | float = 1.0, min_buddy_distance: int | float = 0.0, spatial_z_threshold: int | float = 3.1, N_iter: int = 2, instantaneous_tolerance: str | Timedelta = Timedelta('0 days 00:04:00'), lapserate: float | None = None, whiteset: WhiteSet = WhiteSet(empty), use_z_robust_method: bool = True, use_mp: bool = True, min_std=None)[source]#

Spatial buddy check.

The buddy check compares an observation against its neighbors (i.e. spatial buddies). The check loops over all stations, treating each as the center of a buddy group formed by nearby stations. For each center station, the z-score is computed from the buddy sample. If the z-score exceeds spatial_z_threshold, the center station’s observation is labeled as an outlier.

Multiple iterations of this check can be done using N_iter.

A schematic step-by-step description of the buddy check:

  1. A distance matrix is constructed for all interdistances between the stations. This is done using the haversine approximation.

  2. Groups of spatial buddies (neighbours) are created by using the spatial_buddy_radius and min_buddy_distance. Only stations within the distance range [min_buddy_distance, spatial_buddy_radius] are considered as buddies. These groups are further filtered by:

    • removing stations from the groups that differ too much in altitude (based on the max_alt_diff)

    • removing groups of buddies that are too small (based on the min_sample_size)

  3. Observations per group are synchronized in time (using the instantaneous_tolerance for alignment).

  4. If a lapserate is specified, the observations are corrected for altitude differences.

  5. The following steps are repeated for N_iter iterations:

    1. The values of outliers flagged by a previous iteration are converted to NaN’s. Therefore they are not used in any following score or sample.

    2. For each center station:

      • The sample mean, spread (std or MAD depending on use_z_robust_method), and sample size are computed from the buddy stations (center station excluded).

      • If the spread is lower than min_sample_spread, it is replaced by min_sample_spread.

      • The z-score of the center station is calculated.

      • If the z-score exceeds spatial_z_threshold, the center station’s observation is flagged as an outlier. It will be ignored in the next iteration.

    3. If whiteset is provided, any outliers that match the white-listed timestamps are removed from the outlier set for the current iteration. White-listed records participate in all buddy check calculations but are not flagged as outliers in the final results.

Parameters:

  • obstype (str, optional) – The target observation to check. Default is “temp”.

  • spatial_buddy_radius (int or float, optional) – The radius to define spatial neighbors in meters. Default is 10000.

  • min_sample_size (int, optional) – The minimum sample size to calculate statistics on. Default is 4.

  • max_sample_size (int or None, optional) – The maximum number of spatial buddies to use per station. If not None, the spatial buddies for each station are sorted by distance and only the nearest max_sample_size are kept. Must be larger than min_sample_size when specified. The default is None (no limit).

  • max_alt_diff (int or float or None, optional) – The maximum altitude difference allowed for buddies. Default is None.

  • min_sample_spread (int or float, optional) – The minimum sample spread for sample statistics. When use_z_robust_method is True, this is the minimum MAD to use (avoids division by near-zero). When use_z_robust_method is False, this is the minimum standard deviation. This parameter helps represent the accuracy of the observations. Default is 1.0.

  • min_buddy_distance (int or float, optional) – The minimum distance (in meters) required between a station and its buddies. Stations closer than this distance will be excluded from the buddy group. Default is 0.0 (no minimum distance).

  • spatial_z_threshold (int or float, optional) – The threshold (in z-score units) for flagging observations as outliers. Default is 3.1.

  • N_iter (int, optional) – The number of iterations to perform the buddy check. Default is 2.

  • instantaneous_tolerance (str or pd.Timedelta, optional) – The maximum time difference allowed for synchronizing observations. Default is pd.Timedelta(“4min”).

  • lapserate (float or None, optional) – Describe how the obstype changes with altitude (in meters). Default is None.

  • whiteset (WhiteSet, optional) – A WhiteSet instance containing timestamps that should be excluded from outlier detection. The WhiteSet is used to create station-specific and obstype-specific whitelists before applying the buddy check. White-listed records participate in all buddy check iterations as regular records but are not flagged as outliers in the final results. The default is an empty WhiteSet().

  • use_z_robust_method (bool, optional) – If True, the robust z-score method (median/MAD) is used. If False, the classic z-score method (mean/std) is used. Default is True.

  • use_mp (bool, optional) – Use multiprocessing to speed up the buddy check. Default is True.

Return type:

None

Notes

  • This method modifies the outliers in place and does not return anything. You can use the outliersdf property to view all flagged outliers.

  • The altitude of the stations can be extracted from GEE by using the Dataset.get_altitude() method.

  • White-listed records participate in all buddy check calculations but are not flagged as outliers in the final results.

See also

buddy_check_with_safetynets

Buddy check with configurable safety nets.

Examples

Apply buddy check on temperature observations:

>>> dataset.buddy_check(obstype="temp")

Apply buddy check with custom radius and threshold:

>>> dataset.buddy_check(
...     obstype="temp",
...     spatial_buddy_radius=15000,
...     spatial_z_threshold=2.5,
...     N_iter=3,
... )

Changed in version 1.1.0: Breaking changes compared to v1.0.x:

  • Outlier-selection logic revised. The previous implementation evaluated groups of stations and flagged the single worst observation per group. The new implementation evaluates every station individually as the center of its buddy group, so each station is independently tested against its neighbors. This means the same dataset may produce a different set of outliers than before.

  • Default z-score method changed to robust (median / MAD). Statistics are now computed with the median and Median Absolute Deviation (MAD) by default (use_z_robust_method=True). To reproduce the previous mean / std behaviour set use_z_robust_method=False.

  • min_std renamed to min_sample_spread. Passing min_std now raises a DeprecationWarning.

  • New parameter min_buddy_distance (default 0.0) — excludes stations that are closer than this distance (in metres) from the buddy group. Set to 0.0 to reproduce previous behaviour.

  • New parameter max_sample_size (default None) — caps the number of spatial buddies per station to the nearest N stations. Set to None to reproduce previous behaviour (no cap).

  • New parameter use_z_robust_method — selects between the robust (median/MAD) and classic (mean/std) z-score. Set to False to reproduce the previous mean/std behaviour.

On this page
Show Source