metobs_toolkit.dataset.Dataset.buddy_check_with_safetynets#

Dataset.buddy_check_with_safetynets(obstype: str = 'temp', spatial_buddy_radius: int | float = 10000, safety_net_configs: List[Dict] = None, min_sample_size: int = 4, max_alt_diff: int | float | None = None, min_std: int | float = 1.0, spatial_z_threshold: int | float = 3.1, N_iter: int = 2, instantaneous_tolerance: str | Timedelta = Timedelta('0 days 00:04:00'), lapserate: float | None = None, whiteset: WhiteSet = WhiteSet(empty), use_mp: bool = True)[source]#

Spatial buddy check with configurable safety nets.

The buddy check compares an observation against its neighbors (i.e. spatial buddies). The check loops over all the groups, which are stations within a radius of each other. For each group, the z-value of the reference observation is computed given the sample of spatial buddies. If one (or more) exceeds the spatial_z_threshold, the most extreme (=baddest) observation of that group is labeled as an outlier.

Multiple iterations of this check can be done using N_iter.

Optionally, one or more safety nets can be applied. A safety net tests potential outliers against a sample of stations that share a categorical attribute (e.g., LCZ, network). If the z-value computed using the safety net sample is below the specified threshold, the outlier is “saved” and removed from the outlier set for the current iteration.

Safety nets are applied in the order they are specified in safety_net_configs, allowing for multi-level filtering (e.g., first test against LCZ buddies, then against network buddies).

A schematic step-by-step description of the buddy check:

A distance matrix is constructed for all interdistances between the stations. This is done using the haversine approximation.
Groups of spatial buddies (neighbours) are created by using the spatial_buddy_radius. These groups are further filtered by:
- removing stations from the groups that differ to much in altitude (based on the max_alt_diff)
- removing groups of buddies that are too small (based on the min_sample_size)
Observations per group are synchronized in time (using the instantaneous_tolerance for allignment).
If a lapsrate is specified, the observations are corrected for altitude differences.
The following steps are repeated for N-iter iterations:
1. The values of outliers flagged by a previous iteration are converted to NaN’s. Therefore they are not used in any following score or sample.
2. For each buddy group:
  - The mean, standard deviation (std), and sample size are computed.
  - If the std is lower than the minimum_std, it is replaced by the minimum std.
  - Chi values are calculated for all records.
  - For each timestamp the record with the highest Chi is tested if it is larger then spatial_z_threshold. If so, that record is flagged as an outlier. It will be ignored in the next iteration.
3. For each safety net in safety_net_configs (in order):
  - Category buddies (stations sharing the same category value within the specified radius) are identified.
  - The category-buddy sample is tested in size (sample size must be at least min_sample_size). If the condition is not met, the safety net test is not applied.
  - The safety net test is applied:
    - The mean and std are computed of the category-buddy sample. If the std is smaller than min_std, the latter is used.
    - The z-value is computed for the target record (= flagged outlier).
    - If the z-value is smaller than the safety net’s z_threshold, the tested outlier is “saved” and removed from the set of outliers for the current iteration.
4. If whiteset is provided, any outliers that match the white-listed timestamps are removed from the outlier set for the current iteration. White-listed records participate in all buddy check and safety net calculations but are not flagged as outliers in the final results.
5. If whiteset is provided, any outliers that match the white-listed timestamps are removed from the outlier set for the current iteration. White-listed records participate in all buddy check and safety net calculations but are not flagged as outliers in the final results.

Parameters:

obstype (str, optional) – The target observation to check. Default is “temp”.
spatial_buddy_radius (int or float, optional) – The radius to define spatial neighbors in meters. Default is 10000.
safety_net_configs (list of dict, optional) –
List of safety net configurations to apply in order. Each dict must contain:
- ’category’: str, the metadata column name to group by (e.g., ‘LCZ’, ‘network’)
- ’buddy_radius’: int or float, maximum distance for category buddies (in meters)
- ’z_threshold’: int or float, z-value threshold for saving outliers
- ’min_sample_size’: int, minimum number of buddies required for the safety net test
Example:
```
safety_net_configs = [
    {
        "category": "LCZ",
        "buddy_radius": 40000,
        "z_threshold": 2.1,
        "min_sample_size": 4
    },
    {
        "category": "network",
        "buddy_radius": 50000,
        "z_threshold": 2.5,
        "min_sample_size": 3
    }
]
```
The default is None.
min_sample_size (int, optional) – The minimum sample size to calculate statistics on. Used for spatial-buddy samples. Default is 4.
max_alt_diff (int or float or None, optional) – The maximum altitude difference allowed for buddies. Default is None.
min_std (int or float, optional) – The minimum standard deviation for sample statistics. This is used in spatial and safety net samples. Default is 1.0.
spatial_z_threshold (int or float, optional) – The threshold, tested with z-scores, for flagging observations as outliers. Default is 3.1.
N_iter (int, optional) – The number of iterations to perform the buddy check. Default is 2.
instantaneous_tolerance (str or pd.Timedelta, optional) – The maximum time difference allowed for synchronizing observations. Default is pd.Timedelta(“4min”).
lapserate (float or None, optional) – Describe how the obstype changes with altitude (in meters). Default is None.
whiteset (WhiteSet, optional) – A WhiteSet instance containing timestamps that should be excluded from outlier detection. The WhiteSet is used to create station-specific and obstype-specific whitelists before applying the buddy check. White-listed records participate in all buddy check and safety net iterations as regular records but are not flagged as outliers in the final results. The default is an empty WhiteSet().
use_mp (bool, optional) – Use multiprocessing to speed up the buddy check. Default is True.

Return type:

None

Notes

This method modifies the outliers in place and does not return anything. You can use the outliersdf property to view all flagged outliers.
The altitude of the stations can be extracted from GEE by using the Dataset.get_altitude() method.
The LCZ of the stations can be extracted from GEE by using the Dataset.get_LCZ() method.
White-listed records participate in all buddy check and safety net calculations but are not flagged as outliers in the final results.