metobs_toolkit.station.Station.window_variation_check#

Station.window_variation_check(obstype: str = 'temp', timewindow: Timedelta = Timedelta('0 days 01:00:00'), min_records_per_window: int = 3, max_increase_per_second: int | float = 0.0022222222222222222, max_decrease_per_second: int | float = -0.002777777777777778, whiteset: WhiteSet = WhiteSet(empty)) → None[source]#

Test if the increase/decrease in a time window exceeds a threshold.

Checks if the variation of observations in time does not exceed a threshold. This is done by applying a moving window over the time series. The moving window is defined by a duration (timewindow), and tested if the window contains at least a minimum number of records.

If the observations in the window increase/decrease more than a threshold, all observations in the window are flagged as outliers. The threshold is defined by the maximum increase/decrease per second multiplied by the window size in seconds.

Parameters:

  • obstype (str, optional) – The target observation to check. By default “temp”

  • timewindow (pandas.Timedelta) – The duration of the moving window. This should be a pandas Timedelta object. The default is pandas.Timedelta(“1h”)

  • min_records_per_window (int) – The minimum number of non-NaN records required within the time window for the check to be valid. This is dependent on the time resolution of the records. The default is 3

  • max_increase_per_second (int or float, >0) – The maximum allowed increase (per second). This value is extrapolated to the window duration. This value must be positive! The default is 8.0/3600

  • max_decrease_per_second (int or float, <0) – The maximum allowed decrease (per second). This value is extrapolated to the window duration. This value must be negative! The default is -10.0/3600

  • whiteset (WhiteSet, optional) – A WhiteSet instance containing timestamps that should be excluded from outlier detection. Records matching the whiteset criteria will not be flagged as outliers. The default is an empty WhiteSet().

Return type:

None

Notes

  • This method modifies the outliers in place and does not return anything. You can use the outliersdf property to view all flagged outliers.

  • In general, for temperatures, the decrease threshold is set less stringent than the increase threshold. This is because a temperature drop is meteorologically more common than a sudden increase which is often the result of a radiation error.

  • A suitable value for the min_records_per_window depends on the time resolution of the records and the window size.

  • This check is similar to the step check, but not identical. The step check tests a maximum allowed increase/decrease with respect to the previous value. The window variation check uses a moving window to test the maximum allowed variation.

On this page
Show Source