metrics

class mfai.pytorch.metrics.CSINeighborhood(num_neighbors, task, num_classes=0, average='macro')

Bases: Metric

Compute Critical Sucess Index (or Threat Score) over a neighborhood to avoid the phenomenon of double penalty. So a forecast is considered as a true positive if there is a positive observation in the neighborood (define here by the number of neighbors num_ngb) of a positive prediction.

For further information on the CSI: https://resources.eumetrain.org/data/4/451/english/msg/ver_categ_forec/uos2/uos2_ko4.htm

Parameters:

• num_neighbors (int)

• task (Literal['binary', 'multiclass', 'multilabel'])

• num_classes (int)

• average (Literal['macro', False] | None)

binary_dilation_(input_tensor)

Performs IN_PLACE binary dilation of input_tensor Tensor. input_tensor is assumed to be an implicit single channel tensor of shape (MINIBATCH, W, H).

Return type:

Tensor

Parameters:

input_tensor (Tensor)

compute()

Override this method to compute the final metric value.

This method will automatically synchronize state variables when running in distributed backend.

Return type:

Tensor

update(preds, targets)

preds and targets are Tensors of shape (H, w) or (B,C,H,W). If multiclass, takes int value in [0, nb_output_channels] interval.

Return type:

None

Parameters:

• preds (Tensor)

• targets (Tensor)

class mfai.pytorch.metrics.FAR(*args, **kwargs)

Bases: Metric

False Alarm Rate.

FAR = FP / (TP + FP) = 1 - (TP / (TP + FP)) = 1 - Precision

Parameters:

• args (Any)

• kwargs (Any)

compute()

Override this method to compute the final metric value.

This method will automatically synchronize state variables when running in distributed backend.

Return type:

Tensor

update(*args, **kwargs)

Override this method to update the state variables of your metric class.

Return type:

None

Parameters:

• args (Any)

• kwargs (Any)

class mfai.pytorch.metrics.FNR

Bases: Metric

False Negatives Rate.

FNR = FN / (TP + FN) = 1 - (TP / (TP + FN)) = 1 - Sensitivity torchmetrics.Sensitivity is not available yet so we implement the calculation.

compute()

Override this method to compute the final metric value.

This method will automatically synchronize state variables when running in distributed backend.

Return type:

Tensor

update(preds, target)

Override this method to update the state variables of your metric class.

Return type:

None

Parameters:

• preds (Tensor)

• target (Tensor)

class mfai.pytorch.metrics.FSS(neighborood, thresholds=None, num_classes=None, stride=None, mask=False)

Bases: Metric

Fraction Skill Score.

The FSS is normally computed over a sample of forecast-observation pairs, e.g., at different valid times. Mitter- maier (2021) has demonstrated that the FSS is sensitive to the pooling method used to combine the scores of different forecasts. The two possibilities are to average the FSS of all individual forecast-observation pairs, or to aggregate the FSS components (fractions Brier Score (FBS) and worst possible FBS (WFBS)) before computing an overall score. We use this last method to compute the FSS, called pFSS by Necker, T. & al. (2024).

FSS = 1 - (FBS / WFBS)

References: - Mittermaier, M.P. (2021) A “meta” analysis of the fractions skill score: The limiting case and implications for aggregation. Monthly Weather Review, 149(10), 3491–3504. Available from: https://doi.org/10.1175/MWR-D-18-0106.1 - Necker, T. et al. (2024), The fractions skill score for ensemble forecast verification. Quarterly Journal of the Royal Meteorological Society, 150(764), 4457–4477. Available from: https://doi.org/10.1002/qj.4824

Parameters:

• neighborood (int)

• thresholds (None | int | float | list[int | float])

• num_classes (None | int)

• stride (None | int)

• mask (Literal[False] | ~torch.Tensor)

compute()

Override this method to compute the final metric value.

This method will automatically synchronize state variables when running in distributed backend.

Return type:

Tensor

compute_fbs_wfbs(targets, preds)

Compute the fractions Brier Score (FBS) and the worse fractions Brier Score (WFBS).

Parameters:

• preds (Tensor) – tensor that contains predicted values (converted in classes).

• targets (Tensor) – tensor that contains observed values (converted in classes).

Returns:

the fractions Brier Score and the worse fractions Brier Score.

Return type:

fbs, worse_fbs (tuple(float, float))

static to_category(tensor, thresholds)

Return type:

Tensor

Parameters:

• tensor (Tensor)

• thresholds (list[int | float])

update(preds, targets)

Updates the fbs/wfbs list by adding the current fbs/wfbs.

Parameters:

• preds (Tensor) – tensor that contains predicted values.

• targets (Tensor) – tensor that contains observed values.

Return type:

None

class mfai.pytorch.metrics.PR_AUC(task='binary')

Bases: Metric

Area Under the Precsion-Recall Curve.

Parameters:

task (Literal['binary', 'multiclass', 'multilabel'])

compute()

Override this method to compute the final metric value.

This method will automatically synchronize state variables when running in distributed backend.

Return type:

Tensor

update(preds, targets)

Override this method to update the state variables of your metric class.

Return type:

None

Parameters:

• preds (Tensor)

• targets (Tensor)