Skip to content

Latest commit

 

History

History
713 lines (515 loc) · 43.6 KB

CHANGELOG.md

File metadata and controls

713 lines (515 loc) · 43.6 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note: we move fast, but still we preserve 0.1 version (one feature release) back compatibility.

[UnReleased] - 2022-MM-DD

Added

  • Added a new NLP metric InfoLM (#915)

  • Added Perplexity metric (#922)

Changed

Deprecated

Removed

Fixed

  • Fixed a bug in ssim when return_full_image=True where the score was still reduced (#1204)

  • Fixed MPS support for:

[0.9.3] - 2022-08-22

Added

  • Added global option sync_on_compute to disable automatic synchronization when compute is called (#1107)

Fixed

  • Fixed missing reset in ClasswiseWrapper (#1129)
  • Fixed JaccardIndex multi-label compute (#1125)
  • Fix SSIM propagate device if gaussian_kernel is False, add test (#1149)

[0.9.2] - 2022-06-29

Fixed

  • Fixed mAP calculation for areas with 0 predictions (#1080)
  • Fixed bug where avg precision state and auroc state was not merge when using MetricCollections (#1086)
  • Skip box conversion if no boxes are present in MeanAveragePrecision (#1097)
  • Fixed inconsistency in docs and code when setting average="none" in AvaragePrecision metric (#1116)

[0.9.1] - 2022-06-08

Added

  • Added specific RuntimeError when metric object is on the wrong device (#1056)
  • Added an option to specify own n-gram weights for BLEUScore and SacreBLEUScore instead of using uniform weights only. (#1075)

Fixed

  • Fixed aggregation metrics when input only contains zero (#1070)
  • Fixed TypeError when providing superclass arguments as kwargs (#1069)
  • Fixed bug related to state reference in metric collection when using compute groups (#1076)

[0.9.0] - 2022-05-30

Added

  • Added RetrievalPrecisionRecallCurve and RetrievalRecallAtFixedPrecision to retrieval package (#951)
  • Added class property full_state_update that determines forward should call update once or twice ( #984, #1033)
  • Added support for nested metric collections (#1003)
  • Added Dice to classification package (#1021)
  • Added support to segmentation type segm as IOU for mean average precision (#822)

Changed

  • Renamed reduction argument to average in Jaccard score and added additional options (#874)

Removed

Fixed

  • Fixed non-empty state dict for a few metrics (#1012)
  • Fixed bug when comparing states while finding compute groups (#1022)
  • Fixed torch.double support in stat score metrics (#1023)
  • Fixed FID calculation for non-equal size real and fake input (#1028)
  • Fixed case where KLDivergence could output Nan (#1030)
  • Fixed deterministic for PyTorch<1.8 (#1035)
  • Fixed default value for mdmc_average in Accuracy (#1036)
  • Fixed missing copy of property when using compute groups in MetricCollection (#1052)

[0.8.2] - 2022-05-06

Fixed

  • Fixed multi device aggregation in PearsonCorrCoef (#998)
  • Fixed MAP metric when using custom list of thresholds (#995)
  • Fixed compatibility between compute groups in MetricCollection and prefix/postfix arg (#1007)
  • Fixed compatibility with future Pytorch 1.12 in safe_matmul (#1011, #1014)

[0.8.1] - 2022-04-27

Changed

  • Reimplemented the signal_distortion_ratio metric, which removed the absolute requirement of fast-bss-eval (#964)

Fixed

  • Fixed "Sort currently does not support bool dtype on CUDA" error in MAP for empty preds (#983)
  • Fixed BinnedPrecisionRecallCurve when thresholds argument is not provided (#968)
  • Fixed CalibrationError to work on logit input (#985)

[0.8.0] - 2022-04-14

Added

  • Added WeightedMeanAbsolutePercentageError to regression package (#948)
  • Added new classification metrics:
    • CoverageError (#787)
    • LabelRankingAveragePrecision and LabelRankingLoss (#787)
  • Added new image metric:
    • SpectralAngleMapper (#885)
    • ErrorRelativeGlobalDimensionlessSynthesis (#894)
    • UniversalImageQualityIndex (#824)
    • SpectralDistortionIndex (#873)
  • Added support for MetricCollection in MetricTracker (#718)
  • Added support for 3D image and uniform kernel in StructuralSimilarityIndexMeasure (#818)
  • Added smart update of MetricCollection (#709)
  • Added ClasswiseWrapper for better logging of classification metrics with multiple output values (#832)
  • Added **kwargs argument for passing additional arguments to base class (#833)
  • Added negative ignore_index for the Accuracy metric (#362)
  • Added adaptive_k for the RetrievalPrecision metric (#910)
  • Added reset_real_features argument image quality assessment metrics (#722)
  • Added new keyword argument compute_on_cpu to all metrics (#867)

Changed

  • Made num_classes in jaccard_index a required argument (#853, #914)
  • Added normalizer, tokenizer to ROUGE metric (#838)
  • Improved shape checking of permutation_invariant_training (#864)
  • Allowed reduction None (#891)
  • MetricTracker.best_metric will now give a warning when computing on metric that do not have a best (#913)

Deprecated

  • Deprecated argument compute_on_step (#792)
  • Deprecated passing in dist_sync_on_step, process_group, dist_sync_fn direct argument (#833)

Removed

  • Removed support for versions of Pytorch-Lightning lower than v1.5 (#788)
  • Removed deprecated functions, and warnings in Text (#773)
    • WER and functional.wer
  • Removed deprecated functions and warnings in Image (#796)
    • SSIM and functional.ssim
    • PSNR and functional.psnr
  • Removed deprecated functions, and warnings in classification and regression (#806)
    • FBeta and functional.fbeta
    • F1 and functional.f1
    • Hinge and functional.hinge
    • IoU and functional.iou
    • MatthewsCorrcoef
    • PearsonCorrcoef
    • SpearmanCorrcoef
  • Removed deprecated functions, and warnings in detection and pairwise (#804)
    • MAP and functional.pairwise.manhatten
  • Removed deprecated functions, and warnings in Audio (#805)
    • PESQ and functional.audio.pesq
    • PIT and functional.audio.pit
    • SDR and functional.audio.sdr and functional.audio.si_sdr
    • SNR and functional.audio.snr and functional.audio.si_snr
    • STOI and functional.audio.stoi
  • Removed unused get_num_classes from torchmetrics.utilities.data (#914)

Fixed

  • Fixed device mismatch for MAP metric in specific cases (#950)
  • Improved testing speed (#820)
  • Fixed compatibility of ClasswiseWrapper with the prefix argument of MetricCollection (#843)
  • Fixed BestScore on GPU (#912)
  • Fixed Lsum computation for ROUGEScore (#944)

[0.7.3] - 2022-03-23

Fixed

  • Fixed unsafe log operation in TweedieDeviace for power=1 (#847)
  • Fixed bug in MAP metric related to either no ground truth or no predictions (#884)
  • Fixed ConfusionMatrix, AUROC and AveragePrecision on GPU when running in deterministic mode (#900)
  • Fixed NaN or Inf results returned by signal_distortion_ratio (#899)
  • Fixed memory leak when using update method with tensor where requires_grad=True (#902)

[0.7.2] - 2022-02-10

Fixed

  • Minor patches in JOSS paper.

[0.7.1] - 2022-02-03

Changed

  • Used torch.bucketize in calibration error when torch>1.8 for faster computations (#769)
  • Improve mAP performance (#742)

Fixed

  • Fixed check for available modules (#772)
  • Fixed Matthews correlation coefficient when the denominator is 0 (#781)

[0.7.0] - 2022-01-17

Added

  • Added NLP metrics:
    • MatchErrorRate (#619)
    • WordInfoLost and WordInfoPreserved (#630)
    • SQuAD (#623)
    • CHRFScore (#641)
    • TranslationEditRate (#646)
    • ExtendedEditDistance (#668)
  • Added MultiScaleSSIM into image metrics (#679)
  • Added Signal to Distortion Ratio (SDR) to audio package (#565)
  • Added MinMaxMetric to wrappers (#556)
  • Added ignore_index to retrieval metrics (#676)
  • Added support for multi references in ROUGEScore (#680)
  • Added a default VSCode devcontainer configuration (#621)

Changed

  • Scalar metrics will now consistently have additional dimensions squeezed (#622)
  • Metrics having third party dependencies removed from global import (#463)
  • Untokenized for BLEUScore input stay consistent with all the other text metrics (#640)
  • Arguments reordered for TER, BLEUScore, SacreBLEUScore, CHRFScore now expect input order as predictions first and target second (#696)
  • Changed dtype of metric state from torch.float to torch.long in ConfusionMatrix to accommodate larger values (#715)
  • Unify preds, target input argument's naming across all text metrics (#723, #727)
    • bert, bleu, chrf, sacre_bleu, wip, wil, cer, ter, wer, mer, rouge, squad

Deprecated

  • Renamed IoU -> Jaccard Index (#662)
  • Renamed text WER metric (#714)
    • functional.wer -> functional.word_error_rate
    • WER -> WordErrorRate
  • Renamed correlation coefficient classes: (#710)
    • MatthewsCorrcoef -> MatthewsCorrCoef
    • PearsonCorrcoef -> PearsonCorrCoef
    • SpearmanCorrcoef -> SpearmanCorrCoef
  • Renamed audio STOI metric: (#753, #758)
    • audio.STOI to audio.ShortTimeObjectiveIntelligibility
    • functional.audio.stoi to functional.audio.short_time_objective_intelligibility
  • Renamed audio PESQ metrics: (#751)
    • functional.audio.pesq -> functional.audio.perceptual_evaluation_speech_quality
    • audio.PESQ -> audio.PerceptualEvaluationSpeechQuality
  • Renamed audio SDR metrics: (#711)
    • functional.sdr -> functional.signal_distortion_ratio
    • functional.si_sdr -> functional.scale_invariant_signal_distortion_ratio
    • SDR -> SignalDistortionRatio
    • SI_SDR -> ScaleInvariantSignalDistortionRatio
  • Renamed audio SNR metrics: (#712)
    • functional.snr -> functional.signal_distortion_ratio
    • functional.si_snr -> functional.scale_invariant_signal_noise_ratio
    • SNR -> SignalNoiseRatio
    • SI_SNR -> ScaleInvariantSignalNoiseRatio
  • Renamed F-score metrics: (#731, #740)
    • functional.f1 -> functional.f1_score
    • F1 -> F1Score
    • functional.fbeta -> functional.fbeta_score
    • FBeta -> FBetaScore
  • Renamed Hinge metric: (#734)
    • functional.hinge -> functional.hinge_loss
    • Hinge -> HingeLoss
  • Renamed image PSNR metrics (#732)
    • functional.psnr -> functional.peak_signal_noise_ratio
    • PSNR -> PeakSignalNoiseRatio
  • Renamed image PIT metric: (#737)
    • functional.pit -> functional.permutation_invariant_training
    • PIT -> PermutationInvariantTraining
  • Renamed image SSIM metric: (#747)
    • functional.ssim -> functional.scale_invariant_signal_noise_ratio
    • SSIM -> StructuralSimilarityIndexMeasure
  • Renamed detection MAP to MeanAveragePrecision metric (#754)
  • Renamed Fidelity & LPIPS image metric: (#752)
    • image.FID -> image.FrechetInceptionDistance
    • image.KID -> image.KernelInceptionDistance
    • image.LPIPS -> image.LearnedPerceptualImagePatchSimilarity

Removed

  • Removed embedding_similarity metric (#638)
  • Removed argument concatenate_texts from wer metric (#638)
  • Removed arguments newline_sep and decimal_places from rouge metric (#638)

Fixed

  • Fixed MetricCollection kwargs filtering when no kwargs are present in update signature (#707)

[0.6.2] - 2021-12-15

Fixed

  • Fixed torch.sort currently does not support bool dtype on CUDA (#665)
  • Fixed mAP properly checks if ground truths are empty (#684)
  • Fixed initialization of tensors to be on correct device for MAP metric (#673)

[0.6.1] - 2021-12-06

Changed

  • Migrate MAP metrics from pycocotools to PyTorch (#632)
  • Use torch.topk instead of torch.argsort in retrieval precision for speedup (#627)

Fixed

  • Fix empty predictions in MAP metric (#594, #610, #624)
  • Fix edge case of AUROC with average=weighted on GPU (#606)
  • Fixed forward in compositional metrics (#645)

[0.6.0] - 2021-10-28

Added

  • Added audio metrics:
    • Perceptual Evaluation of Speech Quality (PESQ) (#353)
    • Short-Time Objective Intelligibility (STOI) (#353)
  • Added Information retrieval metrics:
    • RetrievalRPrecision (#577)
    • RetrievalHitRate (#576)
  • Added NLP metrics:
    • SacreBLEUScore (#546)
    • CharErrorRate (#575)
  • Added other metrics:
    • Tweedie Deviance Score (#499)
    • Learned Perceptual Image Patch Similarity (LPIPS) (#431)
  • Added MAP (mean average precision) metric to new detection package (#467)
  • Added support for float targets in nDCG metric (#437)
  • Added average argument to AveragePrecision metric for reducing multi-label and multi-class problems (#477)
  • Added MultioutputWrapper (#510)
  • Added metric sweeping:
    • higher_is_better as constant attribute (#544)
    • higher_is_better to rest of codebase (#584)
  • Added simple aggregation metrics: SumMetric, MeanMetric, CatMetric, MinMetric, MaxMetric (#506)
  • Added pairwise submodule with metrics (#553)
    • pairwise_cosine_similarity
    • pairwise_euclidean_distance
    • pairwise_linear_similarity
    • pairwise_manhatten_distance

Changed

  • AveragePrecision will now as default output the macro average for multilabel and multiclass problems (#477)
  • half, double, float will no longer change the dtype of the metric states. Use metric.set_dtype instead (#493)
  • Renamed AverageMeter to MeanMetric (#506)
  • Changed is_differentiable from property to a constant attribute (#551)
  • ROC and AUROC will no longer throw an error when either the positive or negative class is missing. Instead return 0 score and give a warning

Deprecated

  • Deprecated functional.self_supervised.embedding_similarity in favour of new pairwise submodule

Removed

  • Removed dtype property (#493)

Fixed

  • Fixed bug in F1 with average='macro' and ignore_index!=None (#495)
  • Fixed bug in pit by using the returned first result to initialize device and type (#533)
  • Fixed SSIM metric using too much memory (#539)
  • Fixed bug where device property was not properly update when metric was a child of a module (#542)

[0.5.1] - 2021-08-30

Added

  • Added device and dtype properties (#462)
  • Added TextTester class for robustly testing text metrics (#450)

Changed

  • Added support for float targets in nDCG metric (#437)

Removed

  • Removed rouge-score as dependency for text package (#443)
  • Removed jiwer as dependency for text package (#446)
  • Removed bert-score as dependency for text package (#473)

Fixed

  • Fixed ranking of samples in SpearmanCorrCoef metric (#448)
  • Fixed bug where compositional metrics where unable to sync because of type mismatch (#454)
  • Fixed metric hashing (#478)
  • Fixed BootStrapper metrics not working on GPU (#462)
  • Fixed the semantic ordering of kernel height and width in SSIM metric (#474)

[0.5.0] - 2021-08-09

Added

  • Added Text-related (NLP) metrics:
  • Added MetricTracker wrapper metric for keeping track of the same metric over multiple epochs (#238)
  • Added other metrics:
    • Symmetric Mean Absolute Percentage error (SMAPE) (#375)
    • Calibration error (#394)
    • Permutation Invariant Training (PIT) (#384)
  • Added support in nDCG metric for target with values larger than 1 (#349)
  • Added support for negative targets in nDCG metric (#378)
  • Added None as reduction option in CosineSimilarity metric (#400)
  • Allowed passing labels in (n_samples, n_classes) to AveragePrecision (#386)

Changed

  • Moved psnr and ssim from functional.regression.* to functional.image.* (#382)
  • Moved image_gradient from functional.image_gradients to functional.image.gradients (#381)
  • Moved R2Score from regression.r2score to regression.r2 (#371)
  • Pearson metric now only store 6 statistics instead of all predictions and targets (#380)
  • Use torch.argmax instead of torch.topk when k=1 for better performance (#419)
  • Moved check for number of samples in R2 score to support single sample updating (#426)

Deprecated

  • Rename r2score >> r2_score and kldivergence >> kl_divergence in functional (#371)
  • Moved bleu_score from functional.nlp to functional.text.bleu (#360)

Removed

  • Removed restriction that threshold has to be in (0,1) range to support logit input ( #351 #401)
  • Removed restriction that preds could not be bigger than num_classes to support logit input (#357)
  • Removed module regression.psnr and regression.ssim (#382):
  • Removed (#379):
    • function functional.mean_relative_error
    • num_thresholds argument in BinnedPrecisionRecallCurve

Fixed

  • Fixed bug where classification metrics with average='macro' would lead to wrong result if a class was missing (#303)
  • Fixed weighted, multi-class AUROC computation to allow for 0 observations of some class, as contribution to final AUROC is 0 (#376)
  • Fixed that _forward_cache and _computed attributes are also moved to the correct device if metric is moved (#413)
  • Fixed calculation in IoU metric when using ignore_index argument (#328)

[0.4.1] - 2021-07-05

Changed

Fixed

  • Fixed DDP by is_sync logic to Metric (#339)

[0.4.0] - 2021-06-29

Added

  • Added Image-related metrics:
    • Fréchet inception distance (FID) (#213)
    • Kernel Inception Distance (KID) (#301)
    • Inception Score (#299)
    • KL divergence (#247)
  • Added Audio metrics: SNR, SI_SDR, SI_SNR (#292)
  • Added other metrics:
    • Cosine Similarity (#305)
    • Specificity (#210)
    • Mean Absolute Percentage error (MAPE) (#248)
  • Added add_metrics method to MetricCollection for adding additional metrics after initialization (#221)
  • Added pre-gather reduction in the case of dist_reduce_fx="cat" to reduce communication cost (#217)
  • Added better error message for AUROC when num_classes is not provided for multiclass input (#244)
  • Added support for unnormalized scores (e.g. logits) in Accuracy, Precision, Recall, FBeta, F1, StatScore, Hamming, ConfusionMatrix metrics (#200)
  • Added squared argument to MeanSquaredError for computing RMSE (#249)
  • Added is_differentiable property to ConfusionMatrix, F1, FBeta, Hamming, Hinge, IOU, MatthewsCorrcoef, Precision, Recall, PrecisionRecallCurve, ROC, StatScores (#253)
  • Added sync and sync_context methods for manually controlling when metric states are synced (#302)

Changed

  • Forward cache is reset when reset method is called (#260)
  • Improved per-class metric handling for imbalanced datasets for precision, recall, precision_recall, fbeta, f1, accuracy, and specificity (#204)
  • Decorated torch.jit.unused to MetricCollection forward (#307)
  • Renamed thresholds argument to binned metrics for manually controlling the thresholds (#322)
  • Extend typing (#324, #326, #327)

Deprecated

  • Deprecated functional.mean_relative_error, use functional.mean_absolute_percentage_error (#248)
  • Deprecated num_thresholds argument in BinnedPrecisionRecallCurve (#322)

Removed

  • Removed argument is_multiclass (#319)

Fixed

  • AUC can also support more dimensional inputs when all but one dimension are of size 1 (#242)
  • Fixed dtype of modular metrics after reset has been called (#243)
  • Fixed calculation in matthews_corrcoef to correctly match formula (#321)

[0.3.2] - 2021-05-10

Added

  • Added is_differentiable property:
    • To AUC, AUROC, CohenKappa and AveragePrecision (#178)
    • To PearsonCorrCoef, SpearmanCorrcoef, R2Score and ExplainedVariance (#225)

Changed

  • MetricCollection should return metrics with prefix on items(), keys() (#209)
  • Calling compute before update will now give warning (#164)

Removed

  • Removed numpy as direct dependency (#212)

Fixed

  • Fixed auc calculation and add tests (#197)
  • Fixed loading persisted metric states using load_state_dict() (#202)
  • Fixed PSNR not working with DDP (#214)
  • Fixed metric calculation with unequal batch sizes (#220)
  • Fixed metric concatenation for list states for zero-dim input (#229)
  • Fixed numerical instability in AUROC metric for large input (#230)

[0.3.1] - 2021-04-21

  • Cleaning remaining inconsistency and fix PL develop integration ( #191, #192, #193, #194 )

[0.3.0] - 2021-04-20

Added

  • Added BootStrapper to easily calculate confidence intervals for metrics (#101)
  • Added Binned metrics (#128)
  • Added metrics for Information Retrieval ((PL^5032)):
    • RetrievalMAP (PL^5032)
    • RetrievalMRR (#119)
    • RetrievalPrecision (#139)
    • RetrievalRecall (#146)
    • RetrievalNormalizedDCG (#160)
    • RetrievalFallOut (#161)
  • Added other metrics:
    • CohenKappa (#69)
    • MatthewsCorrcoef (#98)
    • PearsonCorrcoef (#157)
    • SpearmanCorrcoef (#158)
    • Hinge (#120)
  • Added average='micro' as an option in AUROC for multilabel problems (#110)
  • Added multilabel support to ROC metric (#114)
  • Added testing for half precision (#77, #135 )
  • Added AverageMeter for ad-hoc averages of values (#138)
  • Added prefix argument to MetricCollection (#70)
  • Added __getitem__ as metric arithmetic operation (#142)
  • Added property is_differentiable to metrics and test for differentiability (#154)
  • Added support for average, ignore_index and mdmc_average in Accuracy metric (#166)
  • Added postfix arg to MetricCollection (#188)

Changed

  • Changed ExplainedVariance from storing all preds/targets to tracking 5 statistics (#68)
  • Changed behaviour of confusionmatrix for multilabel data to better match multilabel_confusion_matrix from sklearn (#134)
  • Updated FBeta arguments (#111)
  • Changed reset method to use detach.clone() instead of deepcopy when resetting to default (#163)
  • Metrics passed as dict to MetricCollection will now always be in deterministic order (#173)
  • Allowed MetricCollection pass metrics as arguments (#176)

Deprecated

  • Rename argument is_multiclass -> multiclass (#162)

Removed

  • Prune remaining deprecated (#92)

Fixed

  • Fixed when _stable_1d_sort to work when n>=N (PL^6177)
  • Fixed _computed attribute not being correctly reset (#147)
  • Fixed to Blau score (#165)
  • Fixed backwards compatibility for logging with older version of pytorch-lightning (#182)

[0.2.0] - 2021-03-12

Changed

  • Decoupled PL dependency (#13)
  • Refactored functional - mimic the module-like structure: classification, regression, etc. (#16)
  • Refactored utilities - split to topics/submodules (#14)
  • Refactored MetricCollection (#19)

Removed

  • Removed deprecated metrics from PL base (#12, #15)

[0.1.0] - 2021-02-22

  • Added Accuracy metric now generalizes to Top-k accuracy for (multi-dimensional) multi-class inputs using the top_k parameter (PL^4838)
  • Added Accuracy metric now enables the computation of subset accuracy for multi-label or multi-dimensional multi-class inputs with the subset_accuracy parameter (PL^4838)
  • Added HammingDistance metric to compute the hamming distance (loss) (PL^4838)
  • Added StatScores metric to compute the number of true positives, false positives, true negatives and false negatives (PL^4839)
  • Added R2Score metric (PL^5241)
  • Added MetricCollection (PL^4318)
  • Added .clone() method to metrics (PL^4318)
  • Added IoU class interface (PL^4704)
  • The Recall and Precision metrics (and their functional counterparts recall and precision) can now be generalized to Recall@K and Precision@K with the use of top_k parameter (PL^4842)
  • Added compositional metrics (PL^5464)
  • Added AUC/AUROC class interface (PL^5479)
  • Added QuantizationAwareTraining callback (PL^5706)
  • Added ConfusionMatrix class interface (PL^4348)
  • Added multiclass AUROC metric (PL^4236)
  • Added PrecisionRecallCurve, ROC, AveragePrecision class metric (PL^4549)
  • Classification metrics overhaul (PL^4837)
  • Added F1 class metric (PL^4656)
  • Added metrics aggregation in Horovod and fixed early stopping (PL^3775)
  • Added persistent(mode) method to metrics, to enable and disable metric states being added to state_dict (PL^4482)
  • Added unification of regression metrics (PL^4166)
  • Added persistent flag to Metric.add_state (PL^4195)
  • Added classification metrics (PL^4043)
  • Added new Metrics API. (PL^3868, PL^3921)
  • Added EMB similarity (PL^3349)
  • Added SSIM metrics (PL^2671)
  • Added BLEU metrics (PL^2535)