Guides

Benchmarking Metrics

We evaluate explanations on the faithfulness and plausibility properties. Specifically, ferret implements three state-of-the-art metrics to measure faithfulness and three for plausibility [1] [2].

[1]

Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (Jacovi & Goldberg, ACL 2020)

[2]

ERASER: A Benchmark to Evaluate Rationalized NLP Models (DeYoung et al., ACL 2020)