A Scorer is the criteria used to quantitatively evaluate LLM outputs. When you test LLMs with Arthur Bench, you attach a Scorer to each test suite you create - this defines how performance will be measured consistently across that test suite.
For a walkthrough on how to extend the Scorer class to create your own scorer specialized to your data and/or use-case to use with Arthur Bench, check out the custom scoring guide
If you would like to contribute scorers to the open source Arthur Bench repo, check out our contributing guide
Here is a list of all the scorers available by default in Arthur Bench (listed alphabetically):
| Scorer | Tasks | Type | Requirements |
|---|---|---|---|
BERT Score (bertscore) |
any | Embedding-Based | Reference Output, Candidate Output |
Exact Match (exact_match) |
any | Lexicon-Based | Reference Output, Candidate Output |
Hallucination (hallucination) |
any | Prompt-Based | Candidate Output, Context |
Hedging Language (hedging_language) |
any | Embedding-Based | Candidate Output |
Python Unit Testing (python_unit_testing) |
Python Generation | Code Evaluator | Candidate Output, Unit Tests (see the code eval guide) |
QA Correctness (qa_correctness) |
Question-Answering | Prompt-Based | Input, Candidate Output, Context |
Readability (readability) |
any | Lexicon-Based | Candidate Output |
Specificity (specificity) |
any | Lexicon-Based | Candidate Output |
Summary Quality (summary_quality) |
Summarization | Prompt-Based | Input, Reference Output, Candidate Output |
Word Count Match (word_count_match) |
any | Lexicon-Based | Reference Output, Candidate Output |
For better understandability we have broken down the Scorers based on the type of procedure each Scorer uses.
The QA correctness scorer evaluates the correctness of an answer, given a question and context. This scorer does not require a reference output, but does require context. Each row of the Test Run will receive a binary 0, indicating an incorrect output, or 1, indicating a correct output.
The Summary Quality scorer evaluates a summary against its source text and a reference summary for comparison. It evaluates summaries on dimensions including relevance and syntax. Each row of the test run will receive a binary 0, indicating that the reference output was scored higher than the candidate output, or 1, indicating that the candidate output was scored higher than the reference output.
The Hallucination scorer takes a response and a context (e.g. in a RAG setting where context is used to ground an LLM’s responses) and identifies when information in the response is not substantiated by the context . The scorer breaks down the response into a list of claims and checks the claims against the context for support. This binary score is 1 if all claims are supported, and 0 otherwise.
BERTScore is a quantitative metric to compare the similarity of two pieces of text. Using bertscore will score each row of the test run as the bert score between the reference output and the candidate output.
The Hedging Language scorer evaluates whether a candidate response is similar to generic hedging language used by an LLM ("As an AI language model, I don't have personal opinions, emotions, or beliefs"). Each row of the Test Run will receive a score between 0.0 and 1.0 indicating the extent to which hedging language is detected in the response (using BERTScore similarity to the target hedging phrase). A score above 0.5 typically suggests the model output contains hedging language.
The Exact Match scorer evaluates whether the candidate output exactly matches the reference output. This is case sensitive. Each row of the Test Run will receive a binary 0, indicating a non-match, or 1, indicating an exact match.
The Readability scorer evaluates the reading ease of the candidate output according to the Flesch Reading Ease Score. The higher the score, the easier the candidate output is to read: scores of 90-100 correlate to a 5th grade reading level, while scores less than 10 are classified as being "extremely difficult to read, and best understood by university graduates."
The Specificity scorer outputs a score of 0 to 1, where smaller values correspond to candidate outputs with more vague language while higher values correspond to candidate outputs with more precise language. Specificity is calculated through 3 heuristic approaches: identifying the presence of predefined words that indicate vagueness, determing how rare the words used are according to word frequencies calculated by popular NLP corpora, and detecting the use of proper nouns and numbers.
For scenarios where there is a preferred output length, word_count_match calculates a corresponding score on the scale of 0 to 1. Specifically, this scorers calculates how similar the number of words in the candidate output is to the number of words in the reference output, where a score of 1.0 indicates that there are the same number of words in the candidate output as in the reference output. Scores less than 1.0 are calculated as ((len_reference-delta)/len_reference) where delta is the absolute difference in word lengths between the candidate and reference outputs. All negative computed values are truncated to 0.
The Python Unit Testing scorer evaluates candidate solutions to coding tasks against unit tests. This scorer wraps the code_eval evaluator interface from HuggingFace. It is important to note that this function requires that solution code uses standard python libraries only.