Evals
Custom Evals
Evals
Custom Evals
Need to use your own custom evals?
There are a number of ways to use custom evals in Athina.
Create Custom Evals in the UI
You can also create custom evaluators. See here for more information.
Create Custom Evals Programmatically
Grading Criteria
Pass / fail based on a custom criterion. “If X, then fail. Otherwise pass.”
Custom Prompt
Use a completely custom prompt for evaluation.
API Call
Use the ApiCall evaluator to make a call to a custom endpoint where you are
hosting your evaluation logic.
Custom Code
Use the CustomCodeEval to run your own Python code as an evaluator.
Create Your Own
Create your own evaluator by extending the BaseEvaluator class.
Contribute your evals
This library is open source and we welcome contributions.
If you have an idea for a new evaluator, please open an issue or submit a PR.