Custom Evals

Need to use your own custom evals?

There are a number of ways to use custom evals in Athina.

• Grading Criteria: Pass / fail based on a custom criterion. “If X, then fail. Otherwise pass.”
• Custom Prompt: Use a completely custom prompt for evaluation.
• Api Call: Use the ApiCall evaluator to make a call to a custom endpoint where you are hosting your evaluation logic.
• Create your own evaluator: Create your own evaluator by extending the BaseEvaluator class.

​

Contribute your evals

This library is open source and we welcome contributions.

If you have an idea for a new evaluator, please open an issue or submit a PR.