Need to use your own custom evals?

There are a number of ways to use custom evals in Athina.

​
Create Custom Evals in the UI

You can also create custom evaluators. See here for more information.

​
Create Custom Evals Programmatically

Grading Criteria

Pass / fail based on a custom criterion. “If X, then fail. Otherwise pass.”

Custom Prompt

Use a completely custom prompt for evaluation.

API Call

Use the ApiCall evaluator to make a call to a custom endpoint where you are hosting your evaluation logic.

Custom Code

Use the CustomCodeEval to run your own Python code as an evaluator.

Create Your Own

Create your own evaluator by extending the BaseEvaluator class.

​
Contribute your evals

This library is open source and we welcome contributions.

If you have an idea for a new evaluator, please open an issue or submit a PR.