Need to use your own custom evals?

There are a number of ways to use custom evals in Athina.

  • Grading Criteria: Pass / fail based on a custom criterion. “If X, then fail. Otherwise pass.”
  • Custom Prompt: Use a completely custom prompt for evaluation.
  • Api Call: Use the ApiCall evaluator to make a call to a custom endpoint where you are hosting your evaluation logic.
  • Custom Code: Use the CustomCodeEval to run your own Python code as an evaluator.
  • Create your own evaluator: Create your own evaluator by extending the BaseEvaluator class.

Contribute your evals

This library is open source and we welcome contributions.

If you have an idea for a new evaluator, please open an issue or submit a PR.