Evaluation with Custom Python Code

Using this eval, you can run your own python code as an evaluator. This evaluator is useful when you want to run a custom code to evaluate the data.

The code should contain a function named main which takes **kwargs as input and returns a boolean value. It should return True if your evaluation creteria is met and False otherwise.

There are multiple places from where you can run the custom code evaluator. For example, you can run it from

• The Athina Automatic Eval
• In Athina Develop IDE flow, or
• From the Athina Python SDK.

Given below are sample codes in these different scenarios that you can run the custom code evaluator from.

# kwargs is a dictionary containing the prompt run fields
# query, context, response and expected_response
def main(**kwargs):
    return len(kwargs['response']) > 100