Evaluation with Custom Python Code

Using this eval, you can run your own python code as an evaluator. This evaluator is useful when you want to run a custom code to evaluate the data. The code should contain a function named main which takes **kwargs as input and returns a boolean value. It should return True if your evaluation creteria is met and False otherwise. There are multiple places from where you can run the custom code evaluator. For example, you can run it from

Given below are sample codes in these different scenarios that you can run the custom code evaluator from.

# kwargs is a dictionary containing the prompt run fields
# query, context, response and expected_response
def main(**kwargs):
    return len(kwargs['response']) > 100

# kwargs is a dictionary containing all the different columns of your dataset
def main(**kwargs):
    return str(kwargs['query']).isalnum()

from athina.evals import CustomCodeEval

# Example data
data = [
    {"text": "This is a short text."},
    {"text": "The Great Barrier Reef is the world's largest coral reef system.\n It is composed of over 2,900 individual reefs and 900 islands stretching for over 2,300 kilometers."}
]

code = """
def main(**kwargs):
    return len(kwargs['text']) > 100 and len(kwargs['text']) < 500
"""

CustomCodeEval(code=code).run_batch(data=data).to_df()