LLM-as-a-Judge (Custom Prompt Eval)

Uses your evaluation prompt

If you have a more complex evaluation prompt that you would like to run within Athina’s framework, we can support that with our CustomPrompt class.

• Input: response, query, context, expected_response (whichever you specify in your prompt).
• Type: boolean
• Metrics: passed (0 or 1)

Example:

Evaluation Inputs:

• eval_prompt: “Think step-by-step. Based on the provided user query and refund policy, determine if the response adheres to the refund policy. User query: Refund policy: response: ”
• query: “How many vacation days are we allowed?”
• context: “Employees are allowed 15 holidays per year, and 5 days of paid leave.”
• response: “Employees are allowed 20 vacation days per year, and 5 days of paid leave.”

​

Demo Video

​

How to use it in the SDK

Simply use the CustomPrompt class and specify your own eval_prompt.

from athina.evals import CustomPrompt

eval_prompt = """
Think step-by-step.
Based on the provided user query, determine if the response answer the query correctly.
If it does then result should be True else False. Add explanation if False

User query: {{query}}
response: {{response}}
"""

data = [
    {
        "query": "Where is France and what is it's capital?",
        "context": ["France is the country in europe known for delicious cuisine", "Tesla is an electric car", "Elephant is an animal"],
        "response": "Tesla is an electric car",
    }
]

batch_run_result = CustomPrompt(
    display_name="Response must answer the query",
    required_args=["query", "response"],
    model="gpt-4o",
    eval_prompt=eval_prompt,
).run_batch(data=data)

See an example notebook → .