> ## Documentation Index
> Fetch the complete documentation index at: https://docs.athina.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM-as-a-Judge (Custom Prompt Eval)

Uses your evaluation prompt

If you have a more complex evaluation prompt that you would like to run within Athina's framework, we can support that with our `CustomPrompt` class.

* Input: `response`, `query`, `context`, `expected_response` (whichever you specify in your prompt).
* Type: `boolean`
* Metrics: `passed` (0 or 1)

**Example:**

**Evaluation Inputs:**

* **eval\_prompt**: "Think step-by-step. Based on the provided user `query` and refund policy, determine if the `response` adheres to the refund policy. User `query`: {`query`} Refund policy: {`context`} `response`: {`response`}"
* **`query`**: "How many vacation days are we allowed?"
* **`context`**: "Employees are allowed 15 holidays per year, and 5 days of paid leave."
* **`response`**: "Employees are allowed 20 vacation days per year, and 5 days of paid leave."

<Warning>
  **Evaluation Results:**

  * **result:** Fail
  * **explanation**: The `response` does not adhere to the refund policy provided. The refund policy is that employees are allowed 15 holidays per year, and 5 days of paid leave.
</Warning>

### Demo Video

<div
  style={{
position: "relative",
paddingBottom: "60.40268456375839%",
height: "0",
}}
>
  <iframe
    src="https://www.loom.com/embed/65fc2adf8c4247db9964e8d7a9d2062d"
    frameBorder="0"
    webkitAllowFullScreen
    mozAllowFullScreen
    allowFullScreen
    style={{
  position: "absolute",
  top: "0",
  left: "0",
  width: "100%",
  height: "100%",
}}
  />
</div>

### How to use it in the SDK[](#how-to-use-it-in-the-sdk)

Simply use the `CustomPrompt` class and specify your own `eval_prompt`.

```python
from athina.evals import CustomPrompt

eval_prompt = """
Think step-by-step.
Based on the provided user query, determine if the response answer the query correctly.
If it does then result should be True else False. Add explanation if False

User query: {{query}}
response: {{response}}
"""

data = [
    {
        "query": "Where is France and what is it's capital?",
        "context": ["France is the country in europe known for delicious cuisine", "Tesla is an electric car", "Elephant is an animal"],
        "response": "Tesla is an electric car",
    }
]

batch_run_result = CustomPrompt(
    display_name="Response must answer the query",
    required_args=["query", "response"],
    model="gpt-4o",
    eval_prompt=eval_prompt,
).run_batch(data=data)
```

[See an example notebook → ](https://github.com/athina-ai/athina-evals).

<Tip>
  Note: Any variables you use in the prompt (for example: `query`, `context`,
  `response`) will be interpolated from your `dataset`.
</Tip>