API Call

If you want to use your own evaluation functions, you can do so by using the ApiCall evaluator. Description:
Performs an API call to a specified endpoint and picks up the evaluation result from the response. This evaluator is useful if you:

Want to run some complex or custom logic
Need to run evaluations on your own servers
Want to use an evaluation metric that we don’t support currently (Send us an email at hello@athina.ai to request a new metric - we can integrate pretty fast).

Info:

Input: response, query, context, response
Type: boolean
Metrics: passed (0 or 1)

Arguments:

url: string - API endpoint to call. Note that this API should accept POST request.
headers: dict - Headers to include in the API call.
payload: dict - Body to send with the API call. This payload will have the Response added to it.

Sample Code:

from athina.evals import ApiCall
from athina.loaders import ResponseLoader
 
# Raw data must contain response and optionally the query, context and expected_response to be passed to the API
raw_data = [
    {
        "response": "Response to be sent to the your own API based evaluator",
        "query": "Query to be sent to the your own API based evaluator"
    }
]
dataset = ResponseLoader().load_dict(raw_data)
 
ApiCall(
    url="https://8e714940905f4022b43267e348b8a713.api.mockbin.io/",
    payload={"evaluator": "custom_api_based_evaluator"},
    headers={"Authorization": "Bearer token"}
).run_batch(data=dataset).to_df()

The dataset should contain the response and optionally the query, context and expected_response to be passed to the API.

Endpoint Result FormatWe expect the API response to be in JSON format with two keys namely result and reason.

result (boolean): should contain the evaluation result (true / false)
reason (string): should contain an explanation for the evaluation result

Logging

Datasets

Evals

GraphQL API

Deprecated