Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.athina.ai/llms.txt

Use this file to discover all available pages before exploring further.

For this example, we will consider the Loader class. Other loaders also work similarly. You can load data for evals from a JSON, Python Dictionary, or directly from your logged inferences on Athina.

​
Loading from a JSON file

from athina.loaders import Loader

dataset = Loader().load_json(json_filename)
That’s all you need to do to load your data! To view the imported dataset as a pandas DataFrame: 
pd.DataFrame(dataset)

​
Loading from a Python Dictionary

from athina.loaders import Loader

# Create batch dataset from list of dict objects
raw_data = [
    {
        "query": "What is the capital of Greece?",
        "context": ["Greece is often called the cradle of Western civilization."],
        "response": "Athens",
    },
    {
        "query": "What is the price of a Tesla Model 3?",
        "context": ["Tesla Model 3 is a fully electric car."],
        "response": "I cannot answer this question as prices vary from country to country.",
    },
    {
        "query": "What is a shooting star?",
        "context": ["Black holes are stars that have collapsed under their own gravity. They are so dense that nothing can escape their gravitational pull, not even light."],
        "response": "A shooting star is a meteor that burns up in the atmosphere.",
    }
]

dataset = Loader().load_dict(raw_data)

​
Loading logged inferences from Athina

Instead of generating inferences, you can just load inferences that you have already logged to Athina. 
# Load last 50 logged inferences
dataset = Loader().load_athina_inferences()
You can optionally apply filters to load a different subset of inferences from Athina 
from athina.loaders import Loader
from athina.interfaces.athina import AthinaFilters

# Load 100 logged inferences matching these filters
filters = AthinaFilters(
    environment="production",
    prompt_slug="refund_prompt_5.4",
    language_model_id="gpt-4",
    customer_id="nike-usa",
    topic="refunds",
)
dataset = Loader().load_athina_inferences(filters=filters, limit=100)

​
Output Format

The output format will be different for different Loaders. The Loader will return a List[DataPoint] type after you call the load function of choice. 
class DataPoint(TypedDict):
    query: str
    context: List[str]
    response: str
    expected_response: str