Open-Source Evals

Blog

Email us

Book a call

athina-api-key

Athina

Docs

API / SDK Reference

Guides

FAQs

Sign-up

AI Hub

Website

Bring your own eval

Athina is a collaborative AI development platform that lets teams built, test and monitor production-grade AI applications.

Introduction

Athina AI

Getting Started

Getting Started with Athina

You can currently create datasets in Athina in the following ways:

Create a Dataset

You can import your inference logs to create a dataset in Athina.

From Logs

You can upload a JSONL file to create a dataset in Athina.

Upload File

You can use our Python SDK to create a dataset in Athina.

Via Python SDK

You can use simple POST API requests to add rows to a dataset.

Via API

Import a HuggingFace Dataset

You can generate synthetic datasets in Athina.

Generate a Synthetic Dataset

Dynamic columns let you run prompts, code execution, retrievals, and more on your datasets

Overview

Send a prompt to a language model to generate an AI response

Run Prompt

API Call

Classify text into pre-defined labels using an LLM

Classification

Extract entities from a previous column using an LLM

Extract Entities

Code Execution

Run evaluations on Athina IDE in a few clicks

Run evaluations (UI)

You can view and compare metrics across datasets in Athina.

View Metrics

Re-generate a dataset with a new prompt or a new model and compare the results side-by-side

Run Experiments

You can compare multiple datasets side-by-side in Athina.

Compare datasets

Join Datasets

Export / Download Datasets

SQL Notebook

Automate workflows with project automations.

Automations

Dataset Automations

Manage Datasets

You can delete a dataset in Athina by following these steps.

Delete a Dataset

Athina Evals

Quick Start

Online Evals

Offline Evals

Preset Evaluators

Custom Evals

Running Evals in UI

Running Evals via SDK

Running Evals in CI/CD

The philosophy behind `athina-evals`, our open-source evaluation library.

Open Source Evaluations

Why Athina Evals

Cookbooks

Eval Cookbooks

Flows

Blocks and flows are composable and can be used to create powerful workflows.

Blocks

Retrieve documents from a knowledge base.

Knowledge Retrieval

Allow your team to annotate and label your datasets.

Annotation

Athina offers a powerful prompt management system that allows you to create, edit, manage, test and version prompts.

Understand how prompts are structured in Athina

Concepts

Learn how to write prompts in Athina's Playground.

Prompt Syntax

You can create prompts in Athina's Prompt Playground, or via API

Create Prompt Template

You can version your prompts in Athina's Prompt Playground, or via API.

Prompt Versioning

You can delete a prompt slug in Athina's Prompt Playground, or via API

Delete Prompt Slug

You can list all prompt slugs stored on Athina via API or Python SDK

List Prompt Slugs

You can duplicate a prompt slug in Athina's Prompt Playground, or via API

Duplicate Prompt Slug

You can run prompts in Athina's Prompt Playground, or via API

You can run multiple prompts in Athina's Prompt Playground, or via API

Multi-Prompt Playground

You can evaluate prompt responses in Athina's Playground.

Run Evaluations on a Prompt Response

You can organize your prompts in Athina's Prompt Management System

Organize Prompts

Advanced Monitoring & Analytics. Your production environment will thank you.

Athina Monitoring

Inference Trace

Analytics

Analytics and Insights

Topic Classification

Query Topic Classification

Export Data from Athina

How can I export my logged inferences?

Athina can run continuous evaluations on your logs to monitor model performance in production

Continuous Evaluation

Model Performance Metrics

You can configure custom models on Athina in the Settings page.

Custom Models

You can set up sampling rules for continuous evals in the Settings.

Sampling Evals

Athina has integrations with the following projects/packages:

Integrations

Self-Hosting

To get started with Athina's Monitoring, the first step is to start logging your inferences.

Logging

Logging Attributes

Log via API Request

Log via Python SDK

Log via TypeScript SDK

Langchain

LiteLLM

🚅 LiteLLM

If you're using OpenAI chat completions in Python, you can get set up in just 2 minutes

OpenAI Chat (1.x)

OpenAI Chat Completion

OpenAI Chat (0.x)

OpenAI Assistant

Athina LLM Application Tracing captures the full context of an execution including retrieval, generation, api calls, and more

Tracing

Athina Tracing integrates with Langchain using Langchain Callbacks (Python). Thereby, our SDK automatically creates a nested trace for every run of your Langchain application.

Tracing for Langchain

Tracing for Langchain (Python)

Supported Models

Update Logs By ID

Update Logs By External Reference ID

You can list all datasets using the Python SDK.

List All Datasets

You can create a dataset programmatically using the Python SDK.

You can add rows to an existing dataset using the Python SDK.

Add Rows to Dataset

You can get a specific dataset by ID or name.

Get Dataset

You can delete a dataset by ID via the Python SDK.

Delete Dataset

Python: Run a single eval

Running Evals

Here is a sample of all the code you need to run a suite of evals.

Python: Run evals on a dataset

Running a suite of evals

Loading data for Evals

Loading data via Llama-Index

Preset Evals

JSON Evals

Summarization QA

Function Evals

Function Based Evaluators

Grounded Evals

Conversation Evaluators

Conversation Evals

LLM-as-a-Judge (Custom Prompt Eval)

Pairwise Evaluation

Custom Code Eval

Evaluation with Custom Python Code

Custom Grading Criteria

Create Your Own Eval

Write your own LLM Eval class

Sample GraphQL Queries

We have provided examples of how to query the Athina AI GraphQL API using cURL and Python. You can use these examples to fetch data from the API and integrate it into your applications.

Curl and Python Examples

Query the Graph API using cURL or Python

Where is data stored?

On-prem deployment

Q. Can Athina's observability platform be deployed on-prem?

Logging Latency

Q. Will the Athina logging SDK increase my latency?

Is Athina using a Proxy method?

Is this SDK going to make a proxy request to OpenAI through Athina?

Can I log inferences from any model?

How to log conversations

Grouping Inferences

Why use LLM-as-a-judge for evaluations?

Can I choose which model to use for running evaluations?

How do you manage costs for LLM evaluation?

Why Not Use Traditional Evaluation Metrics?

_If you're using OpenAI completions in Python, you can get set up in just **2 minutes**_

Getting Started

Datasets

Evals

Flows

Annotation

Prompts

Monitoring

Settings

Integrations

Self Hosting

Datasets