Bring your own eval

athina-api-key

Athina

Open-Source Evals

Blog

Email us

Book a call

Docs

API / SDK Reference

Guides

FAQs

Sign-up

AI Hub

Website

Athina is a collaborative AI development platform that lets teams build, test and monitor production-grade AI applications.

Introduction

Athina AI

Getting Started

Getting Started with Athina

You can currently create datasets in Athina in the following ways:

Create a Dataset

You can import your inference logs to create a dataset in Athina.

From Logs

You can upload a JSONL, CSV, or JSON file to create a dataset in Athina.

Upload File

You can use our Python SDK to create a dataset in Athina.

Via Python SDK

You can use simple POST API requests to add rows to a dataset.

Via API

Import a HuggingFace Dataset

You can generate synthetic datasets in Athina.

Generate a Synthetic Dataset

Dynamic columns let you run prompts, code execution, retrievals, and more on your datasets

Overview

Send a prompt to a language model to generate an AI response

Run Prompt

API Call

Classify text into pre-defined labels using an LLM

Classification

Extract entities from a previous column using an LLM

Extract Entities

Code Execution

Run evaluations on Athina IDE in a few clicks

Run evaluations (UI)

You can view and compare metrics across datasets in Athina.

View Metrics

Re-generate a dataset with a new prompt or a new model and compare the results side-by-side

Run Experiments

You can compare multiple datasets side-by-side in Athina.

Compare datasets

Join Datasets

Export / Download Datasets

SQL Notebook

Automate workflows with project automations.

Automations

Dataset Automations

Manage Datasets

You can delete a dataset in Athina by following these steps.

Delete a Dataset

Athina Evals

Quick Start

Online Evals

Offline Evals

Preset Evaluators

Custom Evals

Running Evals in UI

Running Evals via SDK

Running Evals in CI/CD

The philosophy behind `athina-evals`, our open-source evaluation library.

Open Source Evaluations

Why Athina Evals

Cookbooks

Eval Cookbooks

Flows

Concepts

Variables

Variables in Flows

Learn how to share flows publicly or with selected users.

Sharing Flows

Flow Templates are public flows that you can copy and use in your own flows.

Flow Templates

Blocks and flows are composable and can be used to create powerful workflows.

Blocks

Search

Retrieve documents from a knowledge base.

Knowledge Retrieval

Allow your team to annotate and label your datasets.

Annotation

Understand annotation progress and agreement rates across your datasets.

Metrics

Annotation Metrics

Learn how to set up and configure an annotation project in Athina AI.

Configure Project

Configure an Annotation Project

Learn how to inspect completed annotations and monitor dataset coverage.

View Data

View Annotated Data

Learn how to review and validate completed annotations directly within the dataset.

Review Entries

Review Annotation Entries

Export labeled datasets from Athina in your preferred format.

Export Data

Export Annotated Data

Manage who can create, access, and contribute to annotation projects.

Permissions

Annotation Permissions

Athina offers a powerful prompt management system that allows you to create, edit, manage, test and version prompts.

Understand how prompts are structured in Athina

Learn how to write prompts in Athina's Playground.

Prompt Syntax

You can create prompts in Athina's Prompt Playground, or via API

Create Prompt Template

You can version your prompts in Athina's Prompt Playground, or via API.

Prompt Versioning

You can delete a prompt slug in Athina's Prompt Playground, or via API

Delete Prompt Slug

You can list all prompt slugs stored on Athina via API or Python SDK

List Prompt Slugs

You can duplicate a prompt slug in Athina's Prompt Playground, or via API

Duplicate Prompt Slug

You can run prompts in Athina's Prompt Playground, or via API

You can run multiple prompts in Athina's Prompt Playground, or via API

Multi-Prompt Playground

You can evaluate prompt responses in Athina's Playground.

Run Evaluations on a Prompt Response

You can organize your prompts in Athina's Prompt Management System

Organize Prompts

Advanced Monitoring & Analytics. Your production environment will thank you.

Athina Monitoring

Inference Trace

Analytics

Analytics and Insights

Topic Classification

Query Topic Classification

Export Data from Athina

How can I export my logged inferences?

Athina can run continuous evaluations on your logs to monitor model performance in production

Continuous Evaluation

Model Performance Metrics

You can configure custom models on Athina in the Settings page.

Custom Models

You can set up sampling rules for continuous evals in the Settings.

Sampling Evals

Understand how execution credits work in Athina.

Credits

Athina has integrations with the following projects/packages:

Integrations

Self-Hosting

Self-Hosting On Azure

To get started with Athina's Monitoring, the first step is to start logging your inferences.

Logging

Logging Attributes

Log via API Request

Log via Python SDK

Log via TypeScript SDK

Langchain

LiteLLM

🚅 LiteLLM

If you're using OpenAI chat completions in Python, you can get set up in just 2 minutes

OpenAI Chat (1.x)

OpenAI Chat Completion

OpenAI Chat (0.x)

OpenAI Assistant

Athina LLM Application Tracing captures the full context of an execution including retrieval, generation, api calls, and more

Tracing

Tracing to Athina can be done via simple API requests.

Tracing via API

Tracing to Athina can be done using Python decorators.

Tracing using Python decorators

Athina Tracing integrates with Langchain using Langchain Callbacks (Python). Thereby, our SDK automatically creates a nested trace for every run of your Langchain application.

Getting Started

Datasets

Evals

Flows

Annotation

Prompts

Monitoring

Settings

Integrations

Self Hosting

Datasets