Open-Source Evals

Blog

Email us

Athina

Observability

Evals

Logging

Guides

Sign-up

Overview

Datasets

Monitoring, Analytics & Evaluations for LLM Developers

Athina AI

Getting Started Guide

To get started with Athina's Monitoring, the first step is to start logging your inferences.

Logging Attributes

If you're using OpenAI chat completions in Python, you can get set up in just 2 minutes

OpenAI Chat (1.x)

OpenAI Chat Completion

OpenAI Chat (0.x)

OpenAI Assistant

LiteLLM

🚅 LiteLLM

Langchain

Log via API Request

Log via Python SDK

Athina LLM Application Tracing captures the full context of an execution including retrieval, generation, api calls, and more

Tracing

Athina Tracing integrates with Langchain using Langchain Callbacks (Python). Thereby, our SDK automatically creates a nested trace for every run of your Langchain application.

Tracing for Langchain

Tracing for Langchain (Python)

Where is data stored?

Logging Latency

Q. Will the Athina logging SDK increase my latency?

Is Athina using a Proxy method?

Q. Is this SDK going to make a proxy request to OpenAI through Athina?

On-prem deployment

Q. Can Athina's observability platform be deployed on-prem?

Proxy

Q. Can I log inferences from any model to Athina?

How to update logged inferences

Updating Logged Inferences

Logging user feedback

How to log conversations

Grouping Inferences

_If you're using OpenAI completions in Python, you can get set up in just **2 minutes**_

OpenAI Completion 0.x

OpenAI Completions

OpenAI Completion 1.x

Athina Evals

Quick Start

Automatic Evals

Why Athina Evals

Running Evals

Python: Run a single eval

Here is a sample of all the code you need to run a suite of evals.

Python: Run evals on a dataset

Running a suite of evals

UI: Run evals on a dataset

Run evals on a dataset on Athina Platform

Continuous Evaluation in Production

Preset Evals

RAG evals

Context Contains Enough Information

Faithfulness

Does Response Answer Query

Summarization Q&A

More Evals

Other Evaluators

Function Evals

Function Based Evaluators

Grounded Evals

RAGAS

Conversation Evaluators

Conversation Evals

Groundedness

Custom Evals

API Call

Custom Prompt

LLM Evaluation with Custom Prompt

Custom Grading Criteria

Bring your own eval

Loading data for Evals

Loading data via Llama-Index

Running Evals in CI/CD

FAQs

Frequently Asked Questions

Cookbooks

Eval Cookbooks

Create Your Own Eval

Write your own LLM Eval class

Develop Dashboard

Athina Develop: A Dashboard for your Iterations

Open Source Evaluation: Philosophy

Measuring retrieval and response quality in RAG

Measuring retrieval and response quality in RAG-based LLMs

Prompt Injection: Attacks and Defenses

Running evals as real-time guardrails

Different stages of evaluation

Improving Eval performance

How can I improve the performance / reliability of my evals?

Advanced Monitoring & Analytics. Your production environment will thank you.

Athina Monitoring

Inference Trace

Analytics

Analytics and Insights

Topic Classification

Query Topic Classification

Export Data from Athina

How can I export my logged inferences?

Model Performance Metrics

You can currently create datasets in Athina in two ways: from existing inference logs or by uploading a JSONL file.

Create Dataset

Dataset Attributes

Create Dataset Request Body Fields

Add Dataset Rows

Add Dataset Rows via API

Athina is natively integrated with the following projects/packages:

Integrations

Getting Started

Sample GraphQL Queries

We have provided examples of how to query the Athina AI GraphQL API using cURL and Python. You can use these examples to fetch data from the API and integrate it into your applications.