> ## Documentation Index
> Fetch the complete documentation index at: https://docs.athina.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Online Evals

<Tip>
  Athina users run **millions of evals** on their logged inferences every week.
  [Sign up for free](https://app.athina.ai/)
</Tip>

<Info>
  Evaluating logs in production is the only way to know if your LLM application
  is working correctly in the real world.
</Info>

Online evals are a critical part of running a successful LLM application.

They allow you to measure the quality of your LLM application over time, detect performance and safety issues, and prevent regressions.

### Why use Athina for Online Evals?

* **50+ preset evals**
* **Support for custom evals**
* **Support for popular eval libraries like Ragas, Guardrails, etc**
* **Sampling:** sample a subset of logs
* **Filtering:** only run on logs WHERE X is true
* **Rate limiting:** intelligent throttling to avoid rate limiting issues with your LLM provider
* Use **any model provider** for LLM evals
* **View aggregate analytics**
* View **traces with eval results**
* Track **eval results over time**

### How does it work?

This is a simplified view of the architecture used to run evals on logged inferences in production at scale.

![](https://mintlify.s3.us-west-1.amazonaws.com/athinaai/images/athina-architecture.png)

## Key Features[](#key-features)

<AccordionGroup>
  <Accordion title="Run the same evaluations in dev, CI / CD, and prod">
    Athina's core evaluation framework is open source and can be used to run the same evaluations in development, CI / CD, and production.

    See it on Github: [athina-evals](https://github.com/athina-ai/athina-evals)
  </Accordion>

  <Accordion title="Observability">
    Athina provides a complete observability platform for your LLM application:

    * Detailed trace inspection
    * Manual annotation capabilities
    * Unified online/offline metrics
    * PagerDuty and Slack integrations
    * Data export functionality
    * API/GraphQL access

    ![](https://mintlify.s3.us-west-1.amazonaws.com/athinaai/images/athina-dashboard.png)
  </Accordion>

  <Accordion title="Production-Grade Evaluation Without Ground Truth">
    Evaluate your LLM applications in production with confidence:

    * Advanced LLM-based evaluation techniques for [measuring retrieval and response quality](guides/evals/measuring-retrieval-accuracy-in-rag)
    * State-of-the-art research-backed evaluation metrics
    * Continuous improvement of evaluation methodologies

    ![](https://mintlify.s3.us-west-1.amazonaws.com/athinaai/images/orchestration2.png)
  </Accordion>

  <Accordion title="Cost Management">
    Maximize evaluation coverage while minimizing costs:

    * Smart sampling strategies
    * Configure evals to run on only a subset of logs based on filters
    * Comprehensive cost tracking and optimization
    * Configurable evaluation frequency
  </Accordion>

  <Accordion title="Flexible Evaluation Framework (Open Source)">
    Comprehensive evaluation capabilities:

    * Rich library of 50+ preset evals
    * Customizable evaluation configurations
    * Build and deploy custom evals
    * Multiple model provider support
    * Seamless integration with popular eval libraries (Ragas, Guardrails, etc)
  </Accordion>

  <Accordion title="Enterprise-Scale Automation">
    Fully automated evaluation pipeline:

    * Scalable evaluation infrastructure
    * Centralized eval configuration management
    * Smart eval-to-prompt matching
    * Intelligent rate limiting
    * Multi-provider model support
    * Historical log evaluation capabilities
  </Accordion>

  <Accordion title="Universal Architecture Support">
    Seamlessly adapt to any LLM stack:

    * Multi-provider support (OpenAI, Gemini, etc)
    * Framework-agnostic (Langchain, Llama Index, custom)
    * Complex trace and agent support
    * Flexible architecture adaptation
    * Standardized evaluation layer
  </Accordion>

  {" "}

  <Accordion title="View Traces with Eval Results">
    ![](https://mintlify.s3.us-west-1.amazonaws.com/athinaai/images/orchestration3.png)
  </Accordion>

  <Accordion title="Comprehensive Analytics Suite">
    Deep insights into your LLM application:

    * Application performance metrics
    * Retrieval quality analytics
    * Resource utilization tracking
    * Safety and security monitoring
    * Temporal analysis
    * Statistical distribution insights
    * Multi-dimensional segmentation

    ![](https://mintlify.s3.us-west-1.amazonaws.com/athinaai/images/orchestration4.png)
  </Accordion>

  <Accordion title="Team Collaboration">
    Enterprise-ready collaboration features:

    * Team workspaces
    * Role-based access control
    * Workspace isolation
    * Shared evaluation insights
  </Accordion>
</AccordionGroup>

***

## 👋 Athina

We spent a lot of time working through these problems so you don't need a dedicated team for this. You can see a [demo video ](https://bit.ly/athina-demo-feb-2024) here.

Website: [Athina AI ](https://athina.ai/) (Try our [sandbox ](https://bit.ly/athina-sandbox)).

[Sign Up ](https://app.athina.ai/) for Athina.

[Github ](https://github.com/athina-ai/athina-evals): Run any of our 40+ open source evaluations using our Python SDK to measure your LLM app.