// required
  "source": "api",
  "name": "Demo Dataset",

  // optional
  "description": "This is a sample demo dataset",
  "language_model_id": "gpt-4",
  "prompt_template": [{
    "role": "system",
    "content": "You are an AI that answers user queries given the information from the context"
    "role": "user",
    "content": "Given this context {context}, Answer this query: {query}"
  "dataset_rows": [
          "query": "When did Virgin Australia start operating?",
          "context": [
              "\"Virgin Australia, the trading name of Virgin Australia Airlines Pty Ltd, is an Australian-based airline. It is the largest airline by fleet size to use the Virgin brand. It commenced services on 31 August 2000 as Virgin Blue, with two aircraft on a single route. It suddenly found itself as a major airline in Australia's domestic market after the collapse of Ansett Australia in September 2001. The airline has since grown to directly serve 32 cities in Australia, from hubs in Brisbane, Melbourne and Sydney.\""
          "response": "Virgin Australia commenced services on 31 August 2000 as Virgin Blue, with two aircraft on a single route.",
          "expected_response": "Virgin Australia started services in August 2000"
          "query": "When was Tomoaki Komorida born?",
          "context": [
              "\"Komorida was born in Kumamoto Prefecture on July 10, 1981. After graduating from high school, he joined the J1 League club Avispa Fukuoka in 2000. Although he debuted as a midfielder in 2001, he did not play much and the club was relegated to the J2 League at the end of the 2001 season. In 2002, he moved to the J2 club Oita Trinita. He became a regular player as a defensive midfielder and the club won the championship in 2002 and was promoted in 2003. He played many matches until 2005. In September 2005, he moved to the J2 club Montedio Yamagata. In 2006, he moved to the J2 club Vissel Kobe. Although he became a regular player as a defensive midfielder, his gradually was played less during the summer. In 2007, he moved to the Japan Football League club Rosso Kumamoto (later Roasso Kumamoto) based in his local region. He played as a regular player and the club was promoted to J2 in 2008. Although he did not play as much, he still played in many matches. In 2010, he moved to Indonesia and joined Persela Lamongan. In July 2010, he returned to Japan and joined the J2 club Giravanz Kitakyushu. He played often as a defensive midfielder and center back until 2012 when he retired.\""
          "response": "Tomoaki Komorida was born on July 10,1981.",
          "expected_response": "Tomoaki Komorida was born in July, 1981"

Required Fields


Source of the dataset creation. For creation via API, this field must be ‘api’


Name of the dataset. Name must be unique. If dataset with the same name already exists, API will throw 400


Description of the dataset


Language Model that was used to generate the LLM responses


Prompt template that was used to generate the LLM responses of the dataset


Rows of the dataset.

Currently, a dataset can have a maximum of 1000 rows. If the number of rows > 1000, API will throw 400

Fields in a single dataset row (All fields are optional)


User query to generate the LLM response


Relevant context fetched from the retrieval systems


Generated LLM Response


Ground Truth