❊ Info
These evaluators run a defined function on the response. How does it work A function evaluator runs a provided function along with the arguments for this function on the response and return whether the function passed or not. Required Args Your dataset must contain these fields:response
: The LLM generated response for the user query
Passed
: Boolean(True/False) value specifying whether the function passed or not.
▷ Run the function eval on a single datapoint
▷ Run the function eval on a dataset
- Load your data into a dictionary
- Run the evaluator on your dataset
Following are examples of the various function evaluators we support
Regex
Description: Checks if theresponse
contains the regex pattern.
Arguments:
pattern
:str
Pattern to search for.
Contains Any
Description: Checks if theresponse
contains any word from the list of keywords.
Arguments:
keywords
:List[str]
List of keywordscase_sensitive
:Optional[bool]
. Defaults toFalse
.
Contains None
Description: Checks if theresponse
does not contain any of the specified substrings.
Arguments:
keywords
: List of strings - keywords to check for absence in the context.
Contains
Description:Checks if the
response
contains the specified keyword.
Arguments:
keyword
: string to check for presence in the response.
ContainsAll
Description:Checks if all the provided keywords are present in the
response
.
Arguments:
keywords
: List[str] - The list of keywords to search for in the response.case_sensitive
: bool, optional - IfTrue
, the comparison is case-sensitive. Defaults toFalse
.
ContainsJson
Description:Checks if the
response
contains a valid JSON.
Arguments:
- None
ContainsEmail
Description:Checks if the
response
contains a valid email address.
Arguments:
- None
IsJson
Description:Checks if the
response
is a valid JSON.
Arguments:
- None
IsEmail
Description:Checks if the
response
is a valid email address.
Arguments:
- None
ContainsLink
Description:Checks if the
response
contains any links.
Arguments:
- None
ContainsValidLink
Description:Checks if the
response
contains valid links.
Arguments:
- None
NoInvalidLinks
Description:Checks if the
response
does not contain any invalid links.
Arguments:
- None
ApiCall
Description:Performs an API call to a specified endpoint and picks up the evaluation result from the response. This evaluator is useful when you want to run some complex or custom logic on the response. Arguments:
url
: string - API endpoint to call. Note that this API should accept POST request.headers
: dict - Headers to include in the API call.payload
: dict - Body to send with the API call. This payload will have the Response added to it.
- We expect the API response to be in JSON format with two keys namely
result
andreason
. - Theresult
key should contain the evaluation result which should be a boolean value. - Thereason
key should contain the reason for the evaluation result which should be a string. - The dataset should contain theresponse
and optionally thequery
,context
andexpected_response
to be passed to the API.
Equals
Description: Checks if theresponse
is exactly equal to the specified string.
Arguments:
expected_response
:str
String to compare the response with.
StartsWith
Description: checks if theresponse
starts with the specified substring.
Arguments:
substring
:str
string to check at the start of theresponse
.
EndsWith
Description: checks if theresponse
ends with the specified substring.
Arguments:
substring
:str
string to check at the end of theresponse
.
LengthLessThan
Description: Checks if the length of theresponse
is less than a maximum length.
Arguments:
max_length
:int
the maximum allowable length for theresponse
.
LengthGreaterThan
Description: Checks if the length of theresponse
is more than a minimum length.
Arguments:
min_length
:int
the minimum allowable length for theresponse
.
Length Between
Description: Checks if the length of theresponse
is between the minimum and maximum length.
Arguments:
min_length
:int
the minimum allowable length for theresponse
.max_length
:int
the maximum allowable length for theresponse
.
One Line
Description: Checks if theresponse
is a single line.
Arguments:
- None
CustomCodeEval
Description: Runs a custom code as an evaluator. Arguments:code
:str
Code to be executed. The code should contain a function namedmain
which takes**kwargs
as input and returns a boolean value.
JsonSchema
Description: Validates the JSON structure against a specified JSON schema. Arguments:schema
:str
The JSON schema to validate against.
JsonValidation
Description: Validates the value of a JSON field against a specified condition. Arguments: validations: list A list of validation rules. Each rule is a dictionary with the following keys: json_path: str The JSON path to the field to validate. validating_function: str The name of the validation function to use.validations
:list
The validations list