response
: The LLM generated response for the user queryPassed
: Boolean(True/False) value specifying whether the function passed or not.response
contains the regex pattern.
Arguments:
pattern
: str
Pattern to search for.response
contains any word from the list of keywords.
Arguments:
keywords
: List[str]
List of keywordscase_sensitive
: Optional[bool]
. Defaults to False
.response
does not contain any of the specified substrings.
Arguments:
keywords
: List of strings - keywords to check for absence in the context.Contains
response
contains the specified keyword.
Arguments:
keyword
: string to check for presence in the response.ContainsAll
response
.
Arguments:
keywords
: List[str] - The list of keywords to search for in the response.case_sensitive
: bool, optional - If True
, the comparison is case-sensitive. Defaults to False
.ContainsJson
response
contains a valid JSON.
Arguments:
ContainsEmail
response
contains a valid email address.
Arguments:
IsJson
response
is a valid JSON.
Arguments:
IsEmail
response
is a valid email address.
Arguments:
ContainsLink
response
contains any links.
Arguments:
ContainsValidLink
response
contains valid links.
Arguments:
NoInvalidLinks
response
does not contain any invalid links.
Arguments:
ApiCall
url
: string - API endpoint to call. Note that this API should accept POST request.headers
: dict - Headers to include in the API call.payload
: dict - Body to send with the API call. This payload will have the Response added to it.result
and reason
. - The result
key should contain the evaluation result
which should be a boolean value. - The reason
key should contain the reason
for the evaluation result which should be a string. - The dataset should
contain the response
and optionally the query
, context
and
expected_response
to be passed to the API.response
is exactly equal to the specified string.
Arguments:
expected_response
: str
String to compare the response with.response
starts with the specified substring.
Arguments:
substring
: str
string to check at the start of the response
.response
ends with the specified substring.
Arguments:
substring
: str
string to check at the end of the response
.response
is less than a maximum length.
Arguments:
max_length
: int
the maximum allowable length for the response
.response
is more than a minimum length.
Arguments:
min_length
: int
the minimum allowable length for the response
.response
is between the minimum and maximum length.
Arguments:
min_length
: int
the minimum allowable length for the response
.max_length
: int
the maximum allowable length for the response
.response
is a single line.
Arguments:
code
: str
Code to be executed. The code should contain a function named main
which takes **kwargs
as input and returns a boolean value.schema
: str
The JSON schema to validate against.validations
: list
The validations list