Have you seen our custom evaluation wrappers? They are
a great way to quickly create your own evaluation class.
If you want to write your own evaluation class, you can do so by extending the BaseEvaluator
class.You can extend the BaseEvaluator class. See this example You need to implement the following methods:
Copy
Ask AI
@propertydef _model(self): # Which model was used for this evaluation pass@propertydef name(self): return "UpperCamelCaseName"@propertydef display_name(self): return "Evaluation Display Name"@propertydef metric_ids(self) -> List[str]: return [MetricType.PASSED.value]@propertydef default_function_arguments(self): return {}@propertydef required_args(self): # expects an array of strings from ["query", "context", "response", "expected_response", "text"] pass@propertydef examples(self): passdef is_failure(self) -> Optional[bool]: passdef _evaluate(self, **kwargs) -> EvalResult: pass