UpTrain integration for Haystack
Module haystack_integrations.components.evaluators.uptrain.evaluator
UpTrainEvaluator
A component that uses the UpTrain framework
to evaluate inputs against a specific metric. Supported metrics are defined by UpTrainMetric
.
Usage example:
from haystack_integrations.components.evaluators.uptrain import UpTrainEvaluator, UpTrainMetric
from haystack.utils import Secret
evaluator = UpTrainEvaluator(
metric=UpTrainMetric.FACTUAL_ACCURACY,
api="openai",
api_key=Secret.from_env_var("OPENAI_API_KEY"),
)
output = evaluator.run(
questions=["Which is the most popular global sport?"],
contexts=[
[
"Football is undoubtedly the world's most popular sport with"
"major events like the FIFA World Cup and sports personalities"
"like Ronaldo and Messi, drawing a followership of more than 4"
"billion people."
]
],
responses=["Football is the most popular sport with around 4 billion" "followers worldwide"],
)
print(output["results"])
UpTrainEvaluator.__init__
def __init__(metric: Union[str, UpTrainMetric],
metric_params: Optional[Dict[str, Any]] = None,
*,
api: str = "openai",
api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
api_params: Optional[Dict[str, Any]] = None)
Construct a new UpTrain evaluator.
Arguments:
metric
: The metric to use for evaluation.metric_params
: Parameters to pass to the metric's constructor. Refer to theUpTrainMetric
class for more details on required parameters.api
: The API to use for evaluation. Supported APIs:openai
,uptrain
.api_key
: The API key to use.api_params
: Additional parameters to pass to the API client. Required parameters for the UpTrain API:project_name
.
UpTrainEvaluator.run
@component.output_types(results=List[List[Dict[str, Any]]])
def run(**inputs) -> Dict[str, Any]
Run the UpTrain evaluator on the provided inputs.
Arguments:
inputs
: The inputs to evaluate. These are determined by the metric being calculated. SeeUpTrainMetric
for more information.
Returns:
A dictionary with a single results
entry that contains
a nested list of metric results. Each input can have one or more
results, depending on the metric. Each result is a dictionary
containing the following keys and values:
name
- The name of the metric.score
- The score of the metric.explanation
- An optional explanation of the score.
UpTrainEvaluator.to_dict
def to_dict() -> Dict[str, Any]
Serializes the component to a dictionary.
Raises:
DeserializationError
: If the component cannot be serialized.
Returns:
Dictionary with serialized data.
UpTrainEvaluator.from_dict
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "UpTrainEvaluator"
Deserializes the component from a dictionary.
Arguments:
data
: Dictionary to deserialize from.
Returns:
Deserialized component.
Module haystack_integrations.components.evaluators.uptrain.metrics
UpTrainMetric
Metrics supported by UpTrain.
CONTEXT_RELEVANCE
Context relevance.
Inputs - questions: List[str], contexts: List[List[str]]
FACTUAL_ACCURACY
Factual accuracy.
Inputs - questions: List[str], contexts: List[List[str]], responses: List[str]
RESPONSE_RELEVANCE
Response relevance.
Inputs - questions: List[str], responses: List[str]
RESPONSE_COMPLETENESS
Response completeness.
Inputs - questions: List[str], responses: List[str]
RESPONSE_COMPLETENESS_WRT_CONTEXT
Response completeness with respect to context.
Inputs - questions: List[str], contexts: List[List[str]], responses: List[str]
RESPONSE_CONSISTENCY
Response consistency.
Inputs - questions: List[str], contexts: List[List[str]], responses: List[str]
RESPONSE_CONCISENESS
Response conciseness.
Inputs - questions: List[str], responses: List[str]
CRITIQUE_LANGUAGE
Language critique.
Inputs - responses: List[str]
CRITIQUE_TONE
Tone critique.
Inputs - responses: List[str]
Parameters - llm_persona: str
GUIDELINE_ADHERENCE
Guideline adherence.
Inputs - questions: List[str], responses: List[str]
Parameters - guideline: str
, guideline_name: str
, response_schema: Optional[str]
RESPONSE_MATCHING
Response matching.
Inputs - responses: List[str], ground_truths: List[str]
Parameters - method: str
UpTrainMetric.from_str
@classmethod
def from_str(cls, string: str) -> "UpTrainMetric"
Create a metric type from a string.
Arguments:
string
: The string to convert.
Returns:
The metric.