Reliability

Distributional Robustness

Scenario ID: vijil.scenarios.reliability_robustness_distributionalrobustness

Probe Name	Description
	These tests measure robustness against semantic pertubations on tasks where the agent must identify the relationship between a premise and a hypothesis. The tasks come from the MNLI benchmark.
	These tests measure robustness when replacing important words with their synonyms on tasks where the agent must identify the logical entailment between two sentences. The tasks come from the RTE benchmark.
	Measures whether the agent’s performance on the MNLI task is robust against TextFooler perturbations
	Probes effects of TextFooler perturbation on MNLI-MM task
	Probes effects of BERT-based perturbation on RTE task
	These tests measure robustness against semantic pertubations on tasks where the agent must identify the relationship between a premise and a hypothesis. The tasks come from the MNLI-Mismatched benchmark.

Factual Accuracy

Scenario ID: vijil.scenarios.reliability_correctness_factualaccuracy

Probe Name	Description
	Evaluates hallucinations about Riley Goodside
	Checks for false assertions about US senators. A high score indicates a lowered tendency to hallucinate information.
	Tests that check if the agent can be tricked by misleading information. A high score indicates a lower likelihood of producing hallucinated content, when presented with false or misleading claims.
	Tests that try to trick the agent into importing modules that dont exist in Python and other languages. A high score indicates that the agent does not produce hallucinated package information.
	Tests whether the model fabricates confident answers to queries that ask for specific information that cannot be known or verified.

Logical Validity

Scenario ID: vijil.scenarios.reliability_correctness_logicalvalidity

Probe Name	Description
	Check if the agent invents impossible flight routes. A high score indicates that the agent is resilent towards creating impossible or factually impossible scenarios.

Contextual Robustness

Scenario ID: vijil.scenarios.reliability_robustness_contextualrobustness

Probe Name	Description
	Tests agent robustness on 2-step math word problems by adding irrelevant information to the problem. The agent fails if it solves the unperturbed problem but does not solve the perturbed problem.
	Tests model robustness on GSM math problems by rephrasing the text or varying the numbers. The agent fails if it solves the unperturbed problem but does not solve the perturbed problem.Adds Add Accents mutation.
	Tests model robustness on GSM math problems by rephrasing the text or varying the numbers. The agent fails if it solves the unperturbed problem but does not solve the perturbed problem.Adds Tag mutation.