Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checks testapp for evaluators #1223

Open
wants to merge 47 commits into
base: main
Choose a base branch
from
Open

Conversation

HunterHeston
Copy link
Contributor

Just adding a test application which configures the Checks evaluators.

run genkit eval:run input-file.json --evaluators=checks/dangerous_content,checks/pii_soliciting_reciting,checks/harassment,checks/sexually_explicit,checks/hate_speech,checks/medical_info,checks/violence_and_gore,checks/obscenity_and_profanity

Where input-file.json is a file of this format:

[
  {
    "testCaseId": "test_case_id_1",
    "input": "The input to your model.",
    "output": "Example model output which. This is what will be evaluated."
  }
]

Checklist (if applicable):

  • Tested (manually, unit tested, etc.)
  • Docs updated

hunterheston added 11 commits November 9, 2024 00:56
1. Updated package json, removed vertexai refs.
2. Removed optional deps pulled from vertexai plugin.
3. Action -> EvaluatorAction
4. console.warn -> logger.warn
5. Moved quota project warning into index.ts
1. Updated package json, removed vertexai refs.
2. Removed optional deps pulled from vertexai plugin.
3. Action -> EvaluatorAction
4. console.warn -> logger.warn
5. Moved quota project warning into index.ts
@ssbushi
Copy link
Contributor

ssbushi commented Nov 14, 2024

I am consolidating all evaluator testapps into one... #1294

Since Checks is not very different from existing evals, can we just reuse this testapp instead? You can add a separate Checks evaluator section to the README file.

Base automatically changed from hunterheston-checks-evaluators to main November 14, 2024 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

2 participants