iblai-api-agent-eval
Mirrored from
iblai/base·skills/agents/iblai-api-agent-eval/SKILL.md. This page is generated — edit it in the repository, not here.
Measure and improve an agent's quality from the API: build evaluation
datasets, run experiments that send each question to the agent, then grade the
results with LLM-as-Judge and/or human scores and export to CSV. Use to test an
agent against a dataset and grade the results. The UI twin is
/iblai-vibe-agent-evals,
which calls datasets "benchmarks" and runs "evaluations".
Auth & conventions
- Base URL:
https://api.iblai.app - Header:
Authorization: Api-Token $IBLAI_API_KEYon every request. (The dev docs phrase this asAuthorization: Token <key>— it is the same platform key; use Api-Token.) - Path vars:
{org}=$IBLAI_ORG,{username}=$IBLAI_USERNAME. - Host root:
…/dm/api/ai-mentor/orgs/{org}/users/{username}/evaluations/. Below,…/evals= that root. (ai-mentoris the canonical mount; theai-agentspelling is an accept-only alias for the same routes.) - Not connected yet? Run
/iblai-api-loginfirst to populateIBLAI_ORG,IBLAI_USERNAME, andIBLAI_API_KEY.
Concepts
- These eval datasets are not the agent's RAG datasets.
evaluations/datasets/hold graded test cases (input + expected output) for measuring agent quality. They are unrelated to an agent's knowledge/training datasets in/iblai-api-agent-dataset(RAG documents) — do not cross-wire the two. - Eval data is org-scoped and isolated. Datasets, items, runs, scores, and
score configs belong to
$IBLAI_ORGalone — no other org can read them or grade against them. - Runs and judges are async task records. Starting a run (
POST …/runs/) or an LLM-as-Judge (POST …/evaluate/) dispatches a background task and returns 202 immediately with a task record that movespending → in_progress → completed(orfailed). Poll for status; a run must reachcompletedbefore you judge or export it. - Three grading paths. LLM-as-Judge (
…/evaluate/) scores every item in a run automatically from a free-textcriteriarubric. Scores (…/scores/) are individual human/numeric/boolean/categorical annotations on a trace. Score configs (…/score-configs/) are reusable rubrics a score references viaconfig_id. - Pagination. List endpoints take
?page=(1-indexed) and?limit=(default 50, max 200) and return{count, next, previous, results}. - On the wire the path segment is
users/{user_id}; its value is$IBLAI_USERNAME.
Reads
Datasets
- GET
https://api.iblai.app/dm/api/ai-mentor/orgs/{org}/users/{username}/evaluations/datasets/— list. Filters:?name=(substring),?user_email=(creator, exact).
Every dataset, run, score and judge-record read identifies its creator with
username, user_email and user_full_name (the display name). All three are
read-only and empty for rows written before the convention, or where the user
no longer exists. ?user_email= filters on the same fact it returns.
- GET
…/evaluations/datasets/{dataset_name}/— get one.
Dataset items
- GET
…/datasets/{dataset_name}/items/— list items. Filters:?status=ACTIVE|ARCHIVED,?include_trace=true(attachtrace_input/trace_outputfor items linked to a source trace). - GET
…/datasets/{dataset_name}/items/{item_id}/— get one item (includestrace_input/trace_outputwhen it has asource_trace_id).
Experiments (runs)
- GET
…/datasets/{dataset_name}/runs/— list runs (merges completed runs with in-flight pending records). Filters:?name=(substring),?user_email=. - GET
…/datasets/{dataset_name}/runs/{run_name}/— run details: header +dataset_run_items(each withinput,expected_output,actual_output,error,scores[]) +pending_judges[]. - GET
…/datasets/{dataset_name}/runs/{run_name}/export/— export results as CSV (columns:item_id,input,expected_output,actual_output,trace_id, plus onescore_<name>column per score).
LLM-as-Judge
- GET
…/datasets/{dataset_name}/runs/{run_name}/evaluate/— list judge task records for this run (all statuses). Filters:?status=pending|in_progress|completed|failed,?user_email=. - GET
…/evaluations/judges/— list judge task records across every dataset/run in the org. Filters:?status=,?user_email=,?dataset_name=.
Scores
- GET
…/evaluations/scores/— list scores (human annotations + LLM-judge results). Filters:?dataset_run_id=(scope to a run),?trace_id=(scope to one turn),?name=(exact),?user_email=.
Score configs
- GET
…/evaluations/score-configs/— list reusable scoring rubrics.
Writes
Datasets
- POST
…/evaluations/datasets/— create.
Dataset items
- POST
…/datasets/{dataset_name}/items/— add items. Provide either anitemsarray ({input, expected_output}) or atrace_idsarray (seeds items from existing chat traces — input + agent response copied from each trace). Exactly one of the two. - POST
…/datasets/{dataset_name}/items/upload/— bulk-create from a CSV (multipartfilefield). - PUT
…/datasets/{dataset_name}/items/{item_id}/— update an item (input,expected_output,metadata,status). - DELETE
…/datasets/{dataset_name}/items/{item_id}/— delete an item (async, returns202queued). Destructive — confirm with the user first.
Experiments (runs)
- POST
…/datasets/{dataset_name}/runs/— start an experiment (async; runs the agent —mentor_unique_id— against every dataset item, records responses). Returns202with a task record. - DELETE
…/datasets/{dataset_name}/runs/{run_name}/— delete a run (async,202queued). Destructive — confirm with the user first.
LLM-as-Judge
- POST
…/datasets/{dataset_name}/runs/{run_name}/evaluate/— start an LLM-as-Judge pass (async; a separate LLM scores each item's actual output against its input + expected output using yourcriteria). Writes one score per item namedscore_name, with the judge's reasoning in each score'scomment. Run must becompletedfirst. Returns202with a judge task record.
Scores
- POST
…/evaluations/scores/— create a score (human/numeric/boolean/categorical annotation on a trace). - DELETE
…/evaluations/scores/{score_id}/— delete a score (async,202queued). Destructive — confirm with the user first.
Score configs
- POST
…/evaluations/score-configs/— create a reusable scoring rubric.
Example
Start an experiment run against a dataset (runs async in the background):
curl -X POST \
"https://api.iblai.app/dm/api/ai-mentor/orgs/$IBLAI_ORG/users/$IBLAI_USERNAME/evaluations/datasets/support-qa/runs/" \
-H "Authorization: Api-Token $IBLAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"run_name": "baseline-2026-06", "mentor_unique_id": "d17dc729-60fd-4363-81a0-f67d9318b03e"}'
Notes
- Typical pipeline: create dataset → add items → start run (async) → poll
until
completed→ grade (LLM-as-Judge and/or human scores) → export CSV. - Runs and judges are background tasks — POST returns
202before results exist; poll the run-list or run-details endpoint untilstatusiscompletedbefore callingevaluate/orexport/. - LLM-as-Judge grades a completed run from a free-text
criteriarubric, writing one score per item (score_name) plus reasoning in each score'scomment. Per-item human scores go through…/evaluations/scores/and can reference a reusable rubric from…/evaluations/score-configs/viaconfig_id. - Dataset items can come from direct JSON (
items), a CSV upload, or existing chat traces (trace_ids— input + agent response pulled from each trace). - Deletes (item, run, score) are queued (
202), not immediate. A dataset has no delete endpoint and no update endpoint; the next note is the only way one changes. POST …/datasets/with anamethat already exists does not fail: it answers201, as for a new dataset, and overwrites that dataset's description (with an empty one when none is sent) and its creator; items added next land in it. Before creating, look the name up withGET …/datasets/?name=<name>&limit=200and follownextthrough every page:?name=is a case-insensitive substring match, so the exact name can sit on a later page, and only an exact comparison of names settles it.- Adding items is not idempotent:
POST …/items/and…/items/upload/save rows one at a time, so retrying after a timeout or partial failure adds the same pairs again. List the items before you retry. - Deleting a run leaves its judge records: a new run started under the same
run_namelists the old reviews in…/evaluate/. Use a freshrun_name. - When the judge model call fails, the judge task still ends
completedand writes0with the commentLLM judge call failedfor each item. Check comments before trusting low scores.
Schema
Request bodies. All JSON except the CSV upload (multipart). Fields are optional unless marked required.
Create dataset — POST …/datasets/
namerequired (≤255) ·description·metadata(object)
Add items — POST …/datasets/{dataset_name}/items/ (exactly one of)
items: array of{inputrequired, expected_output}trace_ids: array of strings (seed from existing traces)
Update item — PUT …/items/{item_id}/
input·expected_output·metadata(object) ·status(ACTIVE|ARCHIVED)
CSV upload — POST …/items/upload/ (multipart form)
filerequired — CSV with aninputcolumn and optionalexpected_output; empty-inputrows are skipped
Start run — POST …/runs/
mentor_unique_idrequired — the agent's unique idrun_name(≤255; autorun-<hex>if omitted) ·metadata(object)
LLM-as-Judge — POST …/runs/{run_name}/evaluate/
criteriarequired — the rubric the judge grades againstscore_namerequired (≤255) — name given to the judge's scorellm_provider(defaultopenai) ·llm_name(defaultgpt-4o-mini)max_concurrency(int, default 4, range 1–20)
Create score — POST …/scores/
trace_idrequired ·namerequired (≤255) ·valuerequired (number)data_type(NUMERIC|BOOLEAN|CATEGORICAL, defaultNUMERIC)comment·observation_id·config_id(score-config to validate against) ·dataset_run_id
Create score config — POST …/score-configs/
namerequired (≤255) ·data_type(NUMERIC|BOOLEAN|CATEGORICAL) requiredcategories: array of{value(number), label(string)}— for CATEGORICALmin_value/max_value(numbers — for NUMERIC) ·description
Reference material
references/guide.md— the eval pipeline as ordered steps, judge-rubric guidance, and CSV upload/export limits and columns.