# Gumloop Docs: Evaluations

**来源：** https://docs.gumloop.com/core-concepts/evaluations
**抓取日期：** 2026-07-06（二次归档，原 2026-06-16 版仅含简略概述，此次为完整版）
**状态：** ✅ 完整

---

> Evaluations give you **automated quality assurance** for your agents. After every interaction completes, an AI evaluator analyzes the full conversation transcript and produces a structured report: a grade, per-criterion pass/fail results with rationales, sentiment analysis, applied tags, and extracted data points.
>
> Think of it as having a QA analyst reviewing every single conversation your agent has, 24/7, without you lifting a finger.

## Where to Find Evaluations

Open your agent and click **Evaluations** in the left-hand sidebar.

---

## The Building Blocks

Evaluations have four building blocks:

- **Criteria** answer: "Did the agent do what it was supposed to do?" Each one is a yes/no check.
- **Tags** answer: "What kind of conversation was this?" Use them to filter and find patterns.
- **Data Points** answer: "What specific facts or values came up?" They pull structured data out of unstructured conversation.
- **Sentiment** answers: "How did the user feel?" Optionally let negative sentiment affect the grade.

---

## How Evaluations Work

1. Agent finishes a conversation (reaches "completed" state). Incognito chats and internal system interactions are never evaluated.
2. Evaluator constructs a role-tagged transcript of the entire conversation, including user messages, agent responses, tool calls, and results. Long transcripts are automatically trimmed.
3. A single structured-output LLM call analyzes the transcript against configured criteria, tags, data points, and sentiment settings.
4. Overall grade is determined deterministically (not by the LLM) based on criterion failures, priority levels, call outcome, sentiment, and action failures.
5. Results are persisted. If grade is "Critical," agent owner receives immediate Slack DM alert.

---

## Setting Up Evaluations

Navigate to your agent's **Evaluations** tab in the sidebar.

| Setting | What It Does |
| --- | --- |
| **Enable evaluations** | Toggle to start automatically grading interactions |
| **Default analysis model** | Which LLM runs the evaluation. "Smartest" = most accurate but costs more per token |
| **Sentiment analysis** | Captures overall sentiment (Positive / Neutral / Negative). Optionally affects grade. |
| **Suggest tags automatically** | Lets the evaluator propose new tags beyond your predefined vocabulary |

### Criteria

Each criterion is a clear statement that's either true or false for a given conversation.

**Adding a Criterion** — Click **+ Add criterion**:

| Field | Description |
| --- | --- |
| **Name** | A short label (e.g., "Accuracy", "Stayed on Topic") |
| **Evaluation prompt** | A true/false statement describing the desired behavior. Be specific. |
| **Type** | Categorizes the criterion: Prohibited action, Prohibited words, Voice & tone, or Other |
| **Priority** | Warning (downgrades to Warning) or Critical (downgrades to Critical + Slack alert) |

**Types:**

| Type | When to Use |
| --- | --- |
| **Prohibited action** | Agent must NOT do something (e.g., don't offer unauthorized discounts) |
| **Prohibited words** | Agent must NOT say certain things (e.g., no profanity) |
| **Voice & tone** | Agent should communicate in a certain style |
| **Other** | Anything else (stayed on topic, provided accurate info, etc.) |

**Choosing priority:** Use "Critical" for rules that must never be broken (data leaks, compliance violations). Use "Warning" for quality standards that matter but aren't urgent (tone issues, minor drifts).

**Limit:** 30 criteria per agent.

### Data Points

Data collection lets you **extract structured values** from every interaction.

**Adding a Data Point** — Click **+ Add data point**:

| Field | Description |
| --- | --- |
| **Name** | What you're extracting (e.g., "Confidence Score") |
| **Type** | Text, Boolean, Integer, or Number |
| **Description** | Extraction instructions. Be precise about what to look for and valid values. |

**Data Point Types:**

| Type | Returns | Best For |
| --- | --- | --- |
| **Text** | String | Categories, summaries, reasons |
| **Boolean** | Yes / No | Binary checks (was something done?) |
| **Integer** | Whole number | Counts, quantities |
| **Number** | Decimal | Scores, ratings, percentages |

**Limit:** 40 data points per agent.

---

## Understanding Results

Once an interaction is evaluated, you can see the full results in the interaction detail view's **Overview** tab.

### Grades

| Grade | API Value | Meaning | What Triggers It |
| --- | --- | --- | --- |
| **Pass** | `pass` | Met all criteria | No failures, call wasn't a failure, sentiment not negative |
| **Warning** | `needs_review` | Needs review | Warning-priority criterion failed, OR call outcome "failure", OR negative sentiment affects grade |
| **Critical** | `needs_attention` | Immediate attention | Critical-priority criterion failed, OR tool/action failures during interaction |

### Grade computation logic (deterministic)

The grade is computed deterministically (not by the LLM) after results come back:

1. If there were **action failures** (tool errors) → **Critical**
2. If any **Critical-priority criterion** failed → **Critical**
3. If any **Warning-priority criterion** failed → **Warning**
4. If overall **call outcome** was "failure" → **Warning**
5. If **sentiment** is negative AND affects grade → **Warning**
6. Otherwise → **Pass**

### What's Shown in Results

- **Summary**: One or two sentence narrative of what happened
- **Grade**: Pass / Warning / Critical badge
- **Outcome**: Successful / Failed / Unknown
- **Sentiment**: Positive / Neutral / Negative (if enabled)
- **Tags**: Applied tags from your vocabulary + auto-generated
- **Criteria**: Per-criterion pass/fail with the evaluator's rationale
- **Collected data**: Extracted values for each data point

### Interactions List View

The interactions list includes an **Evaluation** column showing the grade and criteria pass rate at a glance (e.g., "Pass 3/3" or "Critical 1/3"). Lifecycle states also appear: Queued, Evaluating, or Failed.

---

## Running Evaluations Manually

Evaluations run automatically, but you can also trigger them manually from the **Chats** page. You can also select multiple interactions for bulk evaluation.

**When to use manual runs:**

- Backfilling existing conversations after enabling evaluations
- Re-evaluating after changing your criteria/tags/data points
- Retrying a failed evaluation
- Spot-checking specific conversations on demand

---

## Alerts

When an interaction receives a **Critical** grade, the agent owner gets an immediate Slack DM with the agent's name, which criteria failed, and a direct link to the interaction. **No additional setup required**, as long as Slack is connected.

---

## Credits and Costs

Evaluations are billed as **AI credits** under the "AI Utilities" category. Each evaluation is a single LLM call.

**What determines the cost?** Three factors:

1. **Transcript length**: Longer conversations use more input tokens.
2. **Analysis model**: The model you select determines the per-token rate. "Smartest" costs more than faster alternatives.
3. **Schema complexity**: More criteria, tags, and data points = more output tokens to generate.

The evaluator checks that the user has sufficient credits before running. **If credits are insufficient, the evaluation is skipped silently.**

### Where to See Credit Usage

View all evaluation credit usage on your **Usage & Limits** page. Filter by "AI Utilities" to see individual evaluation runs and their credit amounts.

---

## Limits

| Resource | Maximum |
| --- | --- |
| Criteria per agent | 30 |
| Tags per agent | 50 |
| Data points per agent | 40 |
| Tag name length | 100 characters |
| Tag description length | 500 characters |

---

## Re-evaluation

- If you continue a conversation after an evaluation has run, new messages extend the transcript
- Once conversation reaches "completed" state again, a new evaluation automatically runs
- New evaluation replaces previous result
- Fully automatic—no manual re-run needed

---

## Exporting Evaluation Data

Every evaluation result, including grades, criteria outcomes, extracted data points, tags, and sentiment, is available through the Evaluations API.

### API

Use the **List evaluations** endpoint to pull all results for an agent:

```
curl 'https://api.gumloop.com/api/v1/agents/AGENT_ID/evaluations?page_size=100' \
  -H 'Authorization: Bearer YOUR_API_KEY'
```

Each result includes:

- `grade` (pass / needs_review / needs_attention)
- `criteria_results` with per-criterion pass/fail and rationale
- `data_results` with extracted values for each configured data point
- `applied_tags`
- `sentiment`
- `summary`

Paginate through all results using the `next_cursor` field. Filter by grade with `?grade=needs_attention`.

For a single evaluation, use the Retrieve evaluation endpoint. For aggregate metrics over time, use the Get metrics endpoint.

### Python

```python
import csv
import requests

API_KEY = "YOUR_API_KEY"
AGENT_ID = "YOUR_AGENT_ID"

# Paginate through all evaluations
cursor = None
all_evaluations = []
while True:
    params = {"page_size": 100}
    if cursor:
        params["cursor"] = cursor
    response = requests.get(
        f"https://api.gumloop.com/api/v1/agents/{AGENT_ID}/evaluations",
        headers={"" + "Authorization": f"Bearer {API_KEY}"},
        params=params,
    )
    response.raise_for_status()
    data = response.json()
    all_evaluations.extend(data["evaluations"])
    cursor = data.get("next_cursor")
    if not cursor:
        break

# Export data points to CSV
with open("eval_export.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["interaction_id", "grade", "sentiment", "summary"])
    for ev in all_evaluations:
        writer.writerow([ev["interaction_id"], ev["grade"], ev["sentiment"], ev["summary"]])
```