Classify 100,000 Text Rows in SQL: Jev Benchmarks and a Practical Pipeline

Analytics and data engineering teams often face a table with an unstructured text column: customer feedback, support complaints, product descriptions, or error logs. SQL can group identical text and apply string rules, but grouping records by the meaning of a narrative requires an additional classification step.

A generative Large Language Model (LLM) can extract categorical labels, but throughput and cost depend on the model, input length, batching, and provider limits. For fixed-label tasks, a bounded scoring model is another option worth evaluating.

A different approach is System 1 models. In its September 2026 benchmark, MotherDuck reports that Jev classified 100,000 articles sampled from AG News’s training split in 40 seconds for $0.50, with 89% accuracy against dataset labels. Its gpt-5.6-terra comparison took 31 minutes 59 seconds and cost $37.58. These are vendor-reported results on that four-class news dataset, not a guarantee for consumer complaints or every SQL workload.

Here is how System 1 classification works in modern SQL pipelines, how it differs from traditional LLMs, and how to implement it with practical validation safeguards.

1. The Bottleneck: Unstructured Text in Analytical Pipelines

For a separate worked example, consider consumer complaints from the public CFPB Consumer Complaint Database. Complaint records include an ID and a company response. The CFPB ceased publishing complaint narratives in the live database on August 14, 2026. Previously published narratives are available in its FOIA narratives archive, covering complaints received through August 14, 2026. This worked example uses historical narrative data rather than a continuing feed of public narratives. The question for this pipeline is: What specific outcome did the consumer request? The categories below are a proposed taxonomy, not labels or measured results from the MotherDuck benchmark.

Three alternative approaches involve different trade-offs:

1. Regex and CASE WHEN statements: Useful for explicit patterns, but rules may miss paraphrases such as “reverse the fee,” “waive the charge,” or “credit my balance.” Measure coverage on labeled examples before relying on them.

2. Custom fine-tuned classifiers: A BERT-style encoder can classify text efficiently, but a custom deployment may require labeled training data and model maintenance. The amount of labeling and infrastructure depends on the task and deployment.

3. Generative LLMs: Flexible for taxonomy discovery and open-ended tasks, but they can also misclassify. MotherDuck’s benchmark measured model-specific runtime and cost; those figures should not be generalized to all models or datasets.

Vendor-reported AG News benchmark, September 2026 — 100,000 articles:

Source: MotherDuck’s AG News benchmark. These results do not measure the consumer-complaint pipeline below. MotherDuck’s published example reports NULL predictions separately and excludes them from its accuracy calculation.

2. System 1 vs. System 2: Shifting the Paradigm

The distinction between generative LLMs and specialized scoring models mirrors Daniel Kahneman’s cognitive framework from Thinking, Fast and Slow:

• Generative LLMs: Commonly generate an answer token by token, including when the output is constrained to JSON.

• Scoring models like Jev: TypeSafe describes Jev as returning typed decisions through parallel evaluation rather than free-text generation. “System 1” and “System 2” are an architectural analogy here, not proof that either model implements human cognition or that every question uses exactly one forward pass.

Generative model: input text -> generated tokens -> parsed result
Bounded scorer: input text + allowed labels -> typed decision + probabilities
(Schematic architecture, not a measured latency comparison.)

Because Jev’s closed questions evaluate the candidates provided in the query, several engineering advantages emerge:

• Bounded labels: A valid Choice response selects from the supplied labels. It can still select the wrong label; schema validity is not factual correctness.

• Confidence metrics: Choice and Score include probabilities and a confidence measure derived from their distribution. Validate thresholds against labeled data.

• SQL integration: MotherDuck exposes typed results for aggregation and filtering. Query latency still depends on data size, inference, and execution.

3. The Three SQL Query Primitives

MotherDuck’s preview prompt_jev() function exposes Jev’s three question types in SQL. It is available on supported paid plans and is disabled in eu-central-1 and eu-west-1. Organizations in ap-northeast-1 and ap-southeast-2 can call it, but requests are routed to TypeSafe’s servers in the United States rather than processed in the organization’s own region. It is not a built-in function in standalone DuckDB; a custom UDF would need a separate implementation.

Primitive A: Choice (Categorical Classification)

Used when every row must be assigned to exactly one category from a known taxonomy.

• SQL Role: Feeds directly into GROUP BY and cohort segmentation.

• Output: A struct containing the winning label string, per-label probabilities, and a confidence score between 0.0 and 1.0.

Primitive B: Noul (Boolean Probability)

Used to evaluate whether an assertion about the text is true or false.

• SQL Role: Feeds into WHERE clauses for probability-based filtering.

• Output: A probability score from 0.0 to 1.0 that MotherDuck describes as calibrated, representing the model’s assessment that the statement holds for the input. Validate calibration on your own task.

Primitive C: Score (Ordinal or Continuous Ranking)

Used to rank rows along a subjective scale (such as urgency, severity, or sentiment).

• SQL role: Use a Score for ordering or thresholding against an ordered rubric.

• Output: A probability-weighted position across the supplied levels, which may be fractional. A three-level rubric spans 0 to 2; that does not make the number a calibrated physical magnitude.

4. End-to-End Implementation Workflow

Transforming an unparsed text column into an operational analytics table follows a four-step lifecycle:

[ Step 1: LLM Taxonomy Sampling ]
        │ (Sample 40-50 rows to discover labels + descriptions)
        ▼
[ Step 2: Scale SQL Classification ]
        │ (Apply System 1 scorer across 100k rows in batch)
        ▼
[ Step 3: Confidence Triage & Review Queue ]
        │ (Validate threshold -> Accepted cohort or review; NULL -> Retry)
        ▼
[ Step 4: Downstream OLAP Analytics ]
        │ (Standard SQL GROUP BY, aggregations, trend analysis)

Step 4.1: Discover the Taxonomy Using a Small LLM Sample

Do not guess categories in a vacuum. Instead, use an LLM for what it does best: synthesizing open-ended text.

Start with a manageable sample—for example, 40 to 50 rows—and ask a generative model to propose a small set of categories. Then refine overlapping labels and test coverage on a separate, representative labeled sample. That initial sample does not establish 95% coverage.

For consumer complaints, an illustrative taxonomy looks like this. Some outcomes overlap, so define tie-breaking rules for the primary request and retain an “other” or “unclear” category:

1. refund_or_charge_reversal: Consumer requests returned money, fee waivers, or reimbursements.

2. correct_or_remove_credit_report: Consumer requests correction or deletion of inaccurate credit bureau entries.

3. investigate_transaction_error: Consumer flags unauthorized transactions, billing discrepancies, or deposit errors.

4. stop_harassment_or_contact: Consumer demands collection calls or messages cease immediately.

5. address_identity_theft: Consumer reports fraudulent accounts opened using compromised credentials.

6. explain_account_decision: Consumer requests clarification on denied applications, closures, or interest rate hikes.

7. other: Fallback for edge cases outside the defined taxonomy.

Step 4.2: Execute Bounded Classification in SQL

In MotherDuck, define a constant taxonomy using the documented prompt_jev function and named choice parameter. The following SQL assumes you have loaded historical records with non-null narratives from the FOIA archive into raw_complaints; field names are aliases in that table.

CREATE TABLE complaint_classified AS
SELECT 
    complaint_id,
    company_response,
    narrative,
    prompt_jev(
        narrative,
        'What outcome is the consumer asking for?',
        choice := [
            {'label': 'refund_or_charge_reversal', 
             'description': 'Consumer requests returned money, fee waivers, or reimbursements.'},
            {'label': 'correct_or_remove_credit_report', 
             'description': 'Consumer requests correction or deletion of inaccurate credit bureau entries.'},
            {'label': 'investigate_transaction_error', 
             'description': 'Consumer flags unauthorized transactions or billing discrepancies.'},
            {'label': 'stop_harassment_or_contact', 
             'description': 'Consumer demands collection calls or messages cease.'},
            {'label': 'address_identity_theft', 
             'description': 'Consumer reports fraudulent accounts opened using stolen identities.'},
            {'label': 'explain_account_decision', 
             'description': 'Consumer requests clarification on denied credit or closed accounts.'},
            {'label': 'other', 
             'description': 'Any request not fitting the categories above.'}
        ]
    ) AS outcome
FROM raw_complaints;

The returned outcome column is a typed struct containing:

• outcome.choice: The selected label.

• outcome.confidence: A distribution-derived confidence value, not a guarantee of correctness.

• outcome.probabilities: MotherDuck’s array of structs with value and probability fields, one per allowed label. The direct TypeSafe API instead returns a map.

The MotherDuck reference shows this three-label routing example for a message about a duplicate charge. It illustrates the SQL struct shape serialized as JSON; it is not an observed output from the complaint pipeline above:

{
  "choice": "billing",
  "confidence": 0.91,
  "probabilities": [
    {
      "value": "billing",
      "probability": 0.94
    },
    {
      "value": "technical",
      "probability": 0.04
    },
    {
      "value": "sales",
      "probability": 0.02
    }
  ]
}

Best Practice: Store Taxonomies in Managed Tables

For production, keep a versioned taxonomy in a managed table. MotherDuck requires the criteria to be constant for each query; load one version into a query variable before calling the function:

-- Maintain a versioned taxonomy table
CREATE TABLE taxonomies (
    taxonomy_name VARCHAR,
    version INT,
    label VARCHAR,
    description VARCHAR
);

-- In engines like DuckDB, aggregate categories into a runtime variable
SET VARIABLE complaint_labels = (
    SELECT list({'label': label, 'description': description} ORDER BY label)
    FROM taxonomies
    WHERE taxonomy_name = 'complaint_outcomes' AND version = 1
);

-- Execute classification cleanly
SELECT 
    complaint_id,
    prompt_jev(
        narrative, 
        'What outcome is the consumer asking for?', 
        choice := getvariable('complaint_labels')
    ) AS outcome
FROM raw_complaints;

5. Quality Control: Handling Uncertainty with SQL Review Queues

Never trust model outputs blindly. TypeSafe’s confidence documentation explains how Choice and Score summarize their probability distributions. Select an acceptance threshold using labeled data for your task; confidence alone is not an accuracy guarantee.

Measure the confidence distribution on your own complaint dataset: mean confidence, the share above your validated threshold, and the low-confidence tail. The SQL below uses 0.80 as an illustrative threshold, not a measured optimum.

Isolating Uncertain Rows

Rather than accepting misclassifications into production BI dashboards, construct an automated review queue directly in SQL:

-- Illustrative threshold: validate on labeled data before accepting rows
CREATE VIEW vw_clean_complaints AS
SELECT 
    complaint_id,
    outcome.choice AS requested_outcome,
    outcome.confidence
FROM complaint_classified
WHERE outcome.confidence >= 0.80;

-- Low-confidence review queue: Route to human reviewers or secondary validation
CREATE VIEW vw_review_queue AS
SELECT 
    complaint_id,
    narrative,
    outcome.choice AS tentative_choice,
    outcome.confidence,
    outcome.probabilities
FROM complaint_classified
WHERE outcome IS NULL OR outcome.confidence IS NULL OR outcome.confidence < 0.80
ORDER BY outcome.confidence ASC;

For an ambiguous complaint, inspect the per-label probabilities and the underlying narrative. If a request spans identity theft and credit-report correction, review the taxonomy and tie-breaking rule rather than assuming the highest-scoring label is correct.

Multi-Model Agreement Benchmarking

For validation, compare predictions with independently labeled examples. Agreement between two models can help identify difficult rows, but it is not ground truth: both can make the same mistake. Report the sample, label definitions, model versions, and exclusions alongside any accuracy or agreement result.

MotherDuck’s published accuracy result uses AG News ground-truth labels. Do not treat it as evidence of parity on consumer-complaint outcomes; evaluate that separate task before deployment.

6. Downstream Analytics: The Power of Typed Columns

After labels are materialized, downstream SQL aggregation can reuse them without repeating model inference. Runtime depends on the stored data and query plan. For continuous ingestion from another authorized narrative source, store a classification version and track failed or missing labels before feeding BI reports.

-- Analyze monetary relief rates across customer request categories
SELECT 
    outcome.choice AS requested_outcome,
    COUNT(*) AS total_complaints,
    ROUND(100.0 * AVG((company_response = 'Closed with monetary relief')::INT), 1) AS pct_monetary_relief
FROM complaint_classified
GROUP BY 1
ORDER BY total_complaints DESC;

Read Your Aggregation Results

The query reports the share of non-null company-response values marked “Closed with monetary relief” within each predicted request category. DuckDB’s aggregate rules exclude NULL inputs from AVG, while COUNT(*) includes all rows; check missing responses before comparing these counts and rates. That is a company-response field, not an independently measured compensation amount. Interpret the aggregates only after validating labels, missing narratives, and the dataset’s scope.

7. Architectural Decision Matrix

When should you deploy System 1 bounded scorers versus generative LLMs in your data stack?

            [ New Text Pipeline Task ]
                        │
 Does the task require generating new text,
 summaries, or freeform reasoning?
                /        \
              Yes         No
              /            \
[ Generative LLM ]     Can you define a fixed list of
(GPT-5 / Claude)       outcomes or a scoring rubric?
                               /        \
                             Yes         No
                             /            \
              [ System 1 Model ]    [ Discovery Phase ]
              (SQL Bounded Scorer)  (LLM Cluster Sample)

• Use Generative LLMs When:

* Drafting personalized email responses.

* Writing unstructured executive summaries.

* Designing initial category taxonomies from cold data.

• Use System 1 Models When:

* Categorizing thousands or millions of records against a fixed schema.

* Filtering high-volume data streams based on keyword intent or topic match.

* Assigning priority or urgency scores to incoming tickets.

* Evaluating a fixed-label task against cost and latency budgets, after checking where input data is processed and whether that fits your privacy requirements.

Practical Next Steps for Data Teams

Pairing generative models for taxonomy exploration with bounded scoring for fixed-label execution is a useful architecture to evaluate. MotherDuck’s news benchmark demonstrates the potential, while your own labeled sample, runtime measurements, costs, and review queue determine whether it fits your workload.