S6 · Structured Data Extraction

spans 35% of the blueprint D4 20% D5 15% ~15 items per lab on the exam 20 items in the bank

Domains this lab exercises

Expected items apply the blueprint weight to a 60-item exam. Bank items are what exists for this lab today — a coverage figure, never a score.

Domain	Weight	Expected items	Statements	Bank items for this lab
D4 Prompt Engineering & Structured Output	20%	12.0	6	13
D5 Context Management & Reliability	15%	9.0	6	7

Task statements in play 7 of 30

Each statement is a published objective; every practice item on this site cites one. Expand a row for the guide's key facts and, where the guide names them, the anti-patterns that supply that statement's distractors.

ID	Statement	Domain	Bank items
4.1	Design prompts with explicit criteria to improve precision and reduce false positives Keys The importance of explicit criteria over vague instructions (e.g., "flag comments only when claimed behavior contradicts actual code behavior" vs "check that comments are accurate") How general instructions like "be conservative" or "only report high-confidence findings" fail to improve precision compared to specific categorical criteria The impact of false positive rates on developer trust: high false positive categories undermine confidence in accurate categories Writing specific review criteria that define which issues to report (bugs, security) versus skip (minor style, local patterns) rather than relying on confidence-based filtering Temporarily disabling high false-positive categories to restore developer trust while improving prompts for those categories Defining explicit severity criteria with concrete code examples for each severity level to achieve consistent classification Anti-patterns How general instructions like "be conservative" or "only report high-confidence findings" fail to improve precision compared to specific categorical criteria	D4	3
4.2	Apply few-shot prompting to improve output consistency and quality Keys Few-shot examples as the most effective technique for achieving consistently formatted, actionable output when detailed instructions alone produce inconsistent results The role of few-shot examples in demonstrating ambiguous-case handling (e.g., tool selection for ambiguous requests, branch-level test coverage gaps) How few-shot examples enable the model to generalize judgment to novel patterns rather than matching only pre-specified cases The effectiveness of few-shot examples for reducing hallucination in extraction tasks (e.g., handling informal measurements, varied document structures) Creating 2-4 targeted few-shot examples for ambiguous scenarios that show reasoning for why one action was chosen over plausible alternatives Including few-shot examples that demonstrate specific desired output format (location, issue, severity, suggested fix) to achieve consistency Providing few-shot examples distinguishing acceptable code patterns from genuine issues to reduce false positives while enabling generalization Using few-shot examples to demonstrate correct handling of varied document structures (inline citations vs bibliographies, methodology sections vs embedded details) Adding few-shot examples showing correct extraction from documents with varied formats to address empty/null extraction of required fields	D4	3
4.3	Enforce structured output using tool use and JSON schemas Keys Tool use (tool_use) with JSON schemas as the most reliable approach for guaranteed schema-compliant structured output, eliminating JSON syntax errors The distinction between tool_choice: "auto" (model may return text instead of calling a tool), "any" (model must call a tool but can choose which), and forced tool selection (model must call a specific named tool) That strict JSON schemas via tool use eliminate syntax errors but do not prevent semantic errors (e.g., line items that don't sum to total, values in wrong fields) Schema design considerations: required vs optional fields, enum fields with "other" + detail string patterns for extensible categories Defining extraction tools with JSON schemas as input parameters and extracting structured data from the tool_use response Setting tool_choice: "any" to guarantee structured output when multiple extraction schemas exist and the document type is unknown Forcing a specific tool with tool_choice: {"type": "tool", "name": "extract_metadata"} to ensure a particular extraction runs before enrichment steps Designing schema fields as optional (nullable) when source documents may not contain the information, preventing the model from fabricating values to satisfy required fields Adding enum values like "unclear" for ambiguous cases and "other" + detail fields for extensible categorization Including format normalization rules in prompts alongside strict output schemas to handle inconsistent source formatting	D4	5
4.4	Implement validation, retry, and feedback loops for extraction quality Keys Retry-with-error-feedback: appending specific validation errors to the prompt on retry to guide the model toward correction The limits of retry: retries are ineffective when the required information is simply absent from the source document (vs format or structural errors) Feedback loop design: tracking which code constructs trigger findings (detected_pattern field) to enable systematic analysis of dismissal patterns The difference between semantic validation errors (values don't sum, wrong field placement) and schema syntax errors (eliminated by tool use) Implementing follow-up requests that include the original document, the failed extraction, and specific validation errors for model self-correction Identifying when retries will be ineffective (e.g., information exists only in an external document not provided) versus when they will succeed (format mismatches, structural output errors) Adding detected_pattern fields to structured findings to enable analysis of false positive patterns when developers dismiss findings Designing self-correction validation flows: extracting "calculated_total" alongside "stated_total" to flag discrepancies, adding "conflict_detected" booleans for inconsistent source data	D4	5
4.5	Design efficient batch processing strategies Keys The Message Batches API: 50% cost savings, up to 24-hour processing window, no guaranteed latency SLA Batch processing is appropriate for non-blocking, latency-tolerant workloads (overnight reports, weekly audits, nightly test generation) and inappropriate for blocking workflows (pre-merge checks) The batch API does not support multi-turn tool calling within a single request (cannot execute tools mid-request and return results) custom_id fields for correlating batch request/response pairs Matching API approach to workflow latency requirements: synchronous API for blocking pre-merge checks, batch API for overnight/weekly analysis Calculating batch submission frequency based on SLA constraints (e.g., 4-hour windows to guarantee 30-hour SLA with 24-hour batch processing) Handling batch failures: resubmitting only failed documents (identified by custom_id) with appropriate modifications (e.g., chunking documents that exceeded context limits) Using prompt refinement on a sample set before batch-processing large volumes to maximize first-pass success rates and reduce iterative resubmission costs	D4	4
5.5	Design human review workflows and confidence calibration Keys The risk that aggregate accuracy metrics (e.g., 97% overall) may mask poor performance on specific document types or fields Stratified random sampling for measuring error rates in high-confidence extractions and detecting novel error patterns Field-level confidence scores calibrated using labeled validation sets for routing review attention The importance of validating accuracy by document type and field segment before automating high-confidence extractions Implementing stratified random sampling of high-confidence extractions for ongoing error rate measurement and novel pattern detection Analyzing accuracy by document type and field to verify consistent performance across all segments before reducing human review Having models output field-level confidence scores, then calibrating review thresholds using labeled validation sets Routing extractions with low model confidence or ambiguous/contradictory source documents to human review, prioritizing limited reviewer capacity	D5	5
5.6	Preserve information provenance and handle uncertainty in multi-source synthesis Keys How source attribution is lost during summarization steps when findings are compressed without preserving claim-source mappings The importance of structured claim-source mappings that the synthesis agent must preserve and merge when combining findings How to handle conflicting statistics from credible sources: annotating conflicts with source attribution rather than arbitrarily selecting one value Temporal data: requiring publication/collection dates in structured outputs to prevent temporal differences from being misinterpreted as contradictions Requiring subagents to output structured claim-source mappings (source URLs, document names, relevant excerpts) that downstream agents preserve through synthesis Structuring reports with explicit sections distinguishing well-established findings from contested ones, preserving original source characterizations and methodological context Completing document analysis with conflicting values included and explicitly annotated, letting the coordinator decide how to reconcile before passing to synthesis Requiring subagents to include publication or data collection dates in structured outputs to enable correct temporal interpretation Rendering different content types appropriately in synthesis outputs—financial data as tables, news as prose, technical findings as structured lists—rather than converting everything to a uniform format	D5	6

The lab

Rendered from certs/ccar-f-architect-foundations/scenarios/s6.md — the lab document is the source, this page is a view of it.

Primary domains: D4 Prompt Engineering & Structured Output (20%) · D5 Context Management & Reliability (15%) — spans 35%

One of only two routes into D4 (with s5.md). The least glamorous lab, and joint-cheapest route into a 20% domain — do not skip it because the agent scenarios are more interesting.

Brief

A structured data extraction system built with Claude. It extracts information from unstructured documents, validates the output against JSON schemas, and maintains high accuracy. It must handle edge cases gracefully and integrate with downstream systems.

Task statements in play

ID	Statement
4.1	Design prompts with explicit criteria to improve precision and reduce false positives
4.2	Apply few-shot prompting to improve output consistency and quality
4.3	Enforce structured output using tool use and JSON schemas
4.4	Implement validation, retry, and feedback loops for extraction quality
4.5	Design efficient batch processing strategies
5.5	Design human review workflows and confidence calibration
5.6	Preserve information provenance and handle uncertainty in multi-source synthesis

The decisions it tests

Enforce structure, don't request it. Task statement 4.3 says it plainly: structured output is enforced through tool use and JSON schemas, not by asking politely for JSON in the prompt. A schema the model must satisfy is a constraint; "respond in valid JSON" is a hope.

Validation, retry, and feedback loops. Schema validation failure is a signal to feed back and retry, not an exception to swallow. Defensive parsing on the way out, because a confidently malformed response is the normal failure — not a rare one.

Never trust stated confidence. The published sample items across these guides hammer this: a model citing a specific-looking subsection number can be fabricating it, and self-reported confidence is not an accuracy signal. Validate factual claims against the source. This is the whole point of task statement 5.5, confidence calibration — and it pairs with routing low-confidence extractions to human review rather than accepting them.

Batching is a design choice. The Message Batches API: 50% cost savings, up to 24-hour processing window, no guaranteed latency SLA. Appropriate for non-blocking, latency-tolerant workloads (overnight reports, weekly audits, nightly test generation) and inappropriate for blocking workflows (pre-merge checks). It does not support multi-turn tool calling within a single request, and custom_id correlates request/response pairs — which is also how you resubmit only the failed documents. Parallel synchronous requests get you none of the cost saving.

Provenance under uncertainty. When merging extractions from several documents, keep field-level attribution — which document and page each value came from — and represent uncertainty explicitly rather than silently picking a winner.

Reference build

A schema-enforced extractor using tool use, not prompt-requested JSON
Three deliberately malformed documents — a missing field, a contradictory duplicate, an unparseable date — with the graceful degradation for each demonstrated, not described
A validation → feedback → retry loop with a bounded retry count and a defined give-up path
Confidence thresholds routing low-confidence records to a human review queue
Field-level provenance in the output, so every value traces to a source and page
A batch path for the high-volume case, with the cost rationale written down

Traps

Hope over enforcement — "return valid JSON" in the prompt instead of a schema.
Trust the model's confidence — accepting a self-rated confidence score as a quality gate, or sending a cited figure onward without verifying it against the source.
Swallowing validation failures instead of feeding them back.
Unbounded retries with no give-up path.
Parallel synchronous calls for an overnight batch job, where Batches is the answer.
Dropping provenance during multi-document merge.