Letting Enterprises Define Their Own Contract Extractions at Docusign’s Scale

Docusign AI team members who contributed to this blog include: Ignacio Garrido, opens in a new tab applied scientist; Bruce Su, opens in a new tab senior applied scientist; Shengtao Ma, opens in a new tab lead applied scientist; and Ramachandra Kota, opens in a new tab senior director of applied science.

This is the third post in a four-part series on how Docusign processes agreements at scale. Read part 1 on our runtime architecture and part 2 on teacher-student distillation.

In our previous posts, we detailed how Docusign handles inference at the scale of a million documents a day, and how we use teacher-student distillation to train fast, highly accurate models for over 50 standard legal concepts.

For global enterprises, a catalog of 50 out-of-the-box standard fields is a powerful foundation, but not enough to address their specific business needs. The data our customers need to extract from their unstructured contracts varies widely. 

A maritime shipping company needs to pull Cargo Type and Port Locations. A healthcare network tracks Insurance Expiration Dates and Site of Practice. Retailers might be looking for Store Numbers, Monthly Fees, and Renewal Terms. A SaaS company needs to track an Hourly Cap on consulting fees. And a university might need to extract Student IDs and list the specific Faculties Involved. 

For the AI team, this presents a massive engineering gap: how do you extract thousands of highly specific, customer-defined concepts accurately? The typical solutions face considerable challenges at enterprise scale:

1. Train a custom model per customer: Standard machine learning (ML) practices would suggest gathering ground-truth data and fine-tuning a custom model for each customer’s specific fields. At our scale – 1.9 million users, 1 billion people, over a million agreements processed daily – this is economically impossible and bottlenecked by data collection. We cannot run a weeks-long teacher-student distillation cycle for a brand-new, bespoke field a customer wants.

2. Rigid schemas and regular expressions: Contracts are written in complex, highly variable natural language. A concept like "Opt out for AI model training" does not follow a predictable regex pattern and requires semantic reasoning.

3. The brute-force approach with an LLM: A third option is to simply pass the full contract, such as a 100-page PDF, to a frontier LLM along with the user's raw prompt. As we discussed in part 1, doing this for millions of daily documents misses strict latency targets, melts the infrastructure budget, and leaves the pipeline highly vulnerable to hallucinations due to the needle in a haystack problem.

We needed to build a system where a customer could define a completely novel concept at 2:00 PM, and accurately extract it across thousands of legacy contracts by 3:00 PM.

What we built

To solve this, we built Custom Extractions, a multi-stage retrieval and extraction pipeline designed for both precision and scale.

When a user defines a new concept in Docusign Agreement Manager, that configuration is immediately converted into a strict, machine-readable spec. An LLM first processes this spec for query understanding, translating the user's intent into diverse semantic search terms. Next, the pipeline runs retrieval across the document to retrieve only the most relevant text chunks. Finally, we trigger LLM inference on that tightly focused context window to produce a normalized, structured result.

This entire pipeline rides on the exact same AI Orchestrator, context engineering, and caching layers detailed in part 1. By layering Custom Extractions directly onto our existing architecture rather than building parallel infrastructure, we can execute this workflow across massive backlogs at the exact same scale as our standard models.

Figure 1: Custom Extractions Flow - From Broad Definitions to Structured Extractions at Scale

Reaching production accuracy without a per-field distillation cycle

High accuracy comes from the AI harness, not the AI model alone. Instead of relying on a single massive LLM call, we secure this precision by optimizing the definition, retrieval, and inference phases.

Transforming ambiguity into a strict definition

With Custom Extractions, we face the exact same ambiguity problem we saw with standard fields, but the customer writes the definition instead of a legal expert. A user might create a field for a "Discount" and define it simply as "a price reduction." Passing that raw specification directly to an extraction LLM creates a gap between the user's intent and what the pipeline actually needs to succeed, causing the model to struggle across structurally different documents.

To close this gap, Docusign engineers built an AI-assisted description rewrite utility that helps users better align their specifications with what our system expects. By democratizing specification design, this tool enables any user, including those not familiar with legal interpretation, to clearly articulate intent. When a user provides an ambiguous specification, this utility uses a background LLM to iteratively validate, expand, and rewrite it into a precise instruction before it reaches the core extraction pipeline. 

Figure 2.1: Incomplete definition of Early Termination Penalty | Figure 2.2: Refined definition of Early Termination Penalty

In addition to a solid definition, we also require users to provide a few examples directly from their own documents by highlighting both the target value and the specific context used to extract that value (the relevant sentence or sentences). We use this text as demonstrative and semantic anchors during the extraction process.

Figure 3: Select a few examples of Early Termination Penalty

Finally, beyond knowing what to look for, the pipeline needs strict boundaries on the output format. Users enforce this by defining the expected data type, such as text, number, date, or currency. For highly constrained data, users can select a "dropdown" field and provide a fixed list of valid values. We built a dedicated dropdown pipeline to force the LLM into one of those exact options, which preemptively avoids hallucinations.

The retrieval phase: finding the optimal context

The biggest challenge in the retrieval phase is finding the right amount of context. If the extraction LLM is fed too much text, such as a full document with 100+ pages, there are large cost implications alongside a potential for degradation in quality. However, restricting the retrieval to only the exact ground-truth matches also hurts performance, as it deprives the model of the surrounding context it needs to reason about the target value.

Through extensive experimentation, we developed a dynamic snippet-selection pipeline that retrieves the right-sized amount of context for accurate inference. Once the custom extraction is defined, a query generation model translates this machine-readable specification into a set of diverse semantic queries. These generated queries are then used to score and retrieve a carefully tuned number of context chunks, pulling in just enough surrounding sentences to support accurate LLM inference.

Because custom fields are far more unpredictable than standard fields, we could not rely on off-the-shelf embedding models to execute this semantic search. Instead, we fine-tuned our own semantic encoders in-house, specifically tailored to the complex legal language and document structures prevalent in our domain. Our encoder model improved evidence Recall@10 by 27% after fine-tuning. With these gains, this approach allows us to use smaller, faster, and cost-effective retrieval models that deliver comparable quality to larger off-the-shelf alternatives.

Figure 4: Influence of Retrieved Context on e2e Performance

Inference and validation

During the final inference phase, the LLM processes only the focused context window rather than the full document. Grounding the LLM in this tightly focused text reduces latency and minimizes hallucinations. Our experiments showed that this method utilized only 20% of the tokens on average compared to full document inputs.

Raw LLM output is never treated as the final answer. Every response is run through a rigorous post-processing layer: outputs are parsed and validated against a strict schema, and formatting is standardized so downstream systems always receive clean, predictable data. For constrained fields like dropdowns, deterministic validation rules force the output into a valid selection.

Together, retrieval, inference, and validation turn raw agreement text into structured, trustworthy data that teams can build on with confidence.

Validating field performance: test for accuracy (TFA) 

Before this inference pipeline is ever deployed across millions of agreements, the field configuration must be validated. Users run a process known as Test for Accuracy (TFA) on a small, controlled sample of their own documents, manually verifying or correcting the normalized outputs. This gives users complete visibility into the pipeline’s performance before activating bulk processing across their documents.

Layering custom agreement types alongside standard OOTB types

The system also needs to know where to look. A highly specific custom field, like a “Site of Practice,” is entirely irrelevant to a standard Non-Disclosure Agreement. Running every custom field across every document in a massive enterprise repository wastes computation and increases the risk of false positives. 

Similarly, our current standard workflow supports 59 standard types out of the box (OOTB), but customer documents can fall outside those standard types.

To address the above limitation of the existing system, we provide another level of abstraction: Custom Agreement Types. 

Figure 5: Onboarding and Feedback Loop for Custom Agreement Types

During the onboarding flow, the user provides the custom agreement type, definition, and associated examples. Before publishing, TFA validates the user-provided prompt and examples. Once a user publishes the type, it is enabled for AI detection on new documents. An active learning workflow for updating the definition can be triggered when the user provides feedback. 

After publishing, these custom types are integrated directly alongside our standard OOTB types into a single, unified classification flow. By layering custom types with standard ones, the system ensures that highly specialized custom extractions are only triggered on the exact subset of documents where they actually belong.

Figure 6: Classifying a New Document Against Standard and Custom Agreement Types

The user-uploaded document, after the necessary upstream parsing, is preprocessed by limiting token size via snippet selection and truncation. We apply a combination of embedding and regex retrieval to select the top candidates for agreement types. Few-shot examples are selected based on similarity score to the user document. The preprocessed document, candidates, examples, and system prompt are fed into the LLM to infer the prediction of the document. A final step of validation and heuristic rule is done to ensure high precision and reduce hallucination from the LLM.

Evaluating custom extractions with no pre-built ground truth

In Post 2, we relied on triple-annotated datasets created by legal SMEs to rigorously evaluate model accuracy. A custom field created by a user today has zero pre-labeled ground truth. How do you measure and prove accuracy for a bespoke concept that did not exist yesterday?

We divide our evaluation strategy into offline metrics for pipeline tuning and online metrics for production reporting.

For offline evaluation, we rely on two distinct data sources. First, we repurpose our existing internal annotated datasets from standard extractions, treating them as custom fields to benchmark baseline performance. Second, to evaluate novel concepts, we generate synthetic datasets by leveraging LLMs to derive candidate fields from real documents using our internal taxonomy. Finally, due to the open-ended nature of custom extractions, naive string-matching evaluation usually fails. We built a suite of LLM-as-a-judge utilities to properly evaluate the semantic correctness of these extractions, while also accounting for the non-deterministic variance of LLMs across our experiments.

While offline metrics validate the pipeline in the lab, online evaluation measures real-world performance in production. As customers create and activate custom fields, they also provide direct feedback (e.g., accept or revise). Tracking revision rates and acceptance rates gives us a continuous, real-time measure of pipeline health. We observed an upward trending telemetry of acceptance rates above 90% and anecdotes from customer interviews, which confirm that even without a single round of fine-tuning, the adaptive retrieval pipeline delivers a robust performance and self-improves over time.

Applying custom extractions at scale

In Post 1, we detailed the AI Orchestrator and how we decoupled document preprocessing from model inference. Custom Extractions integrate directly with this architecture.

When a user activates a new custom field across a backlog of 10,000 legacy contracts, the platform does not start from scratch. We do not re-run rasterization, general OCR, or document layout analysis. Instead, the custom extraction pipeline pulls the exact same pre-processed text and layout artifacts from our ephemeral blob storage cache that our standard models consume.

Because custom fields are simply layered on top of standard fields, obligations, and clause analysis, we re-derive almost nothing from the source file. The custom pipeline uses the same orchestrator, the same message queueing lanes to separate live traffic from bulk background jobs, and the same efficient hardware routing. This shared infrastructure lets a user-defined field maintain the same latency targets and minimal marginal costs as extracting a standard out-of-the-box field.

Results

To date, hundreds of customer accounts have enabled custom extractions. The time-to-value for a custom data point has dropped from weeks of data collection and model fine-tuning to just a few minutes from the moment a user defines the field to their first production extraction.

While a zero-day custom field defined by a user this afternoon naturally exhibits higher variance than a fully distilled standard model built by our applied science team, custom extractions still achieve a strong baseline accuracy even without a distillation cycle. More importantly, they maintain the exact same fractional-cent cost profile and sub-minute latency as the rest of the Docusign AI platform.

What's next

Because custom fields start with the least amount of initial signal, they stand to benefit the most from user corrections. 

When a user corrects a missed value in production, how does the system actually learn from it without requiring a full retraining cycle? 

In our fourth and final post, we will dive into how we handle feedback, exploring how user corrections automatically generate persistent rules that make Docusign’s AI contract management pipeline permanently smarter over time.