The recent Vals Legal AI Report (VLAIR) reinforced a critical truth: While AI solutions have advanced greatly, contract redlining remains one of the toughest challenges for legal AI. Below, we detail the reasons why—and how we’re tackling them at Dioptra.
The Vals Legal AI Report compared AI tools—including Harvey Assistant, CoCounsel, and Vincent—against human lawyers on tasks ranging from document summarization to contract redlining. AI systems performed well overall, but struggled significantly with redlining. Lawyers scored about 79.7% accuracy for redlining tasks, while the two AI tools assessed managed just mid- to high-50s on average. This performance gap underscores the unique challenges of automating contract redlining skills.
The team behind Dioptra has been building and training sophisticated AI systems for the past decade. Through our own experience developing our redlining Agent, we've identified four core challenges making redlining uniquely challenging for AI:
Lawyer-quality contract redlining combines strategic judgment—based on specific risk profiles, market expectations, regulatory considerations, and business priorities—with precise, consistent, flexible, execution. To suggest meaningful, elegant, and accurate redlines, AI must grasp numerous layers of context—far beyond what standard text analysis can offer. Each of the typical resources used to guide contract reviews—such as playbooks, precedents, and standard forms—fall short individually when used by AI systems to deliver precise and consistent redlines. Traditional playbooks often assume extensive background knowledge that AI simply can't infer. Precedents are often inconsistent, standard forms rarely highlight what's truly important, and generic playbooks often overlook the distinct needs of the business.
It is extremely difficult today for teams building AI redlining systems to measure the accuracy of their systems. This measurement void results from three compounding effects: The confidentiality of contracts drastically limits the availability of high-quality, labeled data needed for AI training. The industry, currently, lacks a standardized framework for annotating redlines, that accounts for the wide variety of language used to achieve accurate outcomes and the highly individualized redlining styles. Finally the currently available AI tooling hasn’t caught up yet, making it extremely hard for the teams building AI redlining products to confidently measure and evaluate the accuracy of their solutions.
AI systems face a challenging tradeoff: highly specialized models can accurately match organization-specific language but lack versatility (both in terms of playbook customization and adaptability to third party papers), while generalized models struggle to meet the precise requirements of complex organizations. Finding this balance—between accuracy and versatility—is a critical obstacle for effective scalable redlining products.
Most LLMs, today, are primarily optimized for general use or coding tasks—not complex legal redlining, which requires the capability to repurpose and manipulate the existing language to achieve a different, desired objective (we’ve been referring to redlining as a different modality at Dioptra). Moreover, the new “reasoning” LLMs are biased to blab and spit out reasoning; which fails to deliver pinpoint accuracy and surgical changes. With experts also noting LLM scaling is hitting a wall, it's clear that out of the box LLMs won’t cut it in the near future.
At Dioptra, we tackle these challenges through an innovative approach: Knowledge Distillation. Our AI Agent "learns" by analyzing clients'past negotiated contracts and playbooks when available, capturing their unique risk tolerances, drafting styles, and negotiation preferences. This distilled insight is compiled into client-owned "playbooks", allowing our AI to produce tailored and accurate redlines.
VLAIR clearly illustrates that specialized AI can yield remarkable efficiency gains. By automating standard redlining tasks, AI allows professionals to focus on strategic judgment and complex decision-making. The emerging AI-human collaborative model will soon become a legal industry norm, driving faster processes, reduced errors, and deeper strategic insights. At Dioptra, we remain committed to leading this balanced integration of AI-based precision and human expertise.
