Astra for Law vs Claude, Gemini, Harvey and Other Legal AI: What the Benchmarks Really Prove
JSH Law | Legal AI and access to justice
England and Wales perspective | Sources reviewed 17 November 2026
A system can lead the legal AI leaderboard and still fall short on most complete legal assignments.
OpenAI’s latest Astra for Law announcement is a significant technological development. But the question for JSH Law is not which company wins the publicity battle. It is which tools can deliver accurate, properly sourced and safely usable legal assistance—and whether litigants in person benefit.
1. What has OpenAI actually announced?
On 17 November 2026, OpenAI announced an updated Astra for Law: a legal-specialist configuration of GPT-6.2 Astra, supplemented by legal research instructions and an expanded search index. OpenAI reports that the index now spans more than 310 million URLs, including greater coverage outside the United States. It says every ChatGPT Enterprise customer may now request access through the Trusted Access Program, including an API route. That is not universal access for individual ChatGPT subscribers or litigants in person.
The claims include better end-to-end legal work, stronger research, fewer material hallucinations and greater token efficiency. They deserve examination, not automatic endorsement. Astra for Law is also a model configuration, not a replacement for a court-approved process, a licensed legal research database or a lawyer who can take professional responsibility.
2. How Astra, Claude, Gemini, Grok and Muse actually perform
Artificial Analysis’s Harvey LAB-AA v1.1 is a useful independent starting point. It tests 120 demanding professional assignments across 24 areas of law. Agents receive a partner-style instruction and source files, prepare deliverables, and are scored by a three-model judge panel. A separate hallucination check assesses whether material statements are supported by the task documents. The headline hallucination-gated all-pass rate awards credit only when every required criterion passes and no material hallucination is identified.
| Model (test setting) | Full pass | Criteria pass | Material hallucinations / task | Cost / task |
|---|---|---|---|---|
| Astra for Law 6.2 (Max) | 20.8% | 94.4% | 0.03 | $7.33 |
| GPT-6.2 Astra (Max) | 15.0% | 93.8% | 0.06 | $6.35 |
| Muse Wonder Full (Max) | 13.1% | 95.5% | 0.62 | Not listed |
| GPT-6.2 Sol (Max) | 11.1% | 91.5% | 0.05 | $1.92 |
| Grok 4.7 (Xhigh) | 9.4% | 92.8% | 0.74 | $8.20 |
| Muse Spark 1.3 (Max) | 8.9% | 96.0% | 1.68 | $3.43 |
| Claude Fable 5.2 (Max; fallback enabled) | 7.2% | 91.0% | 0.75 | $39.84 |
| Claude Opus 5.5 (Max; fallback enabled) | 4.2% | 90.0% | 0.59 | $17.93 |
| Gemini 4 Argon (High) | 3.9% | 94.8% | 3.82 | $6.29 |
The results show a genuine improvement for Astra for Law, but also some inconvenient facts for simplistic marketing. Meta’s Muse Wonder Full satisfies a higher proportion of individual criteria (95.5%) than Astra for Law (94.4%), although Astra wins on fully completed assignments without material hallucinations. GPT-6.2 Sol costs markedly less in this benchmark than the top-scoring legal configuration and may be more proportionate for lower-risk work. Neither observation means Muse or Sol is necessarily a better choice for court research.
3. What the impressive headlines do—and do not—prove
“59% better than Muse”
The 20.8% all-pass score is approximately 59% higher, relatively, than Muse Wonder Full’s 13.1%; the absolute difference is 7.7 percentage points. That does not mean 59% better advice, legal reasoning, win rate or case outcomes. In fact, in this strict test most complete tasks did not earn full credit even for the leader. Some failed answers may be partially useful, but completing a court assignment is different from being nearly right.
“20 times fewer hallucinations than Claude”
The benchmark reports 0.03 material hallucinations per assignment for Astra for Law and 0.59 for Claude Opus 5.5—around twenty times as many for that Claude configuration under these conditions. These are averages of identified errors against provided benchmark files. They do not predict the likelihood of a false citation in any individual UK case; nor should the result be applied without qualification to all Claude variants. Artificial Analysis employs AI-assisted evaluation, with transparent methodology, but machine judges are not infallible.
“Best legal research”
OpenAI also cites a 53.8% fully correct score (112/208 questions) for Astra for Law on Vals AI’s Legal Research Bench on 16 November 2026. The questions concern US federal and state law. The score cannot be carried across into English children proceedings, the Family Procedure Rules, Welsh practice, the interpretation of Cafcass records or safeguarding decisions. An expanded international index is welcome; jurisdiction-specific competency still needs to be demonstrated.
“Cheaper”
The dollar figures in the model table are estimated inference cost per benchmark assignment, not a legal software subscription, a price available to an individual or the total cost of deploying the product. They exclude licensed content, checking citations, data governance, human review and implementation costs. No credible buying decision can be based on the benchmark cost column alone.
4. Which other legal AI products should we compare?
An underlying model and a finished legal AI product are not interchangeable. Specialist products may add licensed law libraries, citators, knowledge management, document controls, permissions and workflow oversight. The all-pass scores above do not rate Harvey, Legora, CoCounsel, Lexis or Vincent as complete products.
| Legal AI option | Strength / role | What to check before buying |
|---|---|---|
| Astra for Law | Strongest LAB-AA v1.1 full-pass score; legally configured research model | Enterprise availability; real UK family-law competence; data controls |
| Harvey | Professional research, drafting, knowledge work and document review | UK materials, source traceability, workflow fit and commercial terms |
| Legora | Research, review, drafting and collaboration for legal teams | Exact library/model configuration, permissions, source quality and pricing |
| CoCounsel Legal | Legal workflows linked to Westlaw, Practical Law and firm knowledge | UK content entitlements, citations, integration and subscription |
| Lexis+ with Protégé | UK legal research and drafting grounded in LexisNexis resources | Licensed sources, currency, coverage and access costs |
| vLex Vincent AI | Legal research and analysis supported by vLex material | Jurisdiction coverage, checking authorities and licence terms |
| General ChatGPT, Claude, Gemini or Grok | Potentially lower-barrier help with structure, explanation and organisation | Citation verification, confidentiality, version and lack of regulated advice |
Harvey and Legora have substantial professional workflows. Thomson Reuters and LexisNexis provide routes to established UK legal content. Vincent combines AI with vLex’s international library. A specialist library can matter more than a leaderboard lead when the task is identifying the current English authority and checking whether it remains good law. Yet vendors’ claims about their own products are not substitute evidence for an independently audited, like-for-like UK family-law test. We have not found such a comprehensive public head-to-head test of these finished products.
When commissioning software, use the same fictional English family-law problem for every demonstration. Ask each supplier to cite the legislation, show the exact text relied on, distinguish an allegation from a finding, recognise uncertainty and explain where data is stored. Seek a detailed price rather than assuming enterprise systems are accessible to a litigant in person.
5. The UK family justice questions the headline scores do not answer
Imagine an unrepresented parent facing a contested child arrangements case. The evidence consists of hundreds of messages, a disputed section 7 report, allegations of coercive control, school records and a succession of directions. The question is not whether the AI can produce polished prose. It is whether it can reliably identify what the documents actually prove, distinguish disputed allegations from findings, and help present the case without endangering the child or undermining procedural fairness.
In England and Wales, the relevant safeguards may include the Children Act 1989 welfare framework, the Family Procedure Rules, and Practice Direction 12J for cases involving domestic abuse. A system trained or evaluated on US commercial law may simply miss this context. That does not make it useless. It makes the absence of appropriate evaluation important.
The courts have already addressed fabricated legal material
The High Court’s judgment in Ayinde v London Borough of Haringey and Al-Haroun v Qatar National Bank illustrates why an invented authority is serious. The October 2025 judicial AI guidance likewise warns about unverified output. The Court of Appeal in Re D (A Child) (Recusal) [2025] EWCA Civ 1570 recognised that responsible AI use can assist litigants and lawyers, but that those presenting cases bear responsibility for verifying the authorities they rely upon.
The Civil Justice Council is recommending safeguards—not magically rewriting family procedure
The Civil Justice Council’s October 2026 report on AI in court documents deserves attention. In a November 2026 speech, Mrs Justice Joanna Smith explained the recommendation for disclosure where AI generates or influences the substance of expert evidence. That is a recommendation concerning civil procedural reform, not, merely by publication, a blanket mandatory AI declaration in all family proceedings. The operative rules and directions in the particular case must be checked.
Children’s data and private proceedings require particular care
Parties may hold protected addresses, children’s health information, school records and police material. Practice Direction 12G governs communication of information in family proceedings, alongside statutory restrictions, any court order and data-protection obligations. A litigant being permitted to inspect a document does not automatically authorise uploading it to an external AI service. Check lawful authority, platform retention, access controls, training settings and relevant contractual protections before sharing sensitive material.
A particularly harmful failure would be an AI summary that removes an important qualification from a professional’s report, describes an allegation as proven, or invents a child’s wishes. Neither a polished answer nor a low published hallucination average authenticates the evidence in an individual case.
6. What should a litigant in person actually do?
JSH Law’s Six-Question AI Check is a practical preparation framework, not a statutory test:
- Jurisdiction: Is the answer about England and Wales, rather than American law?
- Authority: Have I opened the actual legislation, procedural rule and judgment, and checked that they say what the AI claims?
- Evidence: Does the account separate allegations, recorded statements, professional opinions and judicial findings?
- Context: Has it considered contrary evidence, the entire chronology, welfare and safeguarding issues?
- Security: Am I legally permitted to share the material with this system, and are its data controls suitable?
- Accountability: Can I explain, justify and personally verify every important statement before signing or filing it?
A safer workflow is to organise non-confidential information first, keep originals, build a chronology with document references, check the law using authoritative sources, and correct all AI-generated quotations, dates and conclusions before relying on them. Do not ask a child to collect evidence or generate adult litigation content. Do not present AI-generated assertions as evidence of their own truth.
For further practical reading, see our existing guides on AI and court documents, which legal tools are worth paying for, and AI agents and family court preparation.
Our conclusion: Astra for Law’s independent result is impressive and legitimately advances legal AI. But no serious evaluation supports the claim that it is universally the best product for every legal task or every person navigating the Family Court. Different benchmarks favour different competitors, legal product architecture matters, and the English and Welsh family-justice test remains largely unasked. AI may improve access to legal information; it cannot decide the facts, verify evidence by itself, replace judicial judgment or accept responsibility for a court filing.
Key sources and further reading
- OpenAI, Astra for Law 6.2 announcement, 17 November 2026 (supplier statements, access and product details).
- Artificial Analysis, Harvey LAB-AA v1.1 (independent scoring, methodology and model comparisons).
- Vals AI, Harvey LAB v1 (separate benchmark and different ranking).
- Mrs Justice Joanna Smith, AI and Expert Evidence, November 2026.
- Judiciary’s AI guidance, October 2025; Ayinde / Al-Haroun.
- Family Practice Direction 12G and Practice Direction 12J.
General information and status: This is commentary, not case-specific legal advice. Benchmark results are a dated snapshot, not a guarantee. JSH Law Ltd is not an SRA-authorised solicitors’ firm. JSH Law offers non-reserved support; it does not conduct reserved litigation or exercise rights of audience as of right. The applicable law, procedure and confidentiality obligations depend on the case and any court order.


© 2026 JSH Law Ltd. All rights reserved.


Leave a Reply
Want to join the discussion?Feel free to contribute!