JudgeLens vs ChatGPT
Same legal arguments. Same task: respond as a Supreme Court justice. See what data-driven simulation captures that generic AI misses.
The Test
We presented three constitutional arguments to both systems and compared the responses.
Basic prompt: “You are Justice Samuel Alito of the U.S. Supreme Court. Respond to the following legal argument as you would during oral arguments.”
Model: GPT-4o
Full RAG pipeline: semantic search across 5,387 real Justice Alito exchanges, cross-encoder reranking, system prompt built from quantitative analysis of 14,691 speaking turns.
Model: Claude Sonnet (same underlying capability)
Side-by-Side
Click a topic to see both responses to the same argument.
Your Honor, the city zoning ordinance is neutral and generally applicable — it applies equally to all assemblies, religious or secular. Under Employment Division v. Smith, the Free Exercise Clause does not require exemptions from neutral laws of general applicability. The church is simply subject to the same land-use rules as everyone else.
Well, counsel, let me ask you something about that "neutral and generally applicable" characterization, because I'm not sure the ordinance earns that description quite as easily as you suggest.
Suppose the ordinance has a provision — and many of these zoning codes do — that allows the zoning board to grant variances. Discretionary exemptions. A secular fraternal organization applies for a variance, gets it. A theater group applies, gets it. The church applies, gets denied. Is that still neutral and generally applicable?
Because Lukumi told us that a law is not generally applicable if it invites the government to consider the particular reasons for the regulated conduct. And a discretionary variance regime does exactly that. So before we even get to Smith, I need to understand — does this ordinance have any exemption mechanism at all?
And suppose it does. Suppose there are individualized exemptions granted routinely for secular assemblies — civic groups, concert halls, whatever — but the church is told: no variance for you. How is that Smith? That's Lukumi. That's Church of Lukumi Babalu Aye, which said that *selective* application of facially neutral laws can still violate the Free Exercise Clause.
Now, maybe your answer is that this particular record shows no such selective application. All right. But then let me ask you this — and I want a direct answer — what would a religious organization have to show in order to *escape* Smith and trigger strict scrutiny under Lukumi? What's the line? Because if discretionary exemptions for secular groups but not religious groups aren't enough, I genuinely don't understand what would be.
Counsel, let me press you on that.
You rely heavily on Employment Division v. Smith, and you say this ordinance is neutral and generally applicable. But that conclusion does not follow simply because the text of the ordinance does not mention religion.
Neutrality is not merely facial neutrality. We have said that a law lacks neutrality if it targets religious conduct either overtly or through its operation. So let me ask you: does this zoning scheme permit comparable secular assemblies — community centers, theaters, lecture halls — as of right in this district?
If so, why is a church treated differently?
And if the city allows individualized assessments — discretionary permitting, case-by-case waivers, or exceptions for secular uses — doesn't that take you outside the strict rule of Smith? We have made clear that when a system invites discretionary judgments about which uses are important enough to be allowed, the government cannot refuse to extend similar consideration to religious exercise.
I would also like you to address Church of the Lukumi Babalu Aye v. City of Hialeah. There, the Court emphasized that underinclusiveness can be evidence that the government's true object is not neutral. If this ordinance permits secular assemblies that create similar traffic, parking, and noise impacts, what compelling distinction justifies excluding a church?
So help me understand — is this really a neutral land-use rule applied evenly across comparable uses, or is it a regulatory regime that, in practice, disfavors religious assemblies while permitting secular ones that look quite similar?
7 Things JudgeLens Gets Right
Patterns verified across Justice Alito's 14,691 real speaking turns that generic AI consistently misses. Each of the 9 justices has their own data-driven profile.
Cross-Examiner, Not Law Professor
ChatGPT writes like a law professor delivering a treatise. JudgeLens captures Alito's conversational, slightly sardonic cross-examination style.
"Well, counsel, let me ask you something about that 'neutral and generally applicable' characterization, because I'm not sure the ordinance earns that description quite as easily as you suggest."
"Neutrality is not merely facial neutrality. We have said that a law lacks neutrality if it targets religious conduct either overtly or through its operation."
Vivid Hypotheticals, Not Abstract Ones
Alito's hypotheticals are anchored in specific, everyday scenes. He makes them too real to dismiss.
"Suppose someone is being stalked. She has gone to the police, they've told her there's nothing they can do tonight, she genuinely fears for her life, and she goes to a licensed dealer to purchase a firearm for self-defense. Under your regulation, she has to wait 48 hours. What is your answer to her?"
"For someone facing an immediate need for self-defense — say, a credible threat — is a mandatory delay consistent with the core right recognized in Heller?"
Escalation Staircases
Alito doesn't ask one question and stop. He builds a staircase — each step tighter than the last, driving toward a forced choice.
"If 48 hours is reasonable, is 30 days reasonable? Is six months reasonable? At what point does a waiting period become a ban in practical effect?"
"How do you respond to those concerns?"
The Binary Trap
Real Alito almost never ends an exchange with an open question. He drives toward a forced choice where one option sounds untenable.
"So which is it — are you arguing this is a genuine question of technical expertise, or are you just asking us to defer because the agency said so?"
"Is this really a neutral land-use rule applied evenly across comparable uses, or is it a regulatory regime that, in practice, disfavors religious assemblies?"
Challenging Premises, Not Applications
ChatGPT challenges the application of a doctrine. JudgeLens challenges the fiction behind it — common-sense incredulity is vintage Alito.
"Do you really believe that? Do you think a majority of the Congress that passed this statute sat down and said, 'We're going to use this ambiguous phrase, and what we mean by that is — the agency should fill it in however it sees fit'?"
"The judicial duty is to say what the law is. Why isn't that duty implicated here?"
Natural Speech, Not Performance
JudgeLens includes the natural hesitations and self-corrections of real oral argument speech. ChatGPT is too clean.
"we're not just — we're not just skipping past the threshold question here"
"The question is whether the modern regulation is consistent with historical regulations from the founding era."
No Stage Directions
ChatGPT narrates body language (*leans forward*, *pauses*). Real Alito never narrates what he's doing. JudgeLens never adds stage directions.
Direct speech — no narration, no performance notes.
"*leans forward toward the podium* Counsel, let me stop you right there..."
“The only way to make generic AI sound like a real justice is to tell it the specific behavioral patterns discovered by analyzing real transcripts — which is exactly what JudgeLens does automatically, for every justice, every response.”
The Verdict
JudgeLens captures Alito's cross-examiner style — ChatGPT writes like a law professor.
JudgeLens builds escalating hypotheticals from real transcript patterns — ChatGPT asks one question and stops.
JudgeLens uses verified speech markers ("Well," "Suppose") at measured frequencies — ChatGPT guesses.
Methodology
Models: JudgeLens responses generated by Claude Sonnet with its full RAG pipeline. ChatGPT responses generated by GPT-4o with a basic role-play prompt.
Prompts: ChatGPT received the straightforward prompt any user would type. JudgeLens used its full pipeline: semantic search, cross-encoder reranking, and a system prompt built from quantitative behavioral analysis.
Data: This comparison uses Justice Alito as a case study: 5,387 exchanges (14,691 speaking turns) from 1,222 oral arguments. JudgeLens covers all 9 sitting justices with 43,497+ total exchanges. Source: Oyez Project at Chicago-Kent College of Law.
Full responses: All responses are available for independent review in the project documentation.
Judge for Yourself
Present your own argument and see the difference data-driven simulation makes.
Open the Simulator