Read Post
Wed, 23 Apr 2025 19:14:05 -0700
marlon from private IP, post #15661348
/all
AI secretly helped write California bar exam
https://arstechnica.com/tech-policy/2025/04/ai-secretly-helped-write-california-bar-exam-sparking-uproar/?utm_source=fark&utm_medium=website&utm_content=link&ICID=ref_fark
AI secretly helped write California bar exam, sparking uproar
A contractor used AI to create 23 out of the 171 scored multiple-choice questions.
Benj Edwards – Apr 23, 2025 3:05 PM | 74
A computer-generated gavel hovering over a laptop.
A computer-generated gavel hovers over a laptop. Credit: Getty Images
On Monday, the State Bar of California revealed that it used AI to develop a portion of multiple-choice questions on its February 2025 bar exam, causing outrage
among law school faculty and test takers. The admission comes after weeks of complaints about technical problems and irregularities during the exam
administration, reports the Los Angeles Times.
The State Bar disclosed that its psychometrician (a person or organization skilled in administrating psychological tests), ACS Ventures, created 23 of the 171
scored multiple-choice questions with AI assistance. Another 48 questions came from a first-year law student exam, while Kaplan Exam Services developed the
remaining 100 questions.
The State Bar defended its practices, telling the LA Times that all questions underwent review by content validation panels and subject matter experts before
the exam. "The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in
advance of the exam," wrote State Bar Executive Director Leah Wilson in a press release.
According to the LA Times, the revelation has drawn strong criticism from several legal education experts. "The debacle that was the February 2025 bar exam is
worse than we imagined," said Mary Basick, assistant dean of academic skills at the University of California, Irvine School of Law. "I'm almost speechless.
Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable."
Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, called it "a staggering
admission." She pointed out that the same company that drafted AI-generated questions also evaluated and approved them for use on the exam.
Ars Video
How Lighting Design In The Callisto Protocol Elevates The Horror
State bar defends AI-assisted questions amid criticism
Alex Chan, chair of the State Bar's Committee of Bar Examiners, noted that the California Supreme Court had urged the State Bar to explore "new technologies,
such as artificial intelligence" to improve testing reliability and cost-effectiveness.
However, the California Supreme Court told the LA Times that it "was unaware that AI had been used to draft any of the multiple-choice questions" until the
State Bar's Monday press release.
The AI disclosure follows other problems with the February exam. Test takers reported being kicked off online testing platforms, experiencing screen lag and
error messages, and encountering typos and confusing questions. These issues prompted a federal lawsuit against Meazure Learning, the exam administrator, and
calls for an audit of the State Bar.
A unique test of dubious quality
The State Bar of California switched from the National Conference of Bar Examiners' Multistate Bar Examination to its own hybrid in-person and remote testing
model last year while facing a $22 million deficit. It contracted with Kaplan Exam Services for $8.25 million to create test questions.
According to the LA Times, law professors Basick and Moran had previously raised concerns about the quality of the February exam questions, noting that the 50
practice questions released before the exam "still contain numerous errors" even after editing.
Basick expressed additional concerns about the use of first-year law exam questions, arguing that an exam determining minimal competence to practice law
requires a different standard than one assessing first-year law school learning.
The State Bar plans to ask the California Supreme Court to adjust test scores for February exam takers but has resisted returning to the National Conference of
Bar Examiners exams for July, citing test security concerns with remote testing options.
The Committee of Bar Examiners will meet on May 5 to discuss remedies, but Chan said the State Bar is unlikely to release all 200 exam questions or return to
National Conference of Bar Examiners tests soon, as nearly half of California bar applicants want to keep the remote testing option.
Photo of Benj Edwards
Benj Edwards Senior AI Reporter
Benj Edwards is Ars Technica's Senior AI Reporter and founder of the site's dedicated AI beat in 2022. He's also a tech historian with almost two decades of
experience. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC.
Thu, 24 Apr 2025 10:14:19 -0700
whiteguyinchina from private IP
Reply #15119506
Oh that is fucking lame
The state bar does realize that AI is the biggest threat to the legal professions existence and feasibility as a career?
It's almost a moral capitulation
Thu, 24 Apr 2025 10:15:25 -0700
whiteguyinchina from private IP
Reply #18712166
Hey Alexa, write 10 questions on real property
Done!
Replies require login.
