Remote exam software is crashing when the stakes are the highest

Molly Savage never expected she’d be taking the Michigan state bar exam remotely after graduating this year from Wayne State University’s law school. Nevertheless, in July, she found herself sitting down in an office at the law firm where she already worked, taking the final seven-hour test that stood between her and a license to practice law.

Michigan, like many states around the country, had chosen a software company—in this case, ExamSoft—to administer the test and “proctor” it remotely. Savage had had a bit of trouble tracking down a computer that could run the software, which films test-takers through their computer’s camera, and the whole thing felt a bit creepy.

“Who’s actually reviewing this?” she wondered. “Is it a human being who’s reviewing it in real time?”

And then something even stranger happened: Midway through the test, a website meant to deliver passwords used to access the exam failed.

For Savage, the already high-stakes, high-stress test became “kind of chaos,” she said.

The log-in process, the company later said, had been targeted in a “sophisticated” cyberattack. Panicked test-takers around the state were calling ExamSoft, unable to pass the log-in page as the time ticked down on their exams. The company was able to get the test back on track, and according to a statement, “at no time was any data compromised.”

Nici Sandberg, a spokesperson for ExamSoft, said the company has since taken steps to improve its defenses against future attacks. But it’s not the first time ExamSoft’s digital proctoring has run into problems.

“Barmageddon”

In 2015, the company agreed to pay more than $2 million to settle a class action suit after software failures affected the bar exams in 43 states. At the time, the situation spawned a catchy epithet: “barmageddon.” The company declined to comment on those incidents.

And this year, two states with contracts with the proctoring company ILG Technologies, Indiana and Nevada, postponed their remote bar exams after technical glitches were revealed during practice testing. ILG Technologies did not respond to a request for comment. Officials offering the exams in Michigan, Indiana, and Nevada also did not reply to requests for comment.

But that hasn’t stopped more and more state bar associations, along with a growing number of universities, from turning to companies that produce remote proctoring software—the University of Washington, University of Missouri, and the University of California system are just a few of the many schools that have turned to the tools.

One executive from a company called Examity recently told Inside Higher Ed that the company’s growth was up 35 percent over what was expected in its quarterly forecast. Another, from a company called Proctorio, told The Washington Post his company would “probably increase our value by four to five X just this year.”

Even before the pandemic, the tools were being widely adopted. In 2018, Examity reported that it had seen five straight years of more than 50 percent growth. That growth is likely to continue unabated as schools remain remote.

The states The Markup contacted did not respond to requests for the dollar amounts of contracts with exam companies, but ExamSoft, for example, has been known to charge each exam-taker $134.50 in licensing fees as part of its contract. Normally, thousands of people take the bar around the country each year.

Tracking eyes and faces

Meanwhile, privacy advocates say the technology, which purports to track eyes, scan faces, and monitor typing, is troubling.

“A bar exam that forces our future colleagues to use invasive and unproven technology … and potentially endure discrimination is antithetical to everything the law is meant to uphold,” Albert Fox Cahn, executive director of the Surveillance Technology Oversight Project, wrote this month in a letter to New York officials asking them to reconsider the use of ExamSoft for the state’s bar exam, citing concerns of the potential for racial bias in tools like face-scanning. (Some test-takers have recently said they faced exactly that problem with the software—that it appeared to struggle to recognize darker features.)

ExamSoft’s Sandberg said, “[A]ll exams are first reviewed by AI, flagged for potential incidents and then also re-reviewed by professional human proctors.”

The Electronic Frontier Foundation raised similar concerns about the California state bar exam. Both that exam and the New York exam were given remotely through ExamSoft last week.

Open Book Tests or No Tests at All?

The question is, in a pandemic’s world, what the alternatives really are.

One option, “diploma privilege,” is used in Wisconsin. It’s a process that allows some in-state law graduates to become licensed to practice without taking the bar exam. A few other states have given 2020 law school graduates who complete certain requirements such privileges, laying a path to practice as lawyers without taking the exam.

But other states have been more reluctant. In Florida, a group of law graduates filed a petition in the state’s Supreme Court court against the use of ILG software for the state’s exam this year. The graduates asked to have the exam waived for the year and to be allowed to practice after working with a licensed attorney for six months. The court ultimately shot down the request. The Florida Board of Bar Examiners did not respond to a request for comment.

At the University of California, Santa Barbara, faculty sent a letter to the administration objecting to the broad categories of data, from identifying physical details to keystrokes, used by ProctorU, another provider of software used for remote exams. “We recognize that in our collective race to adapt our coursework and delivery in good faith, there are trade-offs and unfortunate aspects of the migration online that we must accept,” the faculty wrote. “This is not one of them.” (The company threatened legal action against the faculty over the letter and said it only collects data for proctoring purposes. Use of the software is currently optional.)

“On campus, we are there to protect our students; we feel responsible for our students,” Jennifer Holt, a film and media studies professor at UC Santa Barbara and part of the faculty association, told The Markup. “That,” she said, “doesn’t change because instruction has moved online.” The use of the software isn’t required, but there’s little alternative beyond changing the format of tests.

Some experts in education question whether the problems go beyond the use of any specific software.

“This is a case where we’re trying to use the technology Band-Aid for a problem that’s much more fundamental,” said David Rettinger, a professor of psychological science at the University of Mary Washington, who studies ethics in education.

Rettinger said educators should consider relying more on different forms of testing, like open-book exams, which may better demonstrate how much students have learned. That could also create a less adversarial learning environment, he said, one that’s built more soundly on trust. It’s not exactly a silver lining, he said, since these are problems that suddenly have to be solved immediately, instead of over time.

Holt, for one, said she’d given the first open-book exam of her career recently. “These are crazy times,” she said. “We’ve got to make adjustments.”

But, she added, “we have to do it in a way that has these students in mind first.”