Hello, friends,
This week, the Biden administration released its Blueprint for an AI Bill of Rights, a wish list of principles that “should guide the design, use, and deployment of automated systems.”
The principles state that Americans should:
- Be protected from unsafe or ineffective systems.
- Not face discrimination by algorithms.
- Be protected from abusive data practices.
- Be informed when an automated system is being used.
- Be able to opt out from automated systems “where appropriate” and have access to a human decision-maker.
These principles, while laudable, do not say much about how we will get there. The White House did not propose legislation, the creation of a new agency, or any other holistic measure to regulate automated systems. It did issue a list of actions agencies are taking within their current mandates, such as an effort by the U.S. Department of Housing and Urban Development to establish standards for home valuation algorithms.
Meanwhile, there are people fighting for algorithmic accountability all over the world without any of these rights to back them up. So I thought this would be a good moment to highlight one of the most gruesome struggles out there—the story of the out-of-control automated system that was inflicted on the people of Michigan.
In 2013, Michigan laid off most of the humans who worked to identify fraudulent unemployment claims and instead installed an automated system to replace them. But it turns out the system was wrong the vast majority of the times that it flagged a claim as fraudulent. As a result, the state wrongfully seized millions of dollars from tens of thousands of people over the two years that the system was operating.
These are the kinds of unsafe, ineffective practices that the AI Bill of Rights seeks to prevent. But there has yet to be independent accountability for Michigan’s mistakes. In 2017, the state passed a law requiring the agency to make fraud determinations manually and said it would refund $20.8 million to residents who were wrongfully accused of fraud. But lawyers say that is not the full amount of refunds owed and doesn’t account for the damages suffered by people who lost jobs, homes, and filed for bankruptcy as a result of this system.
To understand the case of Michigan’s rogue fraud-detection system, I spoke to Jennifer Lord, a civil rights and employment attorney and partner at Pitt, McGehee, Palmer, Bonanni & Rivers in Michigan, who has been fighting for seven years to get that money back for those wrongfully accused. In July of this year, the Michigan Supreme Court finally ruled that workers falsely accused of unemployment fraud can sue the state for the violation of their constitutional rights. She and her colleagues are now proceeding to trial seeking money damages.
Our conversation, edited for clarity and brevity, is below.
Angwin: Let’s start at the beginning. What is MiDAS and what happened in 2013 when Michigan implemented this technology?
Lord: In 2013, our office, which does employment and civil rights law, started getting a ton of calls from our current clients saying that the [Michigan] Unemployment Insurance Agency was billing them for some obscene amount, $10,000 or $25,000, because they allegedly committed fraud when they applied for their unemployment benefits.
We spoke with other civil rights and employment attorneys in the state of Michigan, and we all started digging, and we found a press release where the state of Michigan had nominated itself for this tech award for having rolled out this brand-new computer system, a $46 million system called MiDAS. MiDAS was tasked with deciding whether someone committed fraud in connection with their application for unemployment benefits. For example, if someone was applying for unemployment insurance but still had a job, that would be fraud, and that person would get in trouble. What happened here is that MiDAS was wrong 93 percent of the time, and in about 40,000 instances it incorrectly found fraud.
Angwin: Can you explain to me what exactly transpired for the people who were wrongfully accused?
Lord: Let’s say someone gets disqualified from unemployment benefits on the basis of fraud. Problem number one is that the determination letter they send you doesn’t say what the state thinks you did. There are no specifics, so you don’t know what you’re defending against. The person is supposed to know what they did wrong and then they have 30 days to write a handwritten protest and literally fax it back to the agency. Let’s say it’s received within 30 days, in a perfect world that will go to an administrative law judge who would then rule as to whether you’re entitled to the benefits or not.
The problem is that MiDAS wasn’t sending paper copies of these fraud determination notices. They were sending pings to an electronic portal. To complicate this, MiDAS did a six-year audit so people were being accused of having committed fraud in 2015 when they hadn’t collected benefits since 2010 or 2009. Returning to the problem of online notifications—people weren’t going on their portals anymore. People weren’t responding to these determinations within 30 days, and then they became final and nonappealable.
If MiDAS found that you committed fraud, you would owe back the benefits you already received and a quadruple penalty. The government was taking this money from people in two ways: the seizure of tax refunds and administrative garnishment. MiDAS interfaced with the state and federal treasury, so it would seize people’s tax refunds to pay down the debt.
Second, the state used something called an administrative garnishment. With a traditional garnishment, there’s a judicial process before wages are taken. With the state administrative garnishment, there’s no process. It’s automatic and not subject to judicial review. This means the first time someone would find out what was happening is when they found a new job and then 25 percent of their new paycheck was garnished, and the garnishment continues until the debt is paid.
Angwin: This sounds like it could be financially devastating. Can you talk about how this affected people?
Lord: We know for a fact that at least 11,000 families in the state of Michigan filed for bankruptcy. That’s a very real harm, and that family isn’t going to be able to get a mortgage, or it will be at a much higher rate, or if they need to lease a new car, they will have a ridiculous rate.
It’s also shown up on people’s background checks. One woman I met had completed her training to become a police officer in the city of Detroit and then failed the background check and didn’t get the job because MiDAS had flagged her as having committed fraud.
Angwin: You have been representing the people wronged by MiDAS for seven years now. Can you give us an overview of the allegations you brought against the state and what has been happening?
Lord: Our lawsuit, Bauserman v. Unemployment Insurance Agency, says that the state of Michigan violated the Michigan constitution when it seized people’s property without due process. For the first three and a half years, the legal issue was that in Michigan, if you’re suing the state, there are extra procedural rules with separate timelines. The state argued that we missed this very specific time frame—that we sued too late. This went all the way up to the Michigan Supreme Court, which ruled that we did not sue too late. That was issue number one. Issue number two, which surprisingly had never been clarified in Michigan, was the question of whether somebody can sue the state for money damages when the state violates the constitution. This ruling just happened in July and was in our favor.
Angwin: Now that you have this ruling from the Michigan Supreme Court, what are the next steps and how long will this take?
Lord: The short answer is we’ve got to start discovery like we were ready to do seven years ago. We also have to get this class certified; right now, we represent two individuals on behalf of a putative class of tens of thousands. One of the first things on the agenda will be to ensure that we officially and legally represent everyone with a claim.
Since it is a tort, we will argue for consequential damages, so at a minimum, money back plus interest, but there also has to be some recognition that this was a real harm. Even if you didn’t declare bankruptcy, the stress of being told you owe $40,000 or $20,000 shortly after you’re fired from your job—there’s not going to be a formula for that.
You never know with litigation, but there’s no inherent reason for this to take any longer than any other case. We need to start getting people under oath and hearing what they have to say and what they remember. I have a feeling that the agency is going to have known that this was a problem almost from the get-go, and the real interesting thing will be figuring out who knew what, when.
Angwin: What have you learned about holding this kind of AI system to account? What should be done in the future to prevent this type of event, and what might other lawyers want to look for when investigating?
Lord: I’ve been surprised how little this worries most people. I don’t know if it’s the fact that in Michigan this story was breaking at the same time as the Flint water crisis, so if you’re looking at two debacles, the one with the brown glass of water is very disturbing. MiDAS just didn’t seem to ignite the imagination like I thought it would. The state of Michigan was basically stealing tens of millions of dollars from its citizens!
How to fix it going forward? One idea is having an independent, knowledgeable, and interdisciplinary group vetting technology before any purchase is made. I can only imagine a skilled salesperson selling the agency on promises of efficiency, cost cutting, and profit increases. What politician is going to say no to that?
I think the problem is that there are some jobs computers and algorithms can’t and shouldn’t do, such as determining whether or not someone committed fraud. It’s wrong that we asked the computer to do this all by itself, with no human intervention. It’s one thing if a computer flags it and then a human takes a deeper look. That’s not what happened because at the same time that the Unemployment Insurance Agency purchased MiDAS, it laid off a third of its workforce, almost its entire fraud detection unit.
The algorithm is alleging that these 40,000 people intentionally defrauded the unemployment insurance agency. I think a judge or a jury should decide the question of intent. When assessing credibility, it should not be done by an algorithm. I don’t think that a computer looking at a series of numbers is capable of determining intent.
As always, thanks for reading.
Best,
Julia Angwin
The Markup
(Additional Hello World research by Eve Zelickson.)