While the ketamine-addled wunderkind of the tech world are panicking about AI achieving consciousness, the actual harms it is causing, such as denying patients needed pain medication, are very real, if not sexy (boldface mine):

The rollout of a Medicare program that uses artificial intelligence to approve or deny care was hasty and error-ridden, according to more than a thousand pages of recently released documents and data.

The pilot program, called the Wasteful and Inappropriate Service Reduction model, or WISeR, launched in January and requires health care providers to seek approvals before they can provide certain procedures and products, like skin substitutes and epidural injections for pain management. The experiment operates in New Jersey, Ohio, Oklahoma, Texas, Arizona, and Washington and will run until 2031…

During the early months of the rollout, both the company, called Innovaccer, and its Medicare contracting partner reported issues that matched complaints coming from providers. The poorly implemented technology resulted in canceled and postponed surgeries and “patients calling our offices crying in pain because their procedures are being delayed while awaiting approvals or guidance,” providers said.

“I have had to watch 3 patients cry at bedside for not hearing back on their prior auth for kyphoplasty/vertabral augmentiation [sic] procedure,” wrote one Ohio-based provider in a feedback survey. “These patients are in deep pain. So many offices of my physicians are not hearing back from Innovaccer and […] there is no way to get a hold of a human to talk to on Innovacer [sic]. What a disgrace to the human race in America. This is Third World.”

And the Trump administration’s belief in AI über alles is contributing to this problem:

Earlier this summer, the Senate unsuccessfully attempted to end WISeR after the Government Accountability Office found that CMS did not follow proper procedure in setting up the Medicare experiment. The federal watchdog’s finding came amid complaints that the pilot was harming people in Washington state.

Congresswoman Suzan DelBene (D-Wash.), who led a House effort to stop WISeR, told STAT in a statement that “the more that American seniors learn about WISeR, the more they are outraged that the Trump administration is allowing tech companies to profit by delaying and denying necessary care.” While the new information answers many questions, she said, it also begs new ones, and she pledged to hold the administration accountable. “It’s incredibly frustrating how reticent HHS has been in sharing information about this hastily put together AI experiment,” she said.

And the people building this system had no idea what they were doing:

More than two months after WISeR’s launch, a Medicare administration contractor, Noridian, realized that one of the vendors did not know the difference between Medicare Part A and Part B. It traced back ongoing provider complaints about claims being incorrectly categorized to Zyter, the company administering WISeR for Arizona. “Zyter acknowledged that access to its portal was limited to ‘outpatient’ requests due to a misunderstanding that outpatient hospital services equate exclusively to Part B,” Noridian wrote, adding that it provided Zyter guidance as to the distinction between the two.

In the same March report, Noridian reported that Zyter’s portal was not configured to handle requests from ambulatory surgical centers, and that it had warned Zyter that simply dismissing requests coded for ASCs “could result in provider abrasion and potential compliance concerns.”…

Despite ambitions of using AI to make the review process quick and easy, providers complained about how non-functional vendors’ portals were and ended up reverting to paper.

“It appears that all documentation has to be faxed, the portal cannot be used to submit the documentation. I have been told that there is a way to submit via the portal, but the customer service representatives have not been able to provide instructions on how to proceed,” wrote a provider from Kettering Health Network in Ohio…

All of the technical hiccups have created greater costs for the Medicare program. According to an email between Medicare staffers, the Medicare contractor for Texas, New Jersey, and Oklahoma alerted CMS that it was going to need more funding than originally planned.

This is going to be a far more common problem with ‘AI’: the people building, testing, and managing the system build bad systems, but, because various leaders are committed to AI, those bad systems will be forced on people and do great harm. In other words, it will just amplify the same old problems, but an ideological commitment–and likely a financial one too–to ‘AI is the future’ will hurt people.

And before anyone excuses this, the story also notes that it is incredibly difficult to get details from CMS about what went wrong. The ideology that we must use AI, consequences be damned, combined with the time-honored need to cover one’s ass will be the far more likely and ugly outcome of rapid AI, not some Lovecraftian godhead. AI won’t be used in some ethereal plane without real-world complications, but within a political and economic context. And when combined with a belief that AI must be adopted at speed–move fast and break patients–the consequences will be bad.