There’s currently a lot of talk about the recent report from Ontario’s Auditor General on AI Scribes. The headlines seem mostly to be dealing with the fact that she found numerous AI Scribe generated reports had errors. The errors happened for various reasons, including AI hallucinations, transcription errors, incorrect entry of medications and so on.
However, to my mind, that’s not the real story.
I feel somewhat conflicted in saying this next part, mostly because I think I generally have a reputation for being an advocate for physicians, their views and their well being. However, the blunt reality is that we are all required to check any report that’s generated by an AI scribe before we sign off on them. Physicians, being human, will make mistakes. For example, this past weekend, I got a message from a colleague of mine, pointing out an error that had been made in an AI-generated note on a patient I saw. That was my fault for not double checking. I think to try and blame some software for those kind of mistakes would be inappropriate.
No, the real story is the continued ineptitude of the healthcare bureaucrats at the Ministry of Health who are in charge of health care IT systems today. If one does a deep dive into the Auditor General’s report, there are many, many legitimate question she has, all of which the hard-working taxpayers of this province deserve an answer to.
In particular she found gaps in how these AI systems were evaluated by Supply Ontario, Ontario Health, and to a certain extent OntarioMD. Yes there were three agencies all involved, triplicating the amount of work necessary and adding to the confusion.
Heck the issues began right from the initial procurement stage. The weighting given to different criteria revealed a fundamental misalignment of priorities. The accuracy of medical notes generated by AI scribes accounted for only four per cent of points awarded to potential vendors, while domestic presence in Ontario was weighted the highest at 30 per cent. Data privacy/legal controls were weighted at 23 per cent and system security controls at 11 per cent.
Think about that for a minute. You could have software from a poorly run company, that was completely inaccurate in its transcription and system security, yet still have it approved if it happened to be Ontario based. Yet a company with the best transcription and system security would lose, if it was from out of province. Even Spence was shocked by this, stating, “In my mind, that doesn’t make sense….when we’re dealing with personal information and we’re dealing with artificial intelligence, I think security is of the utmost importance.”
Additionally, the evaluations didn’t actually watch vendors operate the software in real time! There were no live test. Vendors were apparently given recordings and ran the system offline learning. Spence said, “this allowed vendors to potentially overstate their compliance with security and privacy requirements.”
Well, duh!
Worse, 11 of the approved vendors for AI software didn’t actually meet the mandatory submission requirements. They got approved anyway. Five didn’t even submit risk assessments and privacy impact assessments as part of their bid process. They got approved anyway.
This kind of amateurish, ineffectual assessment is supposed to help increase confidence in healthcare IT?
Most damningly, it appears from the auditor general’s report that there is a broad absence of strategic governance. The auditor general benchmarked the AI strategy against Canadian and international public sector organizations and found that there were no specific actionable items, no clear plan to prioritize AI use across ministry areas, and did not identify any prohibited AI practices or areas where technology posed an unacceptable risk.
Essentially, this report paints a picture of Ontario Health/Supply Ontario/Ontario MD approving AI systems through a process that underweighted accuracy, did not require live demonstrations, accepted incomplete documentation and failed to assess bias risk. All while having no clear plan to rectify these gaps going forward.
The thing is, this kind of insanity has been permeating the politics of IT health systems for decades. I’ve written about the bloated and inefficient bureaucracy for years now. The lack of ability to get a truly integrated health care system speaks to a lack of vision and focus in the bureaucracy. It’s incredibly discouraging that it continues unabated after all these years. It seems that no one has the knowledge, wisdom, ability to fire the incompetent bureaucrats, streamline the process by getting rid of multiple agencies, and apply an overarching vision for health care IT.
And yet, instead of fixing the bureaucratic mess first, streamlining health IT infrastructure, and developing on overarching health IT vision, Ontario is instead now going ahead and launching a Provincial initiative to create a province wide primary care medical record system. The people in charge of choosing the software for this? The same bunch who botched the AI scribe issue.
I can’t wait to read the Auditor General’s report on that one in, say 2029.
justanoldcountrydoctor 