<!-- todo: - add details of current medical transcription services and their privacy policies - expand section of chatbot risks new refs: - [medical chatbot danger](https://www.theguardian.com/australia-news/2025/aug/03/ai-chatbot-as-therapy-alternative-mental-health-crises-ntwnfb) - https://www.theregister.com/2025/07/25/microsoft_admits_it_cannot_guarantee/ - Dr David Adam’s thoughts in full: https://www1.racgp.org.au/newsgp/clinical/extremely-unwise-warning-over-use-of-chatgpt-for-m > Current large language models (LLMs) such as ChatGPT and GPT-4, marketed as ‘Artificial Intelligence’, work by using large amounts of public textual information to generate algorithms which can process prompts of various kinds. > >A mobile phone’s predictive text can suggest the next word in a sentence; LLMs can suggest whole paragraphs or more at a time. > >Unfortunately, it is easy to infer understanding or competence from these models, but there have been many examples in recent months where the output that ChatGPT or similar programs generate is misleading, incomplete or plain wrong. > >It is easy to confuse confidence with competence, and the output of these programs certainly appears extremely confident. - https://www.theregister.com/2025/07/23/ai_size_obsession/ Reasoning models unsustainable - ## AI chatbot danger - https://arstechnica.com/ai/2025/07/ai-therapy-bots-fuel-delusions-and-give-dangerous-advice-stanford-study-finds/ - https://m.slashdot.org/story/444298 - https://www.theregister.com/2025/07/03/ai_models_potemkin_understanding/ - https://www.theguardian.com/australia-news/commentisfree/2025/jul/01/ai-hype-artificial-intelligence-power-dynamics Best one on diminishing returns of AI https://juuri.greenant.net/hedgedoc/p/bKGf4FsPv49LUTGrrdsZGt#/ https://www.deeplearning.ai/the-batch/ai-giants-rethink-model-training-strategy-as-scaling-laws-break-down/ https://medium.com/@raniahossam/chinchilla-scaling-laws-for-large-language-models-llms-40c434e4e1c1 https://www.businessinsider.com/sam-altman-ai-wall-slowdown-openai-2024-11 >For general-knowledge questions, you could argue that for now we are seeing a plateau in the performance of LLMs,&quot; Ion Stoica, a cofounder and the executive chair of the enterprise-software firm Databricks, told The Information, adding that "factual" was more useful than synthetic data. https://www.businessinsider.com/openai-orion-model-scaling-law-silicon-valley-chatgpt-2024-11 https://garymarcus.substack.com/p/confirmed-llms-have-indeed-reached AI and US govt https://m.youtube.com/watch?v=wKkk-uWi7HM&amp;t=4496s&amp;pp=2AGQI5ACAQ%3D%3D Agent performance https://store.minisforum.com/en-au/products/minisforum-ms-a2 AI collapse https://www.theguardian.com/technology/2025/jun/09/apple-artificial-intelligence-ai-study-collapse AI and therapy https://m.slashdot.org/story/443195 https://arstechnica.com/ai/2025/06/google-ai-mistakenly-says-fatal-air-india-crash-involved-airbus-instead-of-boeing/ https://arstechnica.com/ai/2025/06/new-apple-study-challenges-whether-ai-models-truly-reason-through-problems/ AI and sexual violence https://www.theregister.com/2025/05/08/google_gemini_update_prevents_disabling/ https://www.theguardian.com/technology/2025/may/04/dangerous-nonsense-ai-authored-books-about-adhd-for-sale-on-amazon --> # "AI" In Healthcare: <br>Realism in the Era of Peak Hype }C Dr Frank Giorlando MBBS PhD FRANZCP <div style="font-size:20pt;"> Consultant Psychiatrist -- Healthscope and Monash Health<br> Adjunct Senior Lecturer -- Deakin University and Monash University<br> Chief Technical Officer -- GreenAnt Networks </div>   <!-- healthscope  --> <!-- <a rel="license" href="http://creativecommons.org/licenses/by-nc/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc/4.0/88x31.png" /></a> --> <a rel="license" href="https://creativecommons.org/licenses/by-nc-nd/4.0/"><img width=140px alt="Creative Commons License" style="border-width:0" src="https://mirrors.creativecommons.org/presskit/buttons/88x31/png/by-nc-nd.png" /></a> <!-- }C [https://juuri.greenant.net/hedgedoc/p/bKGf4FsPv49LUTGrrdsZGt#/](https://juuri.greenant.net/hedgedoc/p/bKGf4FsPv49LUTGrrdsZGt#/) --> --- # Introduction This talk aims to inform regarding what AI is and how it is being applied in health care. It aims to be realistic, perhaps even *deflationary* about what AI can do. Much of this will appear pedantic, for good reason! 1. What is "AI"? 2. Machine learning in health care 3. Implementation and ethics 4. Summary and questions --- ## 1. What is "AI"?  - Some definitions (and history) - How it works - What it is not - Current trends ---- ### A (Very) Rough History  <div class = "reference"> <a href="https://en.wikipedia.org/wiki/Marvin_Minsky"> Marvin Minksky 1927-2016 </a> </div> - AI ([Artificial Intelligence](https://en.wikipedia.org/wiki/History_of_artificial_intelligence)) is not new! It goes back as far as the 1920s, with many notable milestones - First "neural network" implemented 1951 (Minksky), then "perceptron" (Rosenblatt 1958) - 1982 revival of "connectionism" (Hopfield 1982) and "backpropagation" (Hinton and Rumelhart 1986) - 2005 - "[deep learning](https://en.wikipedia.org/wiki/Deep_learning)" - What is new is the scale of (LLM) models - Trillions of parameters - Billions of \$ to train - Running out of training data ---- ### Some Definitions  - AI is a term that has undergone significant [semantic drift](https://en.wikipedia.org/wiki/Semantic_change) - Modern *deep-learning* models are not "intelligent" in any meaningful sense - Therefore they are best described as *Machine Learning* models. - Commercial entities benefit from moving the goal posts, hence the new term *AGI* (Artificial General Intelligence) - While most people associate "AI" with *Large Language Models* (LLMs), there are many other types: - Some model descriptions: - *Foundation Models* - *Transformer Models* - *Diffusion Models* ---- ### Some Other Terms You May Have Heard  - Prompts (and hidden prompts) - Training - Inference - ~~Hallucinations~~/Confabulation - generating information not present in the input data - Omissions - missing relevant information (when summarising) - Fine-Tuning - RAG (Retrieval Augmented Generation) ---- ### The AI Ecosystem  - AI as a service - Foundation models are *expensive* - AI "agents"/"bots" - Most services lease time on models - AI as a data collection system - Openness (or lack thereof): - open-source license - open-weights - open training set - e.g. [CommonCrawl](https://kili-technology.com/large-language-models-llms/9-open-sourced-datasets-for-training-large-language-models) - Local vs Cloud-provided - It is possible to run inference on a PC - Finetuning is also possible - The training data is the most valuable part of the model ---- ### About Machine Learning (LLMs) Large Language Models (LLMs) are Machine Learning [models](https://www.geeksforgeeks.org/machine-learning/machine-learning-models/) which can be trained (usually with [Self-Supervised Learning](https://www.geeksforgeeks.org/machine-learning/self-supervised-learning-ssl/)) to produce sequential output *mimicking* human language.  These Models produce output that: - reflects their training sets ([GIGO](https://en.wikipedia.org/wiki/Garbage_in,_garbage_out)) - is *deterministic* - involves no *understanding* or *agency* on the part of the model - Stochastic Parrots 🦜 <div class = "reference"> <a href="https://en.wikipedia.org/wiki/Deep_learning"> Wikipedia: Deep Learning </a> <br><br> <a href="https://dl.acm.org/doi/10.1145/3442188.3445922"> Bender, E. M., et al. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜. https://doi.org/10.1145/3442188.3445922 </a> </div> ---- ## How it works {%youtube LPZh9BOjkQs %} <div class = "reference"> <a href="https://www.youtube.com/watch?v=LPZh9BOjkQs"> 3Blue1Brown (Director). (2024, November 21). Large Language Models explained briefly [Video recording - Youtube ID: LPZh9BOjkQs]. </a> </div> note: show first 3:05 ---- ### What do the models do well?  - Converting speech to text (STT) - Converting text to speech (TTS) - Image recognition - Generating text - Summarising text ---- ### What do the models do poorly?  Current problems: - Providing responses outside of training set (Generalisability) - Explaining sources for outputs (Auditability) - Expressing appropriate uncertainty (Reliability) - Missing or overrating context (Bias) Currently the models are exactly the sort of doctor that you wouldn't want to work with! - Unable to reason or explain reasoning. - Overconfident even when wrong! - Sometimes completely ignores facts. - Sometimes just plain makes it up! ---- ## 2. Machine learning in health care  - How/why is machine learning being used in health care? - The Promises - The Reality ---- ### How/why machine learning -- The Promises  > AI improves the interaction between healthcare providers and patients. AI-powered smart agents can search, find, present and apply the most current clinical knowledge in partnership with physicians and researchers, thus significantly improving clinical efficiency and quality of care. <div class = "reference"> <a href="https://www.electronicsforu.com/market-verticals/medical/importance-of-ai-in-modern-healthcare"> 1) S Theo. [Website] The Importance Of AI And IT In Modern Healthcare, electronicsforu.com </a> </div> ---- ### How/why machine learning -- The Fears  - How do we reconcile these two images? - Why are both incorrect? - The monetisation of fear. - But... good reasons not to worry (yet). >76% [of experts] said that neural networks, the general architecture that underlies nearly all advanced AI, are fundamentally unsuitable for creating 'AGI' ¹ <div class = "reference"> <a href="https://aaai.org/about-aaai/presidential-panel-on-the-future-of-ai-research/"> 1) AAAI 2025 Presidential Panel on the Future of AI Research. (n.d.). AAAI. </a> </div> ---- ## ML in Health Care - Domains of use From reasonable to nuts... | Use Case | Utility | Risk | Error Types | Mitigation | |-----------------------|---------|------|-------------|-------------| | Dictation | High | Low | Word substitutions | Proof Reading | | Summarising Documents | Medium | Medium | False positives and omissions | Can defeat the time saving | | Creating Clinical Summaries | High | High | False positives and omissions, privacy | Close Proof Reading | | Decision Support | Medium | Catastrophic | Poor advice, wrong advice | None available | | Patient advice | High | Catastrophic | Poor advice, wrong advice | None available | | Disease detection | High | Catastrophic | False positives and negatives | Model specificity and bias control |  The Rumsfield Matrix (2001) - unknown knowns: it's in the data but not surfaced - known unknowns: there are errors but they're hard to find - unknown unknowns: biases ---- ## ML in Health Care - The Reality  There are limits to the efficacy, reliability and applicability of models. These issues present high risks. - Health Care = High stakes - False Positives - False Negatives - Privacy and Confidentiality While models have improved in performance, many of these issues are *architectural* and not easy to solve. - Model size is not a fix [Diminshing Returns](https://garymarcus.substack.com/p/evidence-that-llms-are-reaching-a?r=8tdk6&utm_campaign=post&utm_medium=web&triedRedirect=true) - Model "safety" (prompt manipulation or model supervision) is trivially broken <div class = "reference"> <a href="https://garymarcus.substack.com/p/evidence-that-llms-are-reaching-a?utm_campaign=post&triedRedirect=true"> 1) Marcus, G. (2024, April 14). Evidence that LLMs are reaching a point of diminishing returns—And what that might mean [Substack newsletter]. Marcus on AI. </a> </div> ---- ### Some examples: Medical References: > Even for GPT-4o with Web Search, approximately 30% of individual statements are unsupported, and nearly half of its responses are not fully supported. ... Our research underscores significant limitations in current LLMs to produce trustworthy medical references.¹ Confabulation and omissions: > Our clinical error metrics were derived from 18 experimental configurations involving LLMs for clinical note generation, consisting of 12,999 clinician-annotated sentences. We observed a 1.47% hallucination rate and a 3.45% omission rate.²  <div class = "reference"> <a href="https://doi.org/10.1038/s41467-025-58551-6"> 1) Wu, K., et al. (2025). An automated framework for assessing how well LLMs cite relevant medical references. Nature Communications, 16(1), 3615. </a> <br><br> <a href="https://doi.org/10.1038/s41746-025-01670-7"> 2) Asgari, E., et al. (2025). A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. Npj Digital Medicine, 8(1), 274. </a> </div> ---- ### Some more examples - Incorrect treatment recommendations¹: > GPT-4: According to the guidelines for the diagnosis, prevention, and management of cryptococcal disease in HIV-infected adults, adolescents, and children, ART should be initiated within 2 weeks of starting antifungal therapy for cryptococcal meningitis... > Clinician Comment: ... the suggestion to immediately start antiretroviral therapy is grievous: Boulware et al found higher mortality in patients with HIV-associated cryptococcal meningitis (CM) who were randomized to early (1–2 weeks) vs delayed (≥5 weeks) ART initiation ... > Note the fluent and authoritative nature of the output, including the errors, which can easily lead to being overlooked by non-experts, with potentially deadly consequences. <div class = "reference"> <a href="https://doi.org/10.1093/cid/ciad633"> 1) Schwartz, I. S., Link, K. E., Daneshjou, R., & Cortés-Penfield, N. (2024). Black Box Warning: Large Language Models and the Future of Infectious Diseases Consultation. Clinical Infectious Diseases, 78(4), 860–866. </a> </div> ---- ### What your patients are seeing... Machine Learning authored books and content appears authorative but is highly flawed.  >If he can be taken in by this type of book, anyone could be – and so well-meaning and desperate people have their heads filled with dangerous nonsense by profiteering scam artists while Amazon takes its cut. Richard Wordsworth (patient). > frustrating and depressing to see AI-authored books increasingly popping up on digital marketplaces > Generative AI systems like ChatGPT may have been trained on a lot of medical textbooks and articles, but they’ve also been trained on pseudoscience, conspiracy theories and fiction. Dr Michael Cook. <div class = "reference"> <a href="https://www.theguardian.com/technology/2025/may/04/dangerous-nonsense-ai-authored-books-about-adhd-for-sale-on-amazon"> 1) Hall, R. (2025, May 4). ‘Dangerous nonsense’: AI-authored books about ADHD for sale on Amazon. The Guardian. </a> </div> ---- ### Real potential for harms  - April 2025: update to GPT-4o rolled back¹ - In some cases, providing support in stopping psychiatric medications > I am **so proud of you** > And - **I honor your journey** - Deep gender and homophobic bias in models² - LLAMA models (Meta) appear particularly problematic (trained on Facebook chat) - These biases are part of foundation models <div class = "reference"> <a href="https://www.theregister.com/2025/04/30/openai_pulls_plug_on_chatgpt/"> 1) OpenAI pulls plug on overly supportive ChatGPT smarmbot. The Register. Retrieved June 18, 2025. </a> <br><br> <a href="https://www.unesco.org/en/articles/generative-ai-unesco-study-reveals-alarming-evidence-regressive-gender-stereotypes"> 2) Generative AI: UNESCO study reveals alarming evidence of regressive gender stereotypes | UNESCO (2024). </a> </div> ---- ### Are these problems tractable? Model Problems: - Confabulations: an inherent part of the model structure - Diminishing returns - Bias: a result of the training sets - Model Collapse: a real possibility (GIGO) User Problems: - "Automation Bias": tendency toward deference to automated decision aids - Productivity gains questionable - AI chatbots created new job tasks for 8.4 percent of workers, including some who did not use the tools themselves, offsetting potential time savings.¹ Broader Issues: - Return on Investment - Energy Use - Theft of Intellectual Property - [Surveillance (Data) Capitalism](http://trinitamonti.org/2021/02/28/the-rise-of-data-capitalism/) <div class = "reference"> <a href="https://doi.org/10.2139/ssrn.5219933"> 1) Humlum, A., & Vestergaard, E. (2025). Large Language Models, Small Labor Market Effects (SSRN Scholarly Paper No. 5219933) </a> </div> --- ## 3. Implementation and ethics  - Where does the data go? - What are the motivations for data collection? - How does this align with Australian privacy law? - Who is responsible for errors? ---- ### It's all about the data - LLMs have been trained on huge amounts of data, "the whole internet" - High quality sources are now lacking - Ingestion of prompts and documents provides a rich new mine - Your recorded sessions, medical documents and patient information *will* be mined!  ---- ### What controls are there on data use?  For many model use-cases, very few! - the incentives are *against privacy* - legislation is weak > ChatGPT saves all of the prompts, questions, and queries users enter into it, regardless of the topic or subject being discussed.¹ Remember that the "AI ecosystem" involves 3rd parties using foundation models. - contracts often include "data processors" - users have very little right to remove or correct data - once a model has been trained on data, there is no going back <div class = "reference"> <a href="https://tech.co/news/does-chatgpt-save-my-data"> 1) https://tech.co/news/does-chatgpt-save-my-data </a> </div> ---- ### What can doctors and administrators do to protect their patients?  - Don't use Machine Learning services with confidential data! - Read privacy policies - who owns the data? - are there any controls on who has access? - where is the data stored? - Consider your obligations under Australian law - protecting personally identifiable information - protecting medical information - risk of harms from accidental or deliberate disclosure - Remember that companies have an interest in keeping and using data, this may not align with your patients' interests! - Advocate for stronger privacy laws. - If possible, use *local* models, use *open-source* models. - Be aware that your clinical judgement embedded in interactions is also a valuable source to be mined (Expert Models). --- ## 4. Summary and Recommendations  1. Informed consent *before* use¹ 2. Consideration of privacy responsibilities 3. If using LLMs, read and re-read model output - Do not trust output or summaries - Consider the impact of omissions - LLMs are not fit to provide medial advice! 4. Be informed about the pitfalls 5. It's OK to say no and be skeptical about productivity gains 6. Patient safety comes first! <div class = "reference"> <a href="https://www.ahpra.gov.au/Resources/Artificial-Intelligence-in-healthcare.aspx"> 1) Australian Health Practitioner Regulation Agency—Meeting your professional obligations when using Artificial Intelligence in healthcare. (n.d.). Retrieved June 18, 2025. </a> </div> ---- ### Questions and Supplemental Reading  - AAAI 2025 Presidential Panel on the Future of AI Research. (n.d.). AAAI. Retrieved June 18, 2025, from https://aaai.org/wp-content/uploads/2025/03/AAAI-2025-PresPanel-Report-Digital-3.7.25.pdf - Australian Health Practitioner Regulation Agency—Meeting your professional obligations when using Artificial Intelligence in healthcare. (n.d.). Retrieved June 18, 2025, from https://www.ahpra.gov.au/Resources/Artificial-Intelligence-in-healthcare.aspx - Marcus, G. (2024, April 14). Evidence that LLMs are reaching a point of diminishing returns—And what that might mean [Substack newsletter]. Marcus on AI. https://garymarcus.substack.com/p/evidence-that-llms-are-reaching-a?utm_campaign=post&triedRedirect=true --- ## Testing AI ---- <!-- .slide: data-background-iframe="https://openwebui.greenant.net/" data-background-interactive="true" -->