Introducing the First Intelligent AI Knowledge Agent in Healthcare, Using Patients’ Medical Records for More Accurate Information and Routing
We started K Health with the mission of creating access to high quality medicine.
After seven years, we have an AI system that’s used in clinical settings, based around K’s AI-driven medical chat. Furthermore, we have integrated our offering with a world-class medical primary care offering that is now embedded into leading health systems.
And now, we have built the first Generative AI (GenAI) offering in healthcare that is more accurate and personalized to you.
A Major Departure From Other LLM-Driven AI Information Applications
We’re now adding an AI Knowledge Agent to our offering, with plans to be live soon. You can request access to demo the Knowledge Agent here.
The Agent, composed of an array of large language models (LLMs) enriched by our in-house algorithms, is a major departure from other LLM-driven AI information applications, such as ChatGPT, in the following ways:
- Personalized to you: Its answers to questions take into account relevant aspects of a patient’s medical history and electronic medical record (EMR), to provide patients with more personalized information.
- Doctor’s office digital front door: It’s built into our direct-to-consumer affiliated virtual clinic, and embedded into major health systems, to provide an intelligent “digital front door” to route patients to the right place to resolve their medical needs–whether they’re big or small, acute, chronic or preventative.
- Patient navigator: Our AI Agent can navigate you through the healthcare system to be in front of the right information, doctors and specialists. This massively empowers patients while giving doctors ‘AI super powers’ to diagnose and treat patients 24X7. This also means we can help you reach primary care and speciality doctors, labs and tests all within a single health offering.
- More accurate: K Knowledge Agent is purpose-built to provide more accurate medical information (see below)–with less hallucinations than other GenAI tools built on LLMs.
Read the article about K’s AI Knowledge Agent in Digital Health Wire
Hallucinations Don’t Have a Place in Healthcare
GenAI LLMs have a tendency to hallucinate–or essentially make up answers.
Hallucinations are fine if you’re writing a humorous wedding toast, but not in medicine given the need for highly relevant and accurate information. This is doubly important given the complexity and subtle nature of medicine.
This is why we took a different approach when creating our AI Knowledge Agent:
Optimized for accuracy
We’re optimized to provide full and more accurate answers by using select high quality health sources. If the answer to your question cannot be found in the sources, our AI will tell you it doesn’t know the answer. We also use a number of agents to verify the answer is adherent to the sources. These mechanisms dramatically reduce the number of potential mistakes.
Smartly integrated to the EMR
Integrating information from a patient’s medical history is key as it gives our models the capability to give personalized information based on a patient’s detailed medical information. However, this wealth of complicated information poses a new challenge to LLM models, and when this is done naively, it leads to inaccuracies.
In order to generate more accurate, personalized answers we used a multiple-agent approach: one agent filters only the parts of the EMR relevant to the question, and a different agent answers the question based on the filtered EMR.
How Does Our AI Knowledge Agent Stack up
We measured our Knowledge Agent against other LLMs for specific medical questions, and we looked at two crucial metrics:
- Hallucination Rate: This metric measures the number of mistakes, i.e. the number of times the LLM contradicts a statement that was written in the gold standard answer.
- Comprehensiveness: This metric measures how many of the clinically crucial statements are included in the predicted answer.
We also looked at coverage, which measures how many questions the LLM decided to answer.
In a manuscript currently under review for publication in a top-tier natural language processing (NLP) conference (preprint), we evaluated our model’s answers against state of the art models on a dataset containing 202 highly curated questions and answers. It is shown that our non-personalized answers are 9% more comprehensive and hallucinate 36% less than GPT-4, our best competitor.
Here are the full results:
| Method | Coverage | Comprehensiveness | Hallucination Rate |
| K Health Knowledge Agent | 96% | 62.9 | 15.4 |
| Bard | 95% | 62.5 | 28.4 |
| GPT-4 | 100% | 57.5 | 23.9 |
| Bing | 99.5% | 57.3 | 25.9 |
| GPT-3.5 | 100% | 56.2 | 27.9 |
Lastly, when comparing our AI Knowledge Agent to physicians who work in K Health’s affiliated clinics we can see where our Agent is particularly strong.
Preliminary results on 50 personalized clinical questions grounded in the patient’s EMR show that while both our Knowledge Agent and the physicians had a single mistake, our Knowledge Agent was more comprehensive. While physician answers reached a score of 0.40 in comprehensiveness, our answers reached a score of 0.62 (55% better).
Specific Examples of Demo Patients Simulating Real-Life Medical Situations
The following are specific examples of demo patients simulating real-life medical situations. Our AI pulls out the pertinent data points from the patient’s medical history to answer the question the user asks.
We included both the Knowledge Agent’s answer as well as the perspective on the Agent’s answer from Dr. Zehavi Kugler, Vice President Medical Sciences at K Health and Primary Care Physician.
In all cases, a user can click a button in the same experience to engage with K’s clinical expert AI and be in front of a doctor in a matter of minutes:
Let’s start with a user who is brand new to our Knowledge Agent. We do not have any knowledge about their medical history, therefore the Agent’s answer will be accurate but cannot be personalized. Next we ask the Agent the same question, but ground it with the patient’s EMR. Now, you get a highly tailored answer:
Here is a different patient, with a different medical history, asking the same question:
And next, the same question but to a patient with a different set of background diseases:
Patient with a history of pulmonary embolism asking about a medication that could potentially increase risk:
Here is another common symptom of higher heart rate for an asthmatic with pulmonary embolism. This could be due to several medical reasons. In all cases, a user can click a button in the same experience to engage with K’s clinical expert AI and be in front of a doctor in a matter of minutes:
And here is another common symptom of fatigue for a patient with complex background diseases. Again, the user can easily be in front of a doctor in minutes 24X7:
When people take multiple medications and supplements, you must take into account the potential of different active ingredients clashing. Here is a great example of the Knowledge Agent taking drug<>drug interactions into account:
When providing medical knowledge, you must also account for patients that have symptoms that may be side effects of medications:
