We Got Over 600 Ideas in a Week | The 229 Podcast with Derek De Young

This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

We Got Over 600 Ideas in a Week

[00:00:06] Bill: Alright, it's the 229 Podcast, and today I'm joined by Derek DeYoung. Derek, with Epic, and you guys are notorious for not, being real clear with your titles. I'm gonna call you a software developer, is that accurate?
[00:00:18] Derek: You nailed it, yep.
[00:00:20] Bill: Well, but you seem to be the guy who's answering questions around Agent Factory right now.
[00:00:26] Derek: Yep, yep, so… You're absolutely rude. I work in research and development at Epic. About a year ago now, I joined the Agent Factory team right when we started it, and have been working on it ever since. And if… at UGM, Judy put my face on a screen that says I'm the owner of it, so there we go. That's how it works.
[00:00:48] Bill: Yeah, so you are now owner, but owner is not a title, it's just sort of a responsibility.
[00:00:54] Derek: Yep, we, very famously do not have real titles at Epic, and we get told when we go to conferences we can make them up. So, there we go.
[00:01:04] Bill: That's fantastic. You know, I tried to get away with that at my company. I tried to say to people, hey, we're not gonna have titles. They wanted titles. They were, like, they absolutely, I don't know, they didn't know how to position themselves when they're talking to other people without a title. I was like, alright, I'll give you… but software developer's probably pretty accurate for the work that you're doing, I would imagine.
[00:01:24] Derek: Absolutely. Yep. What I usually say is we do research and development, so the development is one part of it, but there's also a lot of, of course, research that goes into what do we need to be doing next. So I like… I always like saying I work in research and development.
[00:01:38] Bill: Well, let's, we're gonna dive into Agent Factory today. I want to start with, you know, for those people who are out there who What has been the biggest misconception about Agent Factory?
[00:01:52] Bill: To date. Maybe not the… maybe that shouldn't be my first question, but I'm curious, because when I talk to people about Agent Factory, it's still a little bit all over the board what they perceive that it's going to do or can do.
[00:02:04] Derek: Yeah, it's a great first question, because I… So, do a lot of webinars, talk with a lot of people now about Agent Factory, and usually one of the first things I say is. My main job here is to manage expectations, because we are still very early in this journey, and it's not just at Epic. If you look across the industry, not just healthcare, of what is happening with software as we move from
[00:02:29] Derek: You know, this… age of, like, a user-based, user interface-based, software into this orchestration-based, agent-based infrastructure as well. There's a lot of changes happening, a lot of learning happening.
[00:02:41] Derek: And we're on that journey at Epic as well. But the main thing I would say is we're still early. Like, we are building this, and one of the main things that we wanted to do as we're building it is build it with the community. It's one of the things that we're really passionate about at Epic, is building in collaboration with our users. So…
[00:03:01] Derek: we put the Agent Factory Builder, which is the part of it that I think a lot of people are excited about, to allow people to build agents in their own system using the same infrastructure that Epic uses to build our agents. We put it into a preview, or it's called a development preview, and at this phase.
[00:03:23] Derek: everything that you've seen about Agent Factory is actually done with Epic staff building alongside those organizations. So, like any of the examples anyone may have seen on a webinar or at UGM, even the ones presented by customers. There's Epic Staff, excuse me.
[00:03:40] Derek: building alongside those organizations, and the goal of that is to learn, you know, what… how do we do this safely? How do we, you know, what enhancements need to be made? We're still building this platform as… as we're… as we're going here, so there's new enhancements coming out very rapidly, so we'll see something with a customer, it's like, oh, it'd be really easy
[00:03:59] Derek: It would be a lot easier if we could do it that way, and then, you know, two weeks later, there it is in the system, and we can keep building it as we're going. So I think that's, you know, a misconception that's out there today, is…
[00:04:10] Bill: I'll throw some… I'll throw what I think are some mis… so, I hear people say, well, it's just a low-code, no-code environment. I'm like, yeah, I think that's part of it, and then they say, well, it's just a workflow engine. I'm like.
[00:04:23] Bill: well, that's part of it, too, and then they say, well, there aren't really AI agents behind it. I'm like, well, there are… I believe there's… so, I'm getting a lot of different people who are looking at it going, well, it looks an awful lot like, you know, some of the… The old automation platforms that we've had. And you're exposing
[00:04:44] Bill: the… I mean, from an architecture standpoint, it looks like you're exposing some of these things to Agent… the internals of Epic, to Agent Factory, and then you're putting a… a protective sandbox together that people can actually build things that used to take, okay, let's get the development team involved, let's put together a development lifecycle, let's… I mean, that's what it feels like. I mean, how would you describe it?
[00:05:08] Derek: No, yeah, spot on. The… I would say the other misconception is because it's no-code, low-code, it is easy. I think that's another misconception as well. just because it's no code, low-code, does not mean there's no logic that goes into it, and, like, I can build something really quickly. Honestly, you can build something very quickly, but is it something you're going to want to put into production? Is a different question.
[00:05:33] Derek: So it's not also just the builder. That's another misconception, is it's just the builder. There's a lot of other layers around this to make sure that there's trust and confidence in it, monitoring, evals, like, all the other pieces to make sure that organizations have confidence. But what the Agent Factory Builder is.
[00:05:51] Derek: is it is similar to a workflow engine for the workflow portion of Agent Factory, where we call them flows, that we build flows. there's a trigger that's initiated in Epic, we start that flow inside of Epic, and you can build fully deterministic workflows in Agent Factory, or you can build a fully Agentic workflow. So there's a… there's a
[00:06:14] Derek: Kind of a… breadth of what you can do in Agent Factory, from fully Agentic to fully deterministic. In healthcare, a lot of that is needed, where it's not actually… you don't actually want a fully agentic workflow.
[00:06:26] Derek: For something like a RevCycle, you know, work queue automation, maybe what you actually need is something that's 90% deterministic, and you need to have one Agentic component to it that can reason determine its plan, take its action. So, with Agent Factory, you can actually build across the span of that, which is helpful.
[00:06:47] Derek: We are doing what you said, though. So we are exposing some of the core components, tooling of Agent Factory into this sandbox. So we call it a component library. Where you can drag these components into a flow, and actually provide them as tools to an agent.
[00:07:07] Derek: So if you think about an agent as, you know, it's an AI that has access to tools to develop its own plan to solve the goal it was given. So in an agent factory, you can… Create an agent, you can provide it a goal, or a system prompt of that goal, with that goal in it, and then provide it its own tools, and it will choose when to call those tools to complete its task.
[00:07:30] Derek: There's other ways you could orchestrate it as well to do it more deterministically, but again, you can do that span in there.
[00:07:39] Bill: Alright, we're gonna… we're gonna… we're gonna dive into a specific agent. Before we… we get too far, though, I wanna… I want to sort of level set with people, like, what, what, what is actually available today? What's GA? What's early adopter status? What's special program kind of stuff? Or is this, like, hey, it's turned on, and just give us a call, and we'll get you going?
[00:08:02] Derek: Yeah, so this is, again, going back to managing expectations. I know there's a lot of excitement out there. So right now, we're still in what we're calling the development preview, so everything that we're working on.
[00:08:14] Derek: We're working with, you know, specific groups as new functionality becomes available, and development teams across Epic are going out on site to work with groups, potentially even, you know, Epic. The other thing to mention is Agent Factory
[00:08:29] Derek: is actually Epic's platform for agents across the board. It's not just for customers to build. So this is the same platform that Epic is using to develop agents. Of course, we can write our own code, it's not just load code for Epic, but the platform across the board is Epic's platform for agents, building agents, monitoring, evaluations. The Agent Factory Builder is the portion that we're releasing for customers to build upon.
[00:08:53] Derek: So our development teams are building agents in factory as well. They can go out on site and iterate with organizations. This is what we're doing in a development preview. We also have a group called our Operational Aces, which is kind of like a services arm of Epic that can go out and work on specific,
[00:09:14] Derek: like, operational objectives that they have. I want to improve capacity, I want to improve rev cycle. That team has started to get access to factory as well, with R&D on-site as well. But we are still in this dev preview. In October, at the end of October, we're opening up to early adopters. So what that means is we have a checklist.
[00:09:37] Derek: They have to be, you know, on the right version, they have to have the right infrastructure in their system, and then they can submit an idea to get into early adopter. So we're still trying to make sure it's the right fit for Agent Factory.
[00:09:51] Derek: We're not trying to, you know, just open it up wide right now with, you know, thousands of ideas coming in, and then people's expectations aren't meeting reality. We want to make sure it's a good fit for the tool. And then people will come in and train in, get to be trained in October, and then they can build themselves. So, probably starting in November-ish, people will be building in their own systems.
[00:10:15] Derek: We'll likely be doing that for 3-4 months, as we learn with those groups, and then looking to get into GA sometime early, you know, Q1, Q2 next year would be the goal.
[00:10:27] Bill: you know, I can picture UGM or XGM next year, where we're sitting there, and just… people are just talking about their workflows that they've built, and that kind of stuff, almost trading them like playing cards, like… If they're on the right system, it's like, hey, here, you know, here's what we built, here's what we've done. I want you to walk me through, though, what this looks like. The, you know, we…
[00:10:51] Bill: working with workflows for many years, I mean, it's… there's a trigger, there's data, there's reasoning, there's… There's guardrails, there's output, human checkpoints, and those kinds of things. Give me an idea of what it looks like. What am I looking at?
[00:11:05] Bill: And how can I build out that workflow? Does it start… like, everything with AI now feels like I start with just telling it, hey, this is what I would like to build, and it starts putting stuff on the screen, and then I should start moving those things around. Is that how it feels?
[00:11:19] Derek: So right now, in the dev preview, we don't have that specific feature where you can type to it, and it will just build the flow. We're actively working on it, though, so by the time it's in GA, you will actually be able to either talk to it, or type to it. I want to build a flow that does this, and it will actually give you a starting spot, and then you can tweak.
[00:11:38] Derek: For the workflows we do right now, you will come in with a problem. So a good example that we've done is something, let's say, let's do a simple one. So there's a challenge in inpatient where…
[00:11:54] Derek: you know, doctors are having conversations with a patient about, you know, the code status. So, like, do they want to be, like, do not resuscitate? Do they want to be, you know. comfort care, whatever it may be when they go into the surgery. They have these conversations, a lot of times they'll document it in the notes, but it isn't actually aligned with what is placed in the order. So, like, what's actually ordered.
[00:12:17] Derek: And what's ordered is what's going to be impacting what happens if there's a code during a surgery. So, there's misalignment. So we'd worked with a group that wanted to solve this problem.
[00:12:29] Derek: So we built an agent that was actually triggered on note signed. So the moment a signed note in inpatient, using an OPA, so using a lot of the framework that's in Epic today, it's a practice advisory in Epic. And we can use almost every single practice advisory to trigger an agent now, as well, or a flow inside of Agent Factory.
[00:12:51] Derek: So we built an OPA that triggered on the criteria that they wanted, that hit factory, that start node starts the workflow. It actually starts with kind of no data. Basically, we know… there's not, like, a message, right? So someone's not chatting with it to say, this is what I want you to do, it's just an automated trigger. So the data it starts with is the note. Like, this is the note it was triggered on.
[00:13:15] Bill: And the trigger can be discrete data elements, or it can be… or it can be, unstructured data, like, just something triggered in Epic, and it goes, okay, now trigger this flow.
[00:13:27] Derek: Yep, so you can think of it very similar, if you're familiar with Epic Build, you can think of it similar to, like, an extension record or an LPP in Epic. Like, there's these… triggers that happen all over Epic, not just OPAs, something getting added to a work queue, or claims processing. There's hooks all throughout Epic that can initiate work to happen using extensions, and then we are building more and more of these for factory as well. So.
[00:13:52] Derek: The moment a note is signed, we can trigger this flow in factory to start. That starts in factory with a start node, and we can then drag that into a language model, or an agent component, and basically say, tell that agent your goal is to extract out any mentions of code statuses. So basically.
[00:14:15] Derek: And we also feed it deterministically using smart text or a tool to get into the actual ordered code status. So that agent knows the code status that's actually ordered, and the note.
[00:14:27] Derek: And it says… we could tell that agent to look for discrepancies. We can do structured output, so we could basically make a JSON payload, to say, like, here's the 5 data elements I want you to find. Or we can say, just make a summary. In the one that they started with, the organization started with, it was…
[00:14:47] Derek: you know, we're taking out, like, does it agree or not agree? Is there an order recommendation for a change? And then, is there a summary? So that agent produced 3 data elements. They can save those to smart data elements, they can save that directly to what we call a large language model, so like this audit record in Epic. And then we can present that in Epic.
[00:15:10] Derek: So, one of the things we can do on saving it is, you know, trigger a refresh in the screen, which will make OPAs try, like, fire again.
[00:15:19] Derek: So we actually fire an OPA, a second OPA, to say, the agent's done running, did it find anything? If it did, we can present something to the doctor to say, hey, it looks like you mentioned code status, it's different than what was ordered, do we want to change the order? And there's an order recommendation with a summary, with citations back to the… You know, the note. It's just one example.
[00:15:41] Derek: But that is something that could be done. in Agent Factory today pretty easily. We've, you know, did multiple iterations with the organization. It's in production now, and Doing its job very well.
[00:16:00] Derek: So that's one example. We're also doing some in, you know, capacity management for bed planning, transfer center for transfer center summaries. So you can do anything from, like, fully assistive, if you wanted to do something fully autonomous. We don't actually have groups doing that yet, because, again, it's a journey of trust.
[00:16:19] Derek: But… You can do fully assistive or fully autonomous in that… in Agent Factory.
[00:16:26] Bill: That's pretty cool. I mean, as you're describing it, I'm, I have a fair amount of background with N8N and Zapier and those other things, and it's… it's those same… same kind of elements. Like, you… you have a trigger, and then you… you, initiate some workflows, and it's all… I mean, N8N and Zapier work across, you know, the internet architecture, which is…
[00:16:49] Bill: you know, all these… these, calls that go out to these APIs and pull information in, and obviously you're working on top of the Epic APIs, and then massaging that information and generating new data elements, and then you have actions, as it… as it comes out.
[00:17:07] Bill: Give me an… what's… what's… when someone builds this, what's the output? Is the output… it's a flow? Is the flow… can I, like, export the flow as JSON, or is the flow, like… Is it… is it code that gets, like, embedded in the, in the Epic workflow itself?
[00:17:29] Derek: Yeah, so what you build in Factory is… we call it a type of artifact in Factory. Under the hood, it is like a JSON-esque type representation of the, you know, what this flow
[00:17:44] Derek: encompasses. It will have the graph of what this thing looks like, along with, you know, the components that are in there, the settings of that flow, everything like that. That can be… if you work with your Epic representat… like, representative, it can be exported. We also support, like, Data Courier, so…
[00:18:05] Derek: just like any other Epic build that's done in Epic, you'll build this initially in a non-prod environment, and then you'll data courier it to others. So there's actually a record inside of Chronicles, inside of your database, that represents that flow. The build itself is all done inside of Nebula, so that's our cloud-managed platform.
[00:18:28] Derek: So there's a linkage between those two, and when you data courier something from one environment to another. We'll actually look at the static build that's done in that flow, because you might be referencing a smart text record, or a flow sheet, and those identifiers may be different from version to version, so we'll update those just like we do with all other Epic build
[00:18:48] Derek: In the system as well. That also allows them to apply their governance structure of moving build to production, the same security policies, the same, you know, sign-offs, whatever that may be. And if they wanted to even go further, because this is Agentic, they could… they would already have the tools and data courier and content management to add on to that, if they wanted to.
[00:19:09] Bill: This is… I mean, it's exciting because you're abstracting a lot of complexity. And then you're putting these tools in the hands of people and saying, look. You're, you're at the, you're at the bedside, you're at… you're in the workflow itself. What's the… what's the thing that we… that you don't have that you need specifically? And now, they could be building stuff that's relevant for…
[00:19:30] Bill: for all Epic clients, or they can be building something that's relevant for just That health system and that community.
[00:19:38] Derek: Yeah, so you mentioned the trading cards thing at UGM and XGM. One of the things that will be available by XGM next year… is something that we're calling the Community Library. You might be familiar with the Community Library as it exists today in Epic for things like smart text records and, you know, other build that's shareable.
[00:19:59] Derek: across Epic organizations, we're going to try and take it a little step further with Agent Factory, where if…
[00:20:06] Derek: a group like ANOVA, for instance, who just built an agent with us in Dev Preview to say a measles outbreak happened, can we do a chart review on any of the potentially affected patients, and say, are they severely immunocompromised? What's the next step we should take?
[00:20:24] Derek: It saved their nurses 15… you know, they did it 15 times faster, these chart reviews, using that agent. And Nova's like, I want to share this with the community. We're like, that's awesome. So we want you to share it as well.
[00:20:37] Derek: So we're gonna build something called the Community Library, and once they build it, test it, build evals for it, they will actually be able to publish it into the community library, along with, you know, a write-up of how it works, how it was tested, you know, any outcomes that they had with it.
[00:20:55] Derek: And inside of the Agent Factory, like, builder, you'll actually be able to go to the community library and see what the community is building and sharing. So I think there will be this mixture in the community of people who are, you know.
[00:21:09] Derek: trying to create things new, but also others that are just watching and saying, this is incredibly valuable, I'm gonna take this into my system and make it work for me, and keep going. And that's fine, that's why we're building it. So we're excited about the community library, what the community can build together.
[00:21:26] Derek: We think it's gonna be great. The other thing I would mention, I kind of hit on this a little bit, but we… you know, Epic is using this platform as well for Epic released agents, Epic released AI functionality.
[00:21:43] Derek: customers can use the builder for creating completely new, unique workflows as well. There's also this middle ground of,
[00:21:52] Derek: what an organization really wants to do is take what Epic made and extend it slightly, or give it a new skill. So whether it's Art, Emmy, or Penny, Art is our agent or persona for clinicians, Penny for operational work, and Emmy for, like, patient-facing in MyChart. So I'll give an example of this. We call it,
[00:22:17] Derek: Judy said this at UGM, I mentioned it as well, but we call it… with Agent Factory, you can do three things. So you can take what Epic's… Take what Epic's shipping out of the box, and there's a lot of functionality. We're going to continue to release functionality every quarter.
[00:22:31] Derek: The second bucket is shaping it. So how do you take what Epic is doing and shaping it to your organization, making it work as a trusted member of their team? And then the third one is making it. So take it, shape it, make it. Making something completely new. what I… if I look forward.
[00:22:49] Derek: What I actually think will be the most popular bucket inside of Agent Factory is going to be this Shaping It bucket, where Epic's going to be creating things like Ask Art. that you can just, you know, a chat agent inside of, you know, a clinical workspace, or ask Penny, or ask Emmy for patients. What a lot of organizations are going to want to do is just extend those. Give them additional skills.
[00:23:13] Derek: For, like, the technical people who are listening, think of it very similar to extending it with a skill like Claude Code or Codex, or ChatGPT. You can provide it a new skill or connectors. So we're working on that right now. We might have a first version of that out before the end of the year, where, like, with Ask Emmy. you can actually extend Ask Emmy to have organization-specific skills.
[00:23:38] Derek: So, like, here's my, visitor guide with my knowledge base and how to talk with my patients, or in Ask Art. I want to build a skill for art that I can ask it a question about maybe something bad happened, and I want to do a root cause analysis for why this happened, and here's my policy of how I build my RCA documents
[00:24:01] Derek: at my organization. Now, art, I'm gonna give this to you, and you just build it. So there's gonna be this kind of middle ground that is shaping it that I think will be a really popular thing that most… that most groups will do.
[00:24:15] Bill: Who, who can build, who can test, who can publish to production? Is that… is that a decision of… is that, like, a health system decision of, you know, where… how they assign those… those different levels within their organization?
[00:24:31] Derek: Yeah, so there's, again, different security classes for different things to do inside of Agent Factory. You can just be an auditor, so you can see what's happening, you know, understanding the monitoring side of it, even viewing the evals, but you can't build. To be a full builder, right now we're requiring a certification, so there will be a training here at Epic to walk through the tools. It is…
[00:24:53] Derek: an advanced toolset, probably one of the more advanced toolsets that we have at Epic, so we're gonna start with the certification, for sure. That, again, will be starting at the end of October.
[00:25:07] Derek: What most organizations are looking at right now, and as we've seen with the early adopter, kind of the dev preview groups, is the groups that will be… the people that will be most successful are not necessarily just
[00:25:20] Derek: you know, software developers, maybe that know agents inside and out. It's actually going to be someone who is really passionate about AI, trying to keep up to date with AI, but also knows Epic inside and out. So likely some of your senior analysts.
[00:25:36] Derek: who know how the system works, have been, you know, working potentially on different areas inside of Epic, they know work queues, they know, you know, print groups, they know extension records, how OPAs work. in addition to the AI side of… you know, I can… I…
[00:25:54] Derek: you know, a logical thinker, curious in exploring the AI, those people are probably going to be the most successful as a builder, rather than someone who is just a software developer and doesn't know about Epic. Because it's not just about building the agent, it's about actually impacting an end user's workflow, and doing the right work inside of Epic, and knowing when, specifically, really importantly.
[00:26:17] Derek: It doesn't need to be an agent. Like, we could do this with a registry and a rule, or we could do this with, you know, better WorkQu build and an LPP inside of claims processing that just automates this thing. I don't need to build an agent to do this.
[00:26:32] Derek: So that's going to actually be a big part of our training, is trying to identify when an agent is actually the right fit for what we're building, because we don't want health systems to just spend a bunch of money on tokens when it could have just been done discretely in the system the right way.
[00:26:48] Bill: I was talking to a CIO who did mention, he goes, you know, the first thing we have to do is make sure that our team doesn't get ahold of this tool and start building stuff that's already in the system. Like, it's… there's so many features within Epic itself that may or may not be turned on. I do want to, you know, from my experience, I've already had a week where I ended up spending $1,000 in tokens.
[00:27:10] Bill: Which gets us to this whole auditing, thing. You know, what's the auditing record look like? What are the inputs, the retrieved sources, all the things that we need to make sure that this thing is,
[00:27:25] Bill: First of all, it's delivering what it should be delivering, it's doing it in the way we expect it to be doing it, and it's doing it as cost-effectively as it could be doing it.
[00:27:35] Derek: Yeah, so there's different levels of this, of course, for who we're presenting this information to. So I'll start at the highest level, and I'll work my way down. So we have dashboards in Epic for… we call it a governance bet dashboard that is directly inside of hyperspace, so you can see all the agents that are working inside of Epic. This can be Epic released agents, of course, for, you know, a lot of the core ones are in there today, and we'll be expanding that.
[00:28:00] Derek: When you… They can, you know, raise alerts when something goes wrong, but you can drill into them, you can see the usage, you can see the feedback, the thumbs up, thumbs down. You can also then tie that to metrics in your system. Of course, because we're working inside of Epic, we also have our entire reporting suite inside of Epic as well. So right in a single dashboard, you can see that. That's more of, like, the operational leader view.
[00:28:25] Derek: We also have, two other dashboards that I think are really valuable in what we call Pulse. We have our feature adoption dashboard, so you can see, for Epic release features, what am I using, at what rate compared to the community? So if…
[00:28:43] Derek: you know, I'm looking to evaluate an Epic release feature. I want to see how the community is using it. You can view that to, you know, gain trust, gain confidence. And then the third one, which is a big focus right now across
[00:28:59] Derek: every industry is cost, so a cost dashboard. So you can actually drill into the cost per feature. If the feature is priced that way. Some features are priced slightly differently, like EMI, which is price per conversation, not per token.
[00:29:14] Derek: But if it is fully usage-based, there are dashboards in Pulse where you can see, updated daily, what is my, you know, cost that I'm seeing across these features, including features built in factory as well. So that's, like, the higher level, more for the operational leads. For… as we get lower, there are…
[00:29:34] Bill: So, am I gonna be able to… just based on that, am I gonna be able to see flow adoption and flow cost, like, broken out?
[00:29:41] Derek: Yep.
[00:29:42] Bill: Okay, that's… that's… well, that's fantastic.
[00:29:45] Derek: Yep, and then as we get down to even the builder, or someone who's, like, doing more of the monitoring, ongoing monitoring. For any of the custom flows that are built. Couple, like, terminology things. So each… interaction with… the flow, we call a run. So, like, if you…
[00:30:07] Derek: you know, a single OPA triggers, that thing runs. That's a run. Sometimes they're conversational, like Ask Art. So if you type in, hello, how are you doing, and you hit enter, that's a run. The entire conversation we call a session. So there's multiple runs in a session.
[00:30:24] Derek: Each run generates what we call a trace. So that is the very technical audit log that you were mentioning, Bill, of, like, what is the audit log? We call that a trace file. That trace file contains Like, every piece, every detail that you would need to debug or to walk through what… why did the agent really do what it chose to do?
[00:30:47] Derek: So in that trace, you can see, of course, how many tokens it used, you can see the exact cost of that run. But more importantly, as you're building, that will allow you to see every single tool call, every single deterministic component that was called, what were the parameters of it, what was returned back. So that is, like, your full audit log of that run.
[00:31:10] Derek: So from a builder perspective that has everything that they need in order to, you know, optimize that flow for cost, maybe we don't actually need the agent to call this tool every single time. Maybe we can do that up front and just feed it into a prompt.
[00:31:25] Derek: kind of up front, or maybe we split this agent into two agents that are working in parallel and combined together for speed. So that provides a lot of information on how to optimize. So all that information is there from the very low level up to the, you know, the metrics bubbling up to those operational dashboards.
[00:31:45] Derek: One of the things I'll mention, because there's a lot of curiosity from the community, is, like, can we get access to those trace files outside of Epic? That's something that we're working on, so we are planning to do that. So you'll be able to either export them, or kind of… maybe we'll create a web service to pull them out as well.
[00:32:05] Derek: There's a lot of groups that wanted to build, like, a, you know, a central… either to do research on them, or to do, like, do a central monitoring of all agents, not just in Epic, but across… their entire software suite.
[00:32:21] Derek: Or, you know, all their software. So that's something that we're exploring. Of course, if it's an Epic released agent, there may be some things that are redacted from it, like system prompts, but other than that, we're looking to support that.
[00:32:35] Bill: You know, one of the things… I just wrote an article this morning on… performance characteristics of machine-to-machine versus humans to machines. Human to machines, you know, we could measure. We could say, well, we have 10,000 users, and peak is, you know, 5,000 clinicians on the system. We're seeing this many patients. We sort of predict the amount of usage.
[00:32:56] Bill: computer-to-machine usage is a little different, and that's what agents are, right? It's… it's computer-generated, and… I mean, this is why the certification becomes so important. Because there are ways to create, potentially, loops, there's ways to create overload on the system, there's ways… I mean, I talked about my $1,000 mistake one week.
[00:33:20] Bill: on AI, where I just said, yeah, go ahead and do that. And all of a sudden, I'm like, oh my gosh, like, what just happened here? I mean, we talk about the abstraction, where abstraction… there's a lot of complexity behind this that people… Maybe you don't have to know everything, but they have to understand the concepts around what's actually going on, I would imagine.
[00:33:42] Derek: Yeah, a good example from our first build-a-thon that we did, which we invited some organizations out here to Epic to do some build. We kind of bucketed people together into different groups, so there's 5, 6 people working on a single agent. And initially, they were all working separately, and they got 2 agents doing the same exact thing. The outcome, like, the output of it looked very similar.
[00:34:06] Derek: One cost, I think it was, like, $1.10 a run, and the other one cost 10 cents. So, like, it was vastly different. And it came down to, you know, what parameters were in these, you know, components, or how was the agent, how did you tell the agent to go do the investigation? So it's definitely something that can…
[00:34:28] Derek: come up is the difference in cost, depending on, you know, how it's done. The other thing I'll mention is we're adding, of course, cost monitoring in there that I talked about. But it's your scenario, and I've done this myself as well, of like, oh, this was not the most efficient way to do this.
[00:34:43] Derek: Can we actually set, like, allowances on some of these agents? So, let's say I put something into production, I want to have a safeguard that this agent will not spend more than $500 a month. Or a week, whatever that may be.
[00:34:59] Derek: And I can set a budget, and once that agent hits that budget, I could kill it, just straight out kill it, or I could send an email to someone, so an alert to say, hey, by the way, we hit… this threshold, there may be something you want to do or change, because I think the transparency around, you know, what is happening in the system is critical here to gain that confidence in it as well.
[00:35:23] Bill: Well, I mean, that gets to the, really, the next question I had, which is, like, who can stop an agent? Like, is it only the developers and the coders, or can somebody on the front lines stop an agent, and how fast, and what, you know, what does that whole thing look like?
[00:35:39] Derek: Yeah, so it's…
[00:35:42] Bill: Because, I mean, we're talking about clinical information here. A doctor might look at it and go, hey, this is spitting out something that's not appropriate, and…
[00:35:50] Derek: Yeah, so it'll be different for each scenario, I'll say. Like, if it is a chat-based interface, like AskArt or, you know, another chat-based interface. and you're typing something, and, you know, it's loading, it's doing its reasoning, and you say… you see it's doing its reasoning wrong, you could, of course, just stop it. Like…
[00:36:11] Derek: That's more like a ChatGPT style, like, any user in there could hit end. The ones that are trickier are the ones that are more trigger-based. That are just happening in the background. You know, a claim comes in with a denial. If you set up an agent to trigger on a denial coming in, it's gonna keep triggering until you remove that trigger.
[00:36:32] Derek: That being said, unless you can add in, like I talked about, like, cost guidelines, or cost, budgets, or potentially other monitoring, alerts, it will just keep running as long as an IT analyst decides to turn that off. Of course, in the dashboards, you'll be able to see that it's running, and if it's going off the rails.
[00:36:55] Derek: But for right now, someone would have to make a configuration change in Epic, which is usually the analysts inside of there. Unless it's, again, an end-user interaction model.
[00:37:08] Bill: the, you know, one of the things with these systems. that we've been talking about is that we used to have release cycles that were very manageable. Like, you know, you get a new release every Tuesday, or every whatever. the… it's crazy, every time I open up one of these agents, and I… I mean, I have four of them running on my desk right now.
[00:37:30] Bill: But it feels like every time I open it up, I get a new release, I get a new update, I get a new whatever. I want to talk a little bit about that. Nebula enhancements are coming out pretty, pretty often. You know, people are at different levels of their, their build. Obviously, you want to be as close to foundation as possible. It makes this as… As, easy as possible, but,
[00:37:53] Bill: you know, which of the changes will affect the agent's behavior, and do we have a way of identifying those before we take those releases, I guess?
[00:38:03] Derek: Yeah, so there's a mixture here. We're doing, even just in Agent Factory, Somewhere between… probably 30 to 60-some enhancements, every single week now. So they'll… that'll continue, probably even go up. That being said, if we are…
[00:38:25] Derek: Making a change to a component, or a tool of an agent, or an agent itself. That would potentially or definitely change how that component works. Inside a factory, there's a banner that shows up on any of your flows or any of your components to say, hey, there's a change available here. That you may want to test.
[00:38:48] Derek: and it would, like, highlight any changes in the system. So by default, if it's, like, a… potentially a breaking change, or something that would change how your flow works. it wouldn't automatically change the flow. We're still going to release the code, but it would be backwards compatible with, like, a banner on the top to say, hey, there's a change here that you would want to test before you upgrade it. So you…
[00:39:09] Derek: in factory, you'd be able to see these are all the ones that are pending some changes. So, like, hey, there's a new notes component that added a, you know, new metadata that comes back from, the notes. You may want to test this because it's going into one of your agents, so you'll see this little banner above it. In your lower environments, you can
[00:39:27] Derek: you know, take that change, and maybe you have to tweak a prompt or something, but then you can push that up to production. So we have, like, transparency on when it may be a breaking change. It's also one of the things to call out, like, when we as Epic build agents, this is what we're doing constantly as well. So, going back to, it's low code, no code, that's true.
[00:39:47] Derek: It also doesn't mean it's trivial, and part of releasing something and building something custom is the ongoing maintenance of that thing as well, which, you know, has to be… you have to be aware of, specifically for you know, the CIOs, or CMIOs, or…
[00:40:02] Derek: as you're looking to do this, if you're building something custom, you can build something custom for sure. It's not just the build part of it, it's the ongoing monitoring, it's the ongoing maintenance that goes into that as well.
[00:40:15] Bill: You know, I was,
[00:40:18] Bill: getting a little off-script here. I was looking at the chat GPT, so they came out last week with their health, or maybe 2 weeks ago with their health thing, and I noticed that they… We're connected to external, data sources, clinicaltrials.gov, there's some CMS stuff, ARX Norm, DailyMed, some other stuff like that. Do you think there will be… that… we will bring that kind of data in as well?
[00:40:41] Derek: Yep, yep, so absolutely. Like, we already have some of those, We're also doing… so we're looking at external connectors, I'll take a step back, in 3 different buckets. So the first bucket is, you know, an Epic-managed connector, so that would be ones that are just
[00:40:58] Derek: we believe are just gonna be generically useful to the entire community. Things like a weather, like, an agent should be able to go check the weather, or things like SharePoint.
[00:41:09] Derek: we'll just make a SharePoint connector, we can do that through Cogito Cloud now, so that will be an available connector that comes.
[00:41:17] Derek: The second bucket is, so the first one is Epic Manage. The second bucket is our healthcare network. So, like, our health grid, if you're familiar with that, like, the payer platform groups, the Aura groups, Discovery, you know, those types of organizations, including our health systems, like, through Care Everywhere.
[00:41:36] Derek: Where those groups, like Payer Platform is a great example. Through Payer Platform, they can start to expose their policy information, and that can just be enabled by default inside of ART. So, when art is doing, you know, chart with art, we can pull in policy information for prior auths. It's a great example. The third bucket is, connectors that are in vendor services.
[00:42:00] Derek: And there's two different types of major connectors in there. One of them is just knowledge connectors, like knowledge bases, public knowledge bases that people want to make available to the Epic community. A great example of that would be, some of them that Epic will maintain.
[00:42:19] Derek: like, FDA labels, some of the ones that you just mentioned. They're public data sets, we can make them available as well. But, like, Walters Kluwer has some functionality as well. They can make those knowledge bases available for connectors inside a factory, and our Epic release functionality. The second part of not just knowledge connectors is
[00:42:41] Derek: you know, app connectors. So you can think of that, if you're familiar, it sounds like you are, Bill, in the AI space, those would be, like, MCP servers. And vendors, or organizations themselves.
[00:42:52] Derek: could create an MCP server, and then, you know, through vendor services, make that available to their agents. And as we continue to enable more and more shaping of Art, Emmy, and Penny, even allowing art to, like, reach outside of Epic using those connectors, those MCP servers.
[00:43:11] Derek: to, you know, try to solve some of those swivel chair interfaces that we have today, where, you know, a doctor's working on one screen, they have to go somewhere else to do something else, and then come back in, which happens a lot in RevCycle as well, it turns out.
[00:43:27] Bill: The human integration engine? Human person integration engine, yeah.
[00:43:31] Derek: So that… we're excited about that. We're actually going to be announcing it more publicly at our Open at Epic conference in October. This concept of connectors, or connector… ability in our showroom inside of Vendor Services.
[00:43:48] Bill: The exciting thing to me on that was we had started taking on some risk at St. Joe's. And the complexity of that, of managing all those different sources of data and bringing that together and creating the dashboard for the clinically integrated network and that kind of stuff, was just…
[00:44:06] Bill: Amazingly hard. Now, we were not an Epic client, I'm sure it would have been a lot easier if we had been an Epic client, but but I see this as a real opportunity for value-based care organizations, because you can… you can reach into the payer world, you can reach into the, public space, and you can… obviously, you have the medical record right there. I mean, there's… there's… anyway, the…
[00:44:26] Bill: And I think this is what's gonna happen with Agent Factory, to be honest with you. I think people who are dealing with these problems on a daily basis, you're gonna give them the tools, and they're all of a sudden gonna start to see
[00:44:38] Bill: these opportunities. Give us an idea of what those… you've had a couple of these meetings up there in Verona, you brought people in. Give me an idea of, you know, how they transpire. Do people come in with a whole host of ideas, or as you're sort of explaining it, they go, oh my gosh, we can… We can… we can now see some things we're gonna do here.
[00:45:00] Derek: Yeah, so we did a number of webinars on our user web. And in one of them, we mentioned, hey, I think we're getting to the spot now where we'd like some people to come out and hit it with a hammer. It's still very much under construction, but we'd like people to come out and hit it with a hammer. So we made, like, a form for people to go on and submit ideas.
[00:45:23] Derek: We thought we would get a handful of them. We got over 600 in a week. That… and it was just the people on the webinar, like, that's the only place we made it public. And we invited, I think it was 20-some groups out, it was around 55 people the first one. It was a mixture, it was actually a lot of fun. It was a mixture of…
[00:45:46] Derek: like, CIOs, CMIOs, and analysts, like traditional Epic analysts who are doing work inside of ambulatory or RevCycle, and then some AI specialists as well. So, like, they all brought
[00:45:58] Derek: kind of this collaboration of people out. We did training, and then we did whiteboarding, and what was fun is we actually spent more time really in front of whiteboards together as groups. rethinking what's possible, because, like, there's two ways to do AI today. It is really this concept of
[00:46:21] Derek: you know, we're gonna apply AI to an existing process, or we're gonna potentially rethink the process, and change how it can be done with this new technology. So we spent a lot of time in front of the whiteboard.
[00:46:33] Derek: And what was fun is it wasn't just one organization working on them, we partnered them together. So it was, like, two or three organizations working together on a single agent to be built in Epic, whether that's, you know, we had denials, what was it good? Oh, we had, like, a revenue guardian agent for Botox procedures, we had a referral triage, we had a prior auth one, we had,
[00:46:58] Derek: ME agent to, like, customize ME with additional… it's kind of that shaping bucket, but also a call center agent. We had a post-discharge follow-up voice agent that was built using Hello World Voice and Agent Factory. So, like, all these groups are working together.
[00:47:16] Derek: Spending a lot of time on the whiteboards. The second day is just 8 hours of build, basically, inside of Agent Factory. We had R&D sitting next to them. And then the third day was show and tell. And that was the fun part, because every single one, I think we had 11 or 10 different agents that were made in the build-a-thon, every single one had something to show.
[00:47:39] Derek: Was it fully polished? No. But, like, you could see that… this could work.
[00:47:44] Bill: See the shape of it.
[00:47:46] Derek: Yeah, like, some of them were, like, very, very, like, this is ready, like, this could probably go into production tomorrow if we wanted to, after maybe some, like, evaluations, but some of them were, like, this functionally works, we would want to optimize this, we'd want to optimize that. But the discussion in that show and tell was really interesting, because they're like, now that I get the tool, I could do this, I could do this, I could do this, I could do this.
[00:48:06] Derek: and just, like, rambling off of where they see they could apply this, not only for end users, but, like, a lot of the ideas that came out of this was like, oh man, if analysts could do this, I could do this, that, and the other thing to make my job easier as an analyst as well.
[00:48:21] Derek: And we're actually building an analyst, build assistant agent into… into… into Epic that's also built on top of Agent Factory and extensible by customers as well.
[00:48:33] Bill: As you were talking, I was thinking, man, call center, I could probably do some really cool stuff in the call center. But regardless, I want to talk about… so we're going to close it out here. I want to talk about what health systems should be thinking about and doing over the next 90 days. Are there… you know, it's… we're getting into an interesting case Where it's,
[00:48:55] Bill: you know, you had the, the make, take, and shape, or… I did that in reverse, doesn't matter. Yep. But you had… you had those things. You know, what… how should I be thinking about this as a leader? If I'm a CIO, CEO of a health system.
[00:49:12] Bill: Is it… is it such that I have the epic roadmap now, and then I have potentially the roadmap worth thinking of, and the things that we could potentially fill in some gaps, or is there a group I should…
[00:49:26] Bill: Please do not say governance at this point, but is there a group I should be standing up, or should I be thinking about certain people, bringing them together to form the first team from our health system around this? I mean, what are some things you'd be saying to those health system leaders as you're sitting in front of them right now?
[00:49:45] Derek: Yeah, I think the biggest thing is you talked about the buckets of, you know, take it, shape it, make it. They also will change the complexity of, you know, the time investments put into those as well. Of course, if you're going to the full make-it bucket, there's going to be a time investment, of course. For some groups, that's gonna be great, and they're gonna wanna do it, and we're excited for that.
[00:50:09] Derek: I would say for each group, no matter what, make sure you're taking a look at that take-it bucket. Epic is spending an enormous amount of effort into trying to create these features and functionality for the masses.
[00:50:23] Derek: And we have now 129 features or skills available, across Epic. I think some of our leading organizations are in the 45 to 60, you know, bucket, if you look at, like, the top 5% of how many of these features are live in production. So there is still an enormous amount that is out there.
[00:50:43] Derek: that we have viewed as, like, very high priority at Epic for getting out functionality, so always take a look at just what's available, how can I embed it into my current users' workflows? Those are going to be the easiest, the quickest, in order to turn on. Are they going to be all perfect on day one?
[00:51:01] Derek: Absolutely not. But, like, the worst it's gonna be is today, because it's gonna continue to get better and better and better. And, of course, collaborate with us, give us feedback. As Shape It comes out, that's gonna be a big bucket. As you're looking at the Make It bucket, I think a lot of what… a lot of people have… you know, really good ideas, of course. Take a look at…
[00:51:25] Derek: how does it actually fit into the workflow, and should we… how do we design, and do we have that workflow well documented? A lot of times they'll have this idea, and then we'll start getting into the weeds of it, and we'll talk about, like, alright, so what's the procedure today?
[00:51:41] Derek: And it's like, oh, it's, you know, this nurse does this thing, and the process is all in her head, and it's not actually documented anywhere. So, like, that process engineering, that structural engineering to actually, like, document these processes is going to be a really important thing if we want to
[00:51:57] Derek: put it into a flow, where… also pull in those people as you're doing it. Don't just make this an IT project, make this an IT plus operations plus end users all together, collaborating. That's where we've seen the most success with this. Kind of this fusion of different roles coming together, because if you can get the end users bought into this
[00:52:19] Derek: operational change, and they feel like they're contributing to it, they're not going to feel like they're being attacked, or like we're trying to take their jobs, because that's not actually what anyone's trying to do. We're just trying to make the most of the resources we have so they can focus where they need to focus. If we could hire the way out of it, we would have already tried to hire the way out of it. There's no shortage of things to do in healthcare.
[00:52:43] Derek: But bringing those people in is a big win, but I really think starting to document the ideas, starting to document the processes around those ideas, and then, you know, getting together a group that can actually think through what is our prioritization here?
[00:52:59] Derek: of these ideas, because each one of these, of course, will have, you know, that utility cost of tokens. Which one's actually going to be worth it? And starting to rank those from an ROI perspective. I would not advise doing, you know, 50 at the same time, because it's not going to work. Start with one or two that are your top priority. Make sure they're well documented. That can start today. That can start right now. As the tools become more available.
[00:53:24] Derek: That'll be there. Also, just because Agent Factory exists, that doesn't mean that… you know. all ideas flow to, you know, the, you know, the Agent Factory R&D team here at Epic. You still have your IS, you still have your TS, you still have the R&D partners that you're working with that know those areas.
[00:53:44] Derek: like the back of their hand, talk with them, work with them. There may be something under development that's coming that, instead of building something custom right now, we could do more of those dev you know, Dev Preview, where the developers come out and actually build something that's planned to be released to the 600 groups. So don't stop communicating with Epic, because now you have tools to do it. If anything, increase communication, because I think that will
[00:54:08] Derek: you know, save you a bunch of time in the long run as well.
[00:54:12] Bill: So it's a team sport. That's what I… that's what I take away from it. It's,
[00:54:16] Derek: It's gonna be a team sport, yeah.
[00:54:18] Bill: Absolutely. Derek, I want to thank you for your time. I love the stuff that you guys are doing. I look forward to staying in touch and seeing how things progress. I think this is the ground floor of something that's going to be really exciting over the next couple years.
[00:54:30] Derek: Awesome, yeah, we're excited about it.