At Becker’s CEO + CFO Roundtable, leaders from Allina Health and Northwestern Medicine shared what they look for before approving an AI pilot, how they bring clinicians into the process, and what convinces them to expand.
An AI pilot can work in two clinics and still be nowhere near ready for the other 98. At Becker’s CEO + CFO Roundtable in Chicago, Allina Health’s Dave Ingham described an AI tool for routing inbox messages that worked in two clinics with similar processes. The remaining clinics handled the work differently, so the pilot couldn’t simply be rolled out across the system.
That example captured a theme from the panel, moderated by Qventus Chief Medical Officer Jason Cohen, MD, with Ingham and Northwestern Medicine’s Saadia S. Sherwani, MD: a pilot can prove that the technology works. Scaling takes a second decision about whether the workflow, the teams, and the economics are ready.
Define the operational problem before you pilot
Northwestern Medicine had already worked on first-case starts and room turnover. Surgical demand was still growing, and operating room capacity was finite. The question was specific: could the system make more of its existing OR time available?
That gave Northwestern a concrete way to judge progress. Sherwani pointed to earlier release of unused OR blocks and changes in surgeon behavior as operational measures that helped explain the financial return. On the panel, she reported that the perioperative initiative unlocked about 1,400 hours of OR capacity per month and delivered approximately 15x ROI across the system.
Allina takes a similar approach to the decision itself: agree on the expected return before committing to a pilot, then make a separate decision about whether to expand. That matters when getting data into and out of the EHR requires real time and resources from teams already balancing other priorities.
Bring the people doing the work into the design
When Allina introduced AI into pre-anesthesia screening calls, nurses had questions about what the change would mean for their work. Ingham’s advice was to involve the managers, directors, and frontline teams who know the workflow and will be responsible for making it work.
The panel also drew a practical distinction between automating a task and handing over a clinical decision. For a prediction such as length of stay, AI can give teams a consistent starting point while clinicians retain the ability to override it. Cohen described a similar approach at Qventus: present outputs as suggestions and review how those suggestions will appear to clinicians before rollout.
Decide what has to be true before expanding
Northwestern’s academic and community hospitals need consistent data definitions and a clear understanding of where their processes differ. Allina’s inbox pilot showed the other side of that lesson: even when the technology works, differences in who handles a task and how it gets done can keep it from spreading.
Expansion can work when the underlying problem is clear. Ingham described an orthopedic referral initiative that identifies patients whose intended referrals didn’t go through and reaches out to them. He reported approximately 7x ROI for the orthopedic use case and said Allina was expanding the approach to cardiac care, neuroscience, and oncology. Each specialty has its own workflow and will need to demonstrate its own results.
For health system leaders deciding where to invest next, the panel offered three useful questions: What operational outcome are we trying to change? Who needs to help design the workflow? What evidence would make us confident enough to expand? Answering those questions before launch makes it easier to know what a successful pilot actually proved.
Want to compare notes on your health system’s AI priorities? Let’s talk about where you see the biggest operational opportunity and what you’d need to see before scaling.
