Fostering Digital Health Innovation: Developing the Software Precertification (PreCert) Program

January 30-31, 2018; Bethesda, MD; Full Report - Draft

Executive Highlights

The ~200-attendee FDA PreCertification Program public workshop took place over two days at NIH, focused on a status update on the in-development program, a panel with the nine pilot participants and the FDA, and how to define and measure a company’s “organizational excellence.” The goal is to enable a medical software developer to become “pre-certified” and receive a potentially streamlined pre-market review. FDA’s goal is a minimum viable PreCert program by the end of 2018 – what digital health leader Bakul Patel and team humbly call “Version 0.1.” This minimum viable PreCert program would have: (i) a library of metrics to measure “organizational excellence” (key performance indicators, or “KPIs”); (ii) clarity around streamlined pre-market review; and (iii) what post-market surveillance should look like. The focus of this meeting was on #1 – aligning on KPIs to define “organizational excellence.”

On Day #1, we heard from representatives from the nine enthusiastic pilot participants and the FDA, along with an excellent stakeholder panel. Day #2 was more challenging than Day #1, as attendees drilled down into the nitty-gritty of key performance indicators (KPIs) of company excellence. How can we objectively measure a great software company? How does a company know that what it is doing is great? How could FDA measure excellence and be assured the products a company is releasing are high quality, safe, and secure? Can metrics apply fairly to big and small companies? This is a much harder task than it sounds, and it speaks to the ambition of what FDA is trying to do with this program.

Visionary leader Bakul Patel (Associate Center Director for Digital Health) summarized the goal of PreCert for medical software in a nice analogy to TSA’s PreCheck program for frequent flyers. Those in TSA PreCheck still have to get security screened at the airport, but the process is far more efficient for them – they get through security faster and with less stress, as they’ve been vetted and accepted into the voluntary TSA PreCheck program ahead of time. The same would be true of FDA PreCert status: Digital health products would still be reviewed (especially high-risk products), but the reviews would be more efficient and streamlined, since the company has been vetted as “excellent.” One attendee took it a step further – if you’re the type of person that flies on an airplane once a year, it’s unlikely you’ll go through the hassle of applying for TSA PreCheck. However, if you fly every week, TSA PreCheck is a worthwhile upfront investment that will pay immediate dividends in saved time and stress. This analogy illustrates all of the things that FDA is wrestling with right now: How do you measure an excellent company? (What allows someone to qualify as a PreCheck flyer?) How often should an excellent company have to share its metrics to prove it is excellent? (When does Pre-Check have to be renewed?) How easy should PreCert status be to obtain and maintain? (Will only seasoned business travelers go through the trouble of applying for PreCheck?) How high should the bar for precertification be? (Does lowering the bar for TSA PreCheck make the PreCheck line longer than the standard security line? Does it overwhelm TSA?) We dig into many of these themes and questions in more detail below.

A notable takeaway from the entire meeting was not found in the specifics, but how FDA designed the agenda – on day #2, eight breakout groups dove into specific, thorny topics (e.g., measuring people, strategy, process) in three different sessions, with each group moderated by one member of FDA and one member from a PreCert pilot company. It was awesome to see members of the FDA taking notes on big white sheets, Post-It notes flying onto the wall, taking feedback incredibly well, and the entire conference coming together for three share-outs. It felt like a design thinking brainstorm with FDA and industry co-participating. FDA really showed – in spades – that it will co-create the program with industry and the broader community. Yes! Indeed, any tough question directed at FDA was lobbed right back at industry: “Please, tell us how you would do it.” We noticed many diabetes-related tech companies in the room, including Abbott, Bigfoot, Dexcom, J&J, Livongo, Medtronic, mySugr, Roche, Tidepool, Verily, Qualcomm, Insulet, Lilly, and WellDoc, among others.

Commenting on the docket is the best way to give your organization’s input: Bit.ly/docketjan18. You can also get progress updates at bit.ly/Precertupdates (note the large feedback button) and ask questions at FDAPre-CertPilot@FDA.hhs.gov. Agency representatives urged attendees to self-organize into coalitions to make submissions to the docket.

The discussion of “Company excellence” metrics was only step one, and it certainly did not arrive at a conclusion in this brief two-day meeting. Next steps also include more discussion on streamlined reviews (once a company is PreCertified, what will FDA reviews look like for the various devices?) and post-market requirements (once a PreCertified product is on the market, what data will need to be monitored and reported to the FDA and at what interval?).

A lot must certainly happen to launch a minimum viable PreCert program by the end of 2018. That said, the program is clearly happening, so the bigger question is how it will be implemented (the KPIs), what the benefits will be (the streamlined review), and how it will adapt over time.

Mr. Patel was refreshingly honest in his closing remarks: “We might not get to 100%, but we need to get to where we can actually try this [program] out. We need to get to a place where we can iterate…We plan on having an open dialogue throughout this process…If you have ideas of how to keep this momentum going, I’d be very open to that. We want to keep this crowdsourcing going forward.” Hear, hear!

Participant Panel Highlights

To be sure, development of the Pre-Cert program is still in the very early days. Notably, all nine pilot participants spoke on a panel on day #1 with FDA’s Bakul Patel: Apple’s Senior Legal Counsel Jennifer Newberger, Fitbit General Manager Adam Pellegrini, J&J Digital Health Policy Lead Diane Johnson, Roche VP Global Regulatory Policy Danelle Miller, Samsung SVP Yong Jin Lee, Tidepool CEO Howard Look, Verily Head of Global Regulatory Affairs Larry Carrier, Pear Therapeutics VP David Amor, and Phosphorous CEO Alex Bisignano. There was tremendous enthusiasm for the program and FDA’s very productive two-day site visits late last year. Apple was by far the most lukewarm about the program, with Ms. Newberger concluding, “The end result here has potential to be awesome, but it also has potential to be messy.” She later implied that Apple already builds great software, so it does not want to change how it does things; rather, it aims to be recognized through this program for what is already does. 

At the end of the panel, Mr. Patel asked the panelists for the one thing they would like the audience to hear. See their largely enthusiastic responses below:

Panelist

Comment

Danelle Miller (VP Global Regulatory Policy, Roche)

“They’re [FDA] serious. That’s what I want to really pass along. The Agency is serious about collaboration and coming up with a process that works for software.”

Howard Look (CEO, Tidepool)

“This is a revolution, a clean sheet of paper. Everyone has an obligation to share ideas and everyone in the world should be sharing ideas to make this as good as it can be. Let’s take advantage of it.”

Yong Jin Lee (Senior VP, Samsung)

“This is an unprecedented opportunity for the digital health industry. We should all participate and provide guidance.”

Diane Johnson (Digital Health Policy Lead, J&J)

Encourage everyone to think big. Not only innovation about product, but innovate how we do clinical studies, sensors, wearables. We could revolutionize other aspects of regulatory life.”

Larry Carrier (Head of Global Regulatory Affairs, Verily)

“I would say nothing is sacred here, and this is an unprecedented event, and let’s get to work.”

Alex Bisignano (CEO, Phosphorus)

“This is an excellent opportunity to build a world with constant innovation and in which we get tools to patients faster. It’s a reality. It’s an open dialogue, and I’m happy to be a part of.”

Adam Pellegrini (General Manager, Fitbit)

“Don’t hold back. If you have a great question, ask it sooner rather than later. You could go through the next couple of days, thinking ‘I should have asked that question a long time ago.’ This is about really active dialogue. Ask the questions, day one, right away, so you go into this incredibly informed. Provide inputs for this historic initiative.”

David Amor (VP, Quality & Regulatory Affairs, Pear Therapeutics)

“Don’t hold back. Don’t hesitate to reach out directly to the Agency on this. We’re looking forward to seeing what other crazy amazing ideas they have that could help facilitate software as a medical device and digital health products.”

Jennifer Newberger (Senior Legal Counsel, Apple)

“We all have to be comfortable being uncomfortable. The end result here has potential to be awesome, but it also has potential to be messy.”

FDA PreCert Core Team Panel

FDA has made it very clear that it is not lowering the bar for clearance or approval – or as one regulator put it, “we’re not creating an easy street.” Conversely, this is about more efficiency and better use of resources – if a company is pre-certified, FDA won’t need to see as much or anything at all. We think it could be game-changing for digital diabetes innovation, assuming it is reasonable for both small and large companies to get into the program.

A big highlight has been FDA’s clear interest in working with the community and in fact even co-creating the program with the community. We counted at least 20 calls for engagement, and the FDA team really appreciates agile development and a truly iterative process that learns over time. The best way to submit a comment to the program is through the docket: Bit.ly/docketjan18v. We would highly encourage every company in healthcare to submit something, even if it is solely to voice support. 

As in the participant panel, each member of the PreCert core team was invited to share their messages for industry and FDA staff about the program. We’ve captured some of their notable answers below:

Name (Position at FDA)

Comment

Francisco Vicenty (Program Manager, OSB)

“This is all about better information, better understanding about what’s going on, a better way to respond and react, so that the company can focus resources on other things that matter to them. As FDA, there are things we do not need to be reviewing. It’s all about getting smarter about our resources.”

Linda Ricci (Associate Director for Digital Health, ODE)

“When I talk to staff, particularly inside the Agency, it’s about making the best use of our resources. Being efficient. Not lowering the bar, just being more efficient with the resources we have. For people who can exceed bar, we don’t have to use the microscope on them. I’m making sure internal stakeholders know that.”

John Murray (Expert Regulatory Review Scientist, OCD)

“You may not know this but most of CDRH is a very conservative, risk-averse bunch. When they first heard about this a year ago, they got that frown on their face. I think we’ve been successful the past year. They understand we’re not creating an easy street, we have the same goals in mind, and when you explain that to them, they say, ‘yeah you’re right, there are many ways to get to the target.’ It’s a work in progress. We need to learn, communicate. We have a digital health hour once a month, where we pick a topic and talk to them. We’re not completely there yet, but as we unveil more of this I think they’ll see it.”

Martin Ho (Associate Director for Quantitative Innovation, OSB)

We have asked industry to think what it should be. To pick battles, focus on things that we can do the best. To have a better allocation of resources to continue to make it a collaboration effort.”

Marisa Cruz (Senior Medical Advisor, OCD)

“My impression of the FDA review staff, when they get to know an individual company, go through the process, they get to understand the dimension of company, where they are succeeding and trying. This effort is meant to expand that relationship to a sector of organizations. A lot of people are trying to meet or exceed the bar.”

Esther Bleicher (Senior Policy Advisor, OCD)

What I’ve heard from FDA staff is that first concern is, ‘are you lowering the bar?’ Once people believe we’re not, they get excited. There is a lot of internal interest. We have this tactical team, an enormous group, who are really excited about helping build the program from the inside.”

Adam Berger (Personalized Medicine Staff, OIR)

“I just want to echo again, people are interested in this program. They’re excited by it. It’s not every day we get to build a new pathway for regulation. We have casual team meetings, center-wide (CDRH) meetings. Every office in the center is heavily involved. Reviewers are interested in how we’ll make the best use of their time – lot of engagement, people are interested.”

Q&A with Tidepool and mySugr

Q: What are the three things that you think FDA needs to hear about the program? 

Tidepool: We've blogged a lot about this, most recently: https://tidepool.org/fda-precert-site-visit-recap/. In short:

  • We think the public wants the FDA to get things through faster, so this is awesome!
  • There's no way the FDA can review/approve every bit of software (or hardware for that matter) so this is awesome!
  • Beware unintended consequences. Let's make sure the program achieves the actual goals of getting safe/effective solutions out quickly.

mySugr:

  • We are delighted to be involved in helping the FDA develop a program which we believe can help the digital health market in this country develop in a major way. Hopefully, such a program will allow companies to better optimize processes/product development and utilize existing tools to meet FDA guidelines.
  • The advantages and incentives of the PreCert program will need to be clear upfront. Otherwise, there is the danger that it could feel just like an added extra step and there may be little interest from companies.
  • Being at the FDA, it’s understandable that the agenda this week has been focused on US companies, but as an international company, we would like to express the importance of aligning with European (and other) regulatory initiatives/directives focusing on digital products.

Q: What has been easier than you expected, and what hasn’t gone so well, if anything? 

Tidepool:

  • The good: The process has been very collaborative. We loved co-creating with the FDA. They definitely have the right motivations for this program.
  • The bad: We wish the 9 companies spent more time together, rather than in silo'd conversations with FDA. (We are having our first meeting with all 9 of us on Thursday).

mySugr:

  • The FDA has welcomed and encouraged feedback of all types from active companies in the space like us. Throughout the course of the two-day event, it has been a very open and collaborative environment in which to voice our priorities and concerns.
  • It seems difficult for people representing large organizations to understand what it’s like for start-ups. Trying to communicate how things are (agile environment, lean, frequent iterations, resources) has proven very difficult, If this program truly wants to be equally available for all organizations, this will need to be addressed.

Q: What do you anticipate will be the biggest pain points over next year? 

Tidepool: No one process is going to work for every company, and a lot of the things that the FDA is trying to do with this recognize this issue. Trying to boil everything down to metrics is tricky. It's also challenging to make a process that scales properly for small and big companies, or even DIY projects, and still achieve the goals.

mySugr: A vast amount of input, ideas, suggestions, and feedback was expressed during the last two days. It will be tough to reduce this to a meaningful and practical set of rules and/or guidelines. Moreover, we envision heated discussions as it relates to the diverging requirements of small and large organizations; it will be a challenging task to identify a common rule set that is applicable to organizations of all sizes and types.

Q: Do you have an impression of how their experience has differed working with Tidepool vs. a large company like Apple or J&J? 

Tidepool: It's hard to tell since we've been mostly silo'd so far. Tidepool is the only organization openly blogging about the process (see here, here, and here). I think the ideas we're sharing might be similar; but Tidepool would prefer to take a collaborative approach, whereas other companies may have more concerns about confidentiality as the process progresses.

mySugr: The diverging requirements for small vs larger companies were visible in many fields. Just to give one example: startups with only a handful of people, often sitting next to each other, do not require many different communication channels, whereas that might not be the case for a large company made up of multiple facilities/sites.

Q: Do you think the Agency has overlooked any factors that you consider critical for an organization to be deemed one of “excellence”?

Tidepool: It's too early to tell. In general, the "excellence principles" and "common validating perspectives" are right on, but the devil is in the details of the actually metrics and KPIs and qualitative evaluation mechanisms that we come up with. Not everything can be boiled down to a number.

mySugr: On the contrary, we had the impression that certain factors and metrics were discussed that are not necessarily indicators for “excellence.” However, we expect this to be subject to further discussion in the next months and are confident that the FDA will come to a meaningful definition of excellence in the context of this program.

Themes and Remaining Questions

1. What Are Meaningful “Company Excellence” Metrics That Apply Fairly to Large AND Small Companies?

The first step in the PreCert program development is defining and measuring what an “excellent” software company looks like. What internal company processes enable excellence? What outcomes show excellence? Can these be measured and standardized across large and small companies? Some felt that many of the proposed KPIs could give large companies an advantage in PreCert – e.g., HR practices and policies, employee retention, inclusion/diversity, brand reputation, social media ratings, market uptake, YOY growth, etc. If PreCert indeed favors large companies, it could be a disaster and even stifle innovation and exacerbate barriers to entry – if large companies already have a regulatory experience advantage to startups in the current process, a high bar for PreCertified status could then give large companies an even bigger advantage. However, FDA’s Bakul Patel emphasized repeatedly (at least five times) that the program must scale to companies large and small, though the devil is in the details – based on the example KPIs discussed on Day #2 (see pictures below from FDA’s slides), small digital health startups might have to put in a fair amount of effort to achieve PreCert status. That said, it could also encourage the truly excellent startups to be deliberate about their processes and show their greatness to the FDA. Hopefully the small organizations in the pilot (e.g., Tidepool, Pear Therapeutics) will make it clear to the Agency what is/isn’t feasible, but it is one of the biggest risks to the program and a potential area of unintended consequences.   

  • Tidepool CEO Mr. Look and startup Pear Therapeutics VP David Amor are both pilot participants and both shared optimism that the program could scale down to their level; of course, a lot of care will need to be taken to ensure the metrics are meaningful and not too burdensome. While their organizations don’t have the infrastructure of larger companies like J&J and Roche, the program’s propose flexibility could grant them the ability to be more agile and more quickly adapt and implement whatever KPIs for excellence that they agree upon with FDA. The proposed monitoring and reporting of metrics could be another challenge for small organizations, but multiple startup execs shared they already have most of the KPI procedures in place – in other words, if a company develops software worthy of being on the market, then chances are, the company is already “excellent.”

Example KPI Metrics from FDA’s Slides (right-most Column)

The FDA breaks things down into five “Excellence Principles”: Patient Safety, Product Quality, Clinical Responsibility, Cybersecurity, and Proactive Culture. There are five enablers of excellence (Leadership, People, Strategy, Partnerships/Resources, Processes) and four areas of “Results” (Customer, People, Society/Public Health, and Business). The slides below show KPIs in a mix of those areas.

2. What Metrics Outside of Traditional Product Safety and Effectiveness – Process, Product Development, Business, Culture – Are Reasonable Indicators of Company Excellence? What Is Not Fair for FDA to Look At?

There was a fair amount of back and forth between the regulatory crew and attendees over what FDA needs to know, should care about, or has jurisdiction over. Several suggested KPIs received staunch negative feedback from attendees, including financial viability, sustainability, social media ratings, etc. Mr. Patel pushed back a couple times against the kneejerk rejection by asking how attendees how they would gauge the excellence of suppliers or their peer companies – “it’s less about whether the FDA needs it, and more about whether the business needs it to be successful. Later on, we can decide if FDA needs it. If we ignore things that FDA doesn’t need, are we ignoring something that you guys care very deeply about as you run your businesses?” We heard at least two other arguments in favor of  FDA leveraging these sorts of KPIs: (i) “As a taxpayer, I don’t want FDA wasting time on companies that won’t be commercially viable”; (ii) “In the interest of continuity of care, it is of interest to FDA that companies are viable enough to not disappear in the middle of a patient’s treatment.” On the whole, however, the room shared Mr. Look’s sentiments: “The brand reputation part – let companies choose their brand. It’s important for a company to understand how their brand is perceived, but that should not be part of the regulatory process. Safety measures, adverse events, efficacy, incorporating the voice of the consumer in the product…Those are direct measures that are legit for the FDA to look at.” In addition, some added that the market will determine which companies are viable and sustainable based on their culture, value, product, reputation, etc… preventing or taking on that process is not FDA’s role.

  • One breakout session arrived at a compromise of sorts, proposing that there should be primary excellence metrics that FDA should care about, and secondary metrics that FDA would like companies to care about, but not report. For example, defect rate, adverse events, efficacy, and incorporation of the voice of the consumer might be primary metrics that FDA tracks, while customer retention, brand reputation, and the consideration of customer impact in executive decision-making measured through surveys might be secondary metrics. We like this solution, though it raises the question of where to draw the line between the two tiers.
  • How many metrics/KPIs does the FDA need to see? Philips’ Ms. Elizabeth George (VP of Global Government Affairs) pointed out that while her company collects “thousands” of metrics internally, KPIs should be kept to a “small few,” both for privacy and simplicity. The real answer is probably that it depends on how much risk the FDA feels a company carries, or how much close monitoring it should have. Based on the working approach to evaluation, it’s hard to know how large the library of KPIs will be and how many companies will need to pick. As noted above, there are five excellence principles, driven by five enablers of excellence and four areas to show results. We’d guess the full “library of KPIs” will have at least metrics, though how many a company needs to show is unclear… Selecting three-to-five metrics per principle would result in 15-25 KPIs, which seems manageable for a software company of any size. It is not in either the company’s nor FDA’s interest to track thousands of KPIs, but the program will clearly learn and evolve over time. Assuming the program gets off the ground, FDA will be able to measure outcomes and evaluate which KPIs predict excellent software/companies.

3. PreCert Strives to Measure What Companies Do, Not How They Do It; Win for Smaller Companies?

The principle of “what, not how” was repeated a few times, meaning FDA cares what the PreCert participant is doing, not its methods. WellDoc’s Dr. Anand Iyer summed it up nicely: “Not every company employs the same best practices. I think the PreCert program should go after the ‘what.’ We don’t care how you do it. In fact, if there’s a more creative way to do it, we want to know…but make no mistake about it, you better hit these metrics because those are non-negotiable.” As far as we could tell, the Agency was very much on board with the “what, not how.” It was positioned as a way to level the playing field for smaller companies – they don’t need to have extensive protocols in place for communication with leadership if the CEO sits right across the desk from the developers at 10-person company. One result of this necessary principle is that objective evaluation may be difficult – each company will have different KPIs and different versions of those KPIs, so developing one standard metric and tracking each company’s performance relative to the others would be nearly impossible.

4. Can the KPIs of “Company Excellence” Be Bundled into a Single Dashboard or Scorecard? How Should This Be Reported to the FDA and When? How Often Should PreCert Status Be Renewed?

FDA is thinking of a more “continuous model for metrics” and showing excellence, meaning PreCert is not envisioned as a yearly audit. Indeed, FDA’s Marissa Cruz expressed sincere discomfort at the idea of annual reporting, which is clearly not FDA’s intention with this program. The final breakout session proposed a variety of scorecards and dashboards that could bundle metrics into a single graph or score that FDA could peek into. There was no consensus on which scorecard/dashboard would be most useful, and some expressed concern at bundling a variety of metrics measured on different scales – e.g., binary yes/no (do you do this process?) vs. % metrics (% of complaints resolved) vs. human factors scores. Still, we love the idea of a continuous dashboard to check on excellence and company health, ideally one that is fairly automated for companies AND for the FDA. Attendees in Adam’s breakout concluded that a mix of people + automation tools will be needed to collect KPIs, implying some will be manual.

5. PreCert Branding, Naming, and Communication to Users Were Major Themes on Both Days; How Do We Make The Program Seem Inspirational, Exciting, Aspirational?

Our own Adam Brown passionately framed the issue on day #1’s stakeholder panel: “Most people don’t know the difference between clearance and approval. Even mainstream media mixes them up – there’s a huge educational gap about the regulatory field. Now you’re talking about adding “pre-certification,” which could be a game-changer in the digital health field. But say I go to an app store and see “FDA Pre-certified.” To me, that doesn’t make a big difference. My biggest message for FDA: When you go out with this publicly, think about naming it differently. Instead of ‘pre-certified,’ it needs to be aspirational, inspirational, and exciting – something that companies will want to apply to and something users will want to have. I don’t think FDA “pre-certified” does that. We need to put a lot of thought into naming and branding, because right now it sounds like clearance and approval, and it’ll just appear in fine print at the bottom. I think it could be much, much more.” We suggested something more along the lines of “FDA Gold Certified for Software” or “FDA Certified for Software Excellence.” We believe the branding could really drive success of the program and the number of people who want to apply – if it connotes an impressive badge, it could help startups raise money, drive adoption, and perpetuate future great products. Of course, FDA will need to balance the nature of the title with the sort of confidence that it denotes. Communication will be absolutely critical, whatever the branding becomes – FDA will have to effectively communicate to multiple stakeholder groups exactly what PreCert means, which will be no easy task based on the comments we’ve heard over the past two days.

  • Many suggested the use of a tiered system, e.g. FDA PreCert Gold, FDA PreCert Silver, and FDA PreCert Bronze to indicate varying degrees of excellence. Others suggested a “Pending FDA PreCert” designation (like pending CDC Recognition for DPP), possibly for those companies that have many of the KPIs but no viable products on the market yet. We like both ideas, though it will require even more deliberation about thresholds for each tier, not to mention more communication and education.

6. Will PreCert Create More Burden for Companies, Who Already Have Plentiful Compliance (e.g., ISO, Other Regulatory Bodies, Internal) to Deal With?

The de facto answer to this question has to be no, otherwise the program will need to be altered until it attracts more applicants. It does, however, seem like it would be some work upfront to establish KPIs and then report them to FDA on somewhat of a regular/continuous  basis. Many industry representatives said that the process would be much better for an early stage company and that they would’ve chosen PreCert had it been around when they submitted their 510(k). Others pointed out that, even if PreCert is 1.2x the work of a 510(k), it is worth the trouble for companies looking to submit more than one product, as the extra work would be an investment upfront spread across all submissions, assuming the streamlined review is a meaningful improvement. The answer to all of these questions is currently unknown, and it is up to FDA and other stakeholders to draw a line in the sand on what is required to report and when. We would love to hear from companies in the pilot how much more work a 510(k) submission is on top of the work to comply with, for example, ISO standards and those of other regulatory bodies.

  • What will the transition from no PreCert to PreCert look like? Will companies have to do both traditional compliance and PreCert compliance? How much time will this save in the initial version vs. improved versions in subsequent years? How quickly will FDA open the floodgates upon rolling out its minimum viable PreCert program in 2019? How much faster will a PreCert review go for a 510(k) and a PMA? Will there be a rapid ramp of applicants to the PreCert program, or will there be a classic adoption curve? It is notable that for now, even some of the companies in the pilot have traditional class II filings with the Agency.

7. Will FDA See Reduced Burden, or Will PreCert Open The Floodgates to Far More Submissions from a Deluge of Companies?

The Agency will, like industry, have to do a lot of legwork up front to optimize the PreCert program and establish mutually agreed upon KPIs with applicants. And like participants have to continually report these KPIs, FDA will have to evaluate the metrics and the program. How much work will this be? Will there be an automated alert that flags companies lagging on their metrics for regulators? Will someone audit KPIs on a semi-frequent basis? How often? How much work will it be to ensure that KPIs are still strongly correlating with the safety and efficacy of products? How much less work will a streamlined review be for the FDA?

  • FDA might have to wrestle with a solution to the “carpool lane” problem, an analogy Mr. Look shared with us. When a carpool lane is first built, it typically has a relatively low barrier to entry (e.g., the driver and one other person in the car). Eventually, people figure this out and begin to carpool more, so the lane fills up and defeats the purpose of installing the lane in the first place. So regulators up the ante by saying the driver must have a passenger and be driving an eco-friendly hybrid car. Eventually, this iteration, too, fills up. So regulators raise the bar yet again. Where is the right place to set the bar so that some but not too many drivers qualify for the carpool lane? How much and how frequently should the bar be raised? FDA’s issue is even more complex: If it sets the bar too low, it will likely be faced with a deluge of PreCert applications and have myriad firms’ KPIs to stay on top of. If it sets the bar too high, then few companies will apply and qualify, defeating the point of creating the program in the first place. FDA has said that it is not lowering the bar for clearance/approval, but pre-certification is largely unexplored territory, and the bar for parameters that aren’t safety and efficacy is not yet defined – where will the bar be set for excellence to minimize burden for FDA while encouraging the right amount of applications? Some market research might help the Agency make this impactful decision.
  • Attendees raised the prospect that some firms will likely apply for pre-certification for the sole purpose of investor or market visibility. Pre-certification would indeed look great as it de-risks an investment and connotes some degree of trust for an ambitious medical software startup. But the opposite is also possible – software companies who don’t intend to create regulated medical grade software might be incentivized to obtain precertification status as a stamp of credibility. It’s hard to know how frequently this would happen, but it could contribute to unnecessary work for regulators – one told us that companies have in the past asked for 510(k) clearance even though their apps weren’t class II because they wanted the credibility that came along with an FDA review. The problem is something to watch out for with PreCert, as the company could make the claim on every product – not just a single one – that FDA rigorously evaluated and pre-certified the firm. One solution would be to require a medical grade app submission within, say, five years of pre-certification. Another would be to implement a “pre-precertification” filter – a less rigorous evaluation (meet a small number of must-have KPIs) or complete a multiple choice self-evaluation (which could be gamed, but punitive action could be taken if it was discovered that a company misled).
  • Will there be public fallout once a severe adverse event occurs from software developed by a pre-certified company? Imagine, for instance, that an insulin titration app from a pre-certified company causes an accidental severe hypoglycemia or even death. How would the public and Congress interpret this balance? This is a perennial problem with innovation and risk: In this case, FDA’s PreCert program could very well boost the amount of beneficial software on the market, thereby facilitating adoption of tools that make living with and treating conditions much safer and less risky. That insulin dose titration app may be saving thousands of lives/hospitalizations – something that’s a hard to measure – but it will be easy to blame PreCert if unforeseen events happen. Just as dosing insulin without assisted titration is highly dangerous, so is the opportunity cost of not moving to a PreCert program with more trustworthy tools available to end users. We believe FDA must communicate this principle very clearly to the public when it introduces PreCert.

8. What’s the Role of Clinical Evaluation and Demonstrated Outcomes in the PreCertification Process?

BCBS’s Dr. Naomi Aronson wondered “how PreCert fits into clinical evaluation…How will we be looking at the clinical results of digital health devices? How will we know what their clinical impact is? What evidence will be generated, and how will it be accessible.” In a room filled with digital health startups, it was a revealing comment from a major payer. Adam, sharing the stage with Dr. Aronson during the panel, pushed back, strongly cautioning FDA against requiring clinical evidence up front. “A lot of companies that will really drive innovation are small, and they don’t have the funds to do clinical trials before going to market. Trials are often slow, expensive, and they can’t keep up with pace of innovation…Getting products to market quickly [from “excellent companies”] allows them to get outcomes, then get clinical validation, then go to payers. That’s a great tradeoff…The gap between what we can do with technology and what is on the market is massive – it’s embarrassing and we can do better. We should think hard about what’s driving that gap and how to close it. My bias is to move fast even if we don’t have clinical evaluation for payers to make decisions.” We’d add that there is an important market reality underlying digital health: products will not be successful, and they will not get paid for, unless they demonstrate outcomes. The bigger question is when those outcomes need to be demonstrated in the regulatory review process (we believe later is okay), and when companies choose to gather such outcomes. Post-market data collection, ideally through NEST (National Evaluation System for Health Technology), is absolutely needed and will be crucial for FDA to determine whether to continue granting pre-certified status to program participants, but requiring clinical outcomes before market would arguably defeat key tenants of PreCert.

9. PreCert Is Its Own Startup, with Input, Learning, Iteration, and a Minimum Viable Program

In all of his communication, FDA’s Bakul Patel is thinking of PreCert as a product – something we’re co-creating with the end users (companies), getting feedback on, and iterating. This goes to Blue Cross Blue Shield’s (BCBS) Dr. Naomi Aronson’s ask that FDA continue to evaluate the program itself, in addition to companies within it. “What are its successes? What are its weaknesses? Is it getting better products out faster?” We love this!

  • Many people throughout the course of the meeting asked why PreCert couldn’t be implemented in combination hardware-software devices, or for drugs over at CDER, etc. The response was always, in essence, “we don’t want to bite off more than we can chew.” Like a startup, FDA is focused narrowly on the application here – software as a medical device. This program is clearly a learning experience for regulators, and if successful, it will hopefully inform application in other areas of the FDA. Of note, a similar program has already been implemented in direct-to-consumer genetic testing, where 23andMe obtained the first clearance and does not need to come back to the FDA for future reviews. It’s easier to see the need for PreCert in the digital setting where the innovation cycle is on the scale of days or weeks, but we do wonder what are the possible applications in pharmaceuticals – imagine drug manufacturers pre-certified to have streamlined reviews of certain molecules or combinations. To that end, today we spoke to a Deloitte consultant who works with CDER, so other branches are hopefully watching this program closely.

10. How Transparent Do PreCert Participants Have to Be with Their Data?

Throughout the meeting, many argued for radical transparency in the PreCert process, metrics, and how the approved/cleared software as a medical device actually works. Dr. Aronson wondered how an organization could meet the transparency criteria – the algorithms themselves (exactly how it works), a registry, outcomes, public reporting of metrics, or simply “some mechanism for bona fide researchers to validate that data.” Tidepool’s Mr. Look, representing the small non-profit voice, strongly shared this view. He advocated multiple times for the open sharing of data and interoperable devices to be a heavily-weighted criterion for inclusion in the PreCert program. Even a speaker from large player Philips said that “In cybersecurity, transparency is critical.” Meanwhile, one speaker from Dexcom argued that full transparency could discourage participation – an anomaly in some metric (e.g., variability from quarter to quarter, weird business trend) could lead to punitive action from FDA. The Agency could address this concern with clear communication around how any divulged information on metrics and products would be used or reported publicly. We consider transparency with regards to “secondary metrics” (those that FDA should want to see tracked, but not care to see the data) to be a nice-to-have that could only help a company’s case, but transparency around “primary metrics” is a must. The “black box” problem of algorithms is another important area for transparency – if a software algorithm’s output is not easily explainable (How did it go from X to Y?), it could have an uphill battle with healthcare providers, among others.

Brian Levine’s Open Public Comment

I spend a lot of my time thinking about software for diabetes and how it has and could potentially augment the use of drugs, technology, and how it could act as a standalone to facilitate and lower barriers to improved life with diabetes, diet, exercise, stress, and beyond. 

Diabetes is a poster child for digital health. Look no further than the fact that of the nine companies CDRH chose for the PreCert pilot, seven are fully or partially committed to diabetes.

Every company in diabetes, from open source non-profit to therapy and technology giant, is thinking about digital health because it will drive the most improvements in care and, as we move more toward paying for outcomes, it will be crucial for selling products.

Diabetes is a unique condition that for many patients requires massive attention; for multiple millions of patients in the US alone, diabetes requires the use of arguably the most dangerous drug in the world – insulin. According to estimates from experts in the field, people with diabetes think about their condition up to 600 times per day, yet they only spend 0.007% of their annual time with a provider. That means that out of ~219,000 times a patient thinks about diabetes, a provider is present for just ~1,500. Filling in those gaps with remote care, automated coaching, and decision support, particularly for dosing insulin, will be critical. This will be increasingly important as more patients with diabetes live longer and move to taking insulin. And while efforts in those areas have only begun to pick up steam in the past few years, they will accelerate massively moving forward. Having a system that encourages more innovation by making the path faster, while preserving safety and surveillance, will do huge good for manufacturers and patients alike.

The thing with diabetes apps, to quote University of Michigan’s highly-respected Dr. Joyce Lee, is that it’s still sort of a “wild, wild west” out there. At just about every scientific conference we attend, Dr. Kit Huckvale’s 2015 paper about mealtime insulin calculators is discussed. In that paper, of 46 apps analyzed, over half said “not to be used for medical purposes,” only 21 made their formulas available, only 9 had insulin on board compensation, only 10 adjusted for exercise level … the list goes on, culminating in the conclusion that 31 had a risk of inappropriate dosing (of one of the world’s most dangerous drugs!).

There is noise out there, and finding the signal is not always easy for patients or providers. For instance, at a European diabetes conference this past September, a panel of endocrinologists expressed heavy doubt on the safety and efficacy of diabetes apps. At one point, one wondered, “who is guiding this process of scrutinizing [mHealth apps]? Do they get approval or do they just appear?” None of the panel members had any idea.

We would strongly encourage establishing a “Pre-Certification stamp” to help patients and providers determine which apps are more likely to be safe and effective. If that does happen, should the designation be limited to a threshold (“this firm meets FDA’s definition of excellence”), or should there be different tiers (“this firm received a B+/silver from FDA’s excellence appraisal process”) and then provide a performance breakdown within each of the excellence principles? This heightened degree of transparency would push manufactures to improve on each metric, and also help patients decide which apps they’re most comfortable with.

The other main recommendation I’m excited to share from our team and myself is to be flexible in defining outcomes – whether we’re talking about software excellence or a product’s efficacy. For diabetes, A1c may still be the standard, but if people are spending more time in range, less time in hypoglycemia, or even thinking about their diabetes 300 times per day as opposed to 600, that would be a hugely successful product – even in the absence of an A1c drop. CDRH has been at the forefront of the push to explore innovative ways to go beyond A1c, and we’d love to see that focus carry over to the Pre-Cert program.

On a similar note, imagine if pre-certified companies could opt to make de-identified, real-world performance data widely available to FDA, patients, and developers through NEST? A cache of such data could enable a learning ecosystem of product improvement and even curation algorithms. Which app is right for which patient?

I want to end by saying thank you to Dr. Shuren, Mr. Patel, and the whole digital health team for your dedication to innovation, transparency, and improving the lives of patients as quickly and safely as possible. 

 

-- by Adam Brown, Brian Levine, and Kelly Close