Glitch with Dexcom Follow Resolved, But a Learning for Diabetes Tech on Communicating Issues to Users? – December 13, 2019

Heartfelt acknowledgement and apology from CEO Kevin Sayer released this morning; Takeaways from our interview with Mr. Sayer on remote monitoring, high user expectations, communication, in how many areas in which companies must excel, and concerns about de-skilling and education

On Saturday, November 30^th, Dexcom began posting Facebook updates of an unexpected issue with Dexcom Follow that prevented caregivers in the US from receiving CGM alerts. As reported in our Closer Look email, CNBC (twice), The New York Times, and WSJ, a number of followers woke up shocked to learn they hadn’t been receiving overnight CGM alerts.

We had a chance to speak with CEO Kevin Sayer this afternoon, who clarified what happened and how Dexcom is moving forward. As noted in his heartfelt apology today – watch the two-minute video on YouTube here (nearly 1,000 people have watched it as of a few hours after it was distributed) – Dexcom moved its Share/Follow platform to a new cloud service provider earlier this year. Ironically, this was done to modernize its data center and allow greater scalability. During the move, however, Dexcom introduced new components that weren’t configured optimally. Those components failed over Thanksgiving due to a server overload; when they did, Follow stopped working and Dexcom was caught off-guard.

The popular media attention as well as much of the patient attention, from this news, from the start, focused on Dexcom’s communication in the period afterward, something Mr. Sayer directly addressed today: “…the hours that passed between the Follow feature going down and our first communication to you were unacceptable.” Dexcom reportedly had a team of ~50 people working on this issue over Thanksgiving weekend, but it took several days to resolve.

See below for some of our big takeaways from speaking with Mr. Sayer today and following this news over the past two weeks – lessons all diabetes technology companies will need to consider in the years ahead, particularly as we rely more on smartphones to drive life-critical medical devices.

Moving forward, Dexcom is accelerating a plan to install an in-app messaging system that would alert users and Followers immediately if the Share system is not working as expected. It is also working on several additional communication redundancies – e.g., a dedicated page on the Dexcom website that will report system functionality 24/7 and serve as a central hub for real-time updates. We see these as extremely valuable steps and salute this leadership team’s humility and focus and commitment on learning amidst hypergrowth.

1. Communication Takeaways

• A big takeaway from this news relates to communication – when and where should companies inform users of product issues? Dexcom announced on Facebook that there was an outage around Saturday at 11 AM EST, approximately eight hours after users were first impacted. The company’s Twitter account is inactive, there was no press release, and limited website updates. Though some suggested emails and/or texts could be sent to users, that would not have been possible in this case due to gaps in Dexcom’s current setup. Dexcom did push an in-app notification to the G5 and G6 apps (see below), which did alert the issue to the CGM user (primary app); however, there was no ability to push the same notification to the Follower app. This communication snafu is mainly why the issue generated such significant social media attention – Dexcom’s Facebook thread describing the problem has generated over 2,300 reactions, 6,200 comments, and 867 shares! This is a lot of reactions, though not compared to hundreds of thousands of CGM users globally. It can be read in full below.

• Drawing the line on notifying customers vs. raising alarm about glitches is a challenge. As Mr. Sayer shared in our interview today, “How soon is too soon? How late is too late?” Mr. Sayer raised a very relevant example, the Urgent Low Soon feature on Dexcom’s G6 (similar to the Urgent High Soon feature) – Dexcom did extensive user research, ultimately deciding on a 20-minute heads-up before a predicted urgent low. Dexcom plans to do additional research on this for Dexcom Follow, trying to balance enough time to react vs. alerts that are so frequent as to be meaningless. “Probably not at five minutes, but it has got to be sooner than half an hour. It has to be prompt.” Of course, this will remain a complicated issue, as he emphasized: when does a problem warrant a full-scale alarm? It’s hard to know immediately just how bad a problem is, and there is also a big risk of raising false alarm – and of course, patient impression about this will vary given the heterogeneity of the population using CGM.

• Dexcom also would have benefitted from a multi-channel approach to alert users of a critical issues – e.g., in-app messaging on all apps, email, all social media channels, website, and engaging third-party PR resources. Updates were limited to Facebook with the outage over Thanksgiving, which all users do not check or follow Dexcom. Dexcom intentionally does not maintain a Twitter presence, though it is re-examining this decision – Twitter can foster an overwhelming volume of complaints, and if they go back to using Twitter, they will do so after further research rather than making a hastier decision. It’s possible that with more extensive PR work, the alarmingly titled stories (CNBC, NYT, WSJ), could have been tempered though overall the nature of the media pieces, were themselves positive in many respects. That said, we didn’t think the stories conveyed particularly much. The placement certainly sheds light on how critical the technology has become, particularly the Share feature, and indeed, for those that don’t know about it, Share is now a core function.  

• More empathetic updates? Some users felt Dexcom’s Facebook updates were too vague, technical, and even some felt they were defensive – see the thread below. The lack of response to Facebook comments fostered a lot of diabetes online community rage. Others felt the updates were factual and that time was being spent less on drafting communications and more on addressing the problems. The truth is probably somewhere in between, but many felt that communications is an area where there hasn’t been enough investment; after our call with Mr. Sayer today, there will clearly be more investment in the future. While Dexcom's core product, primary CGM readings,  continued functioning without interruption, and Share is adjunct, Dexcom didn't provide more specifics. For example, if there really were 50 people hard at work on this, could a photo have been shared showing that – just to emphasize how seriously Dexcom was taking the problem? Another option would have been a post that spoke to users’ frustration with more heart. Others felt that the part about asking Google and Microsoft for help sounded odd – most people do not understand cloud service architecture nuances. To some, this might have implied Dexcom was frantically calling tech companies because the situation was out of control.

2. User Expectations, De-Skilling, Back-Up Plan Takeaways

• A second takeaway is on user expectations, de-skilling (what happens when patients learn less since more of their care is automated), and back-up plans. In 2019, there is largely an expectation that technology will always work, which is of course unrealistic – especially with fast-moving smartphones and cloud infrastructure that outpaces the speed of regulated medical devices. But for those who have come to rely on remote monitoring and CGM (and, increasingly, automated insulin delivery) as can’t-live-without-it devices, inevitable breakdowns can drive massive anxiety and confusion: What do I do without ____? In that sense, this news is also a powerful reminder of how far the field has come – Dexcom Share is only five years old! At the time, using the smartphone as the sole receiver device still seemed like a longshot; now it is available on every CGM. We are also reminded of Dr. Bruce Buckingham’s story from FFL 2019: “One recent night, the on-call fellow received a call at ~1:00 am from a very stressed parent: “Our Dexcom CGM is not working right now; what should we do?!” The fellow replied as expected: “Take a fingerstick.” The parent’s response? “A fingerstick? That’s barbaric!!”

• In our call today, Mr. Sayer noted the need for more education on back-up plans, which multiple supportive patients have also raised in the past several weeks, post-Thanksgiving: “In the event your tech goes out – and it may – here’s what you should be thinking of…” He suggested a joint effort among non-profits – and certainly a range of nonprofits should care about this including all them of any size (ADA, JDRF, Beyond Type 1, DiabetesSisters, diaTribe, TCOYD, and multiple others like DiabetesMine, not a nonprofit, but an honorary one!) This is a very important problem to address for patient safety. Until this education exists, however, it puts companies in a difficult spot – dealing with frustrated users who are uninformed about what to do when something breaks. In a sense, this is not only about education but about teaching resilience in the face of unpredictability – a core skill for managing diabetes.

• De-skilling (i.e., patients learning less at diagnosis because they need to “do” less if they have access to advanced technology) will only become a bigger issue with automated insulin delivery – when someone is put on a system at diagnosis, will they know what to do when the CGM is offline, or supply reorders are delayed, or their pump battery dies? New tools create new problems, and the field clearly needs more thinking about unintended consequences.

3. Excellence in Diabetes Technology Takeaways

• A third takeaway is on being a diabetes technology company in 2019: the bar for excellence now spans across medical hardware; exploding consumer hardware; mobile apps on both Android and iOS; smartwatches; web browsers; cloud infrastructure; social media, email, and text; pharmacy and DME delivery; online ordering; customer support; manufacturing; and far, far beyond! For Dexcom and Abbott – growing at 49% and 63% YOY on $1+ billion in annual sales – maintaining excellence in all these areas at this level of scale is bound to bring challenges. As Mr. Sayer noted in our call today, it can be easy for startups and new-to-CGM companies to underestimate just how hard this is.

4. Interoperability Takeaways

• “We are very excited about interoperability, and we’ve been working with a number of companies for a long time. But it’s not like Bluetooth headphones – this is hard.” This was an excellent point from Mr. Sayer today – as the focus shifts from systems to interoperable components, there will be more points of failure. Companies will have to work together, interoperability will not simply be plug and play – it takes time and investment to make sure the components work together.

• This news is also a reminder that devices that depend on cloud services have another failure point. While this isn’t a concern for on-device algorithms, if an advanced AID algorithm uses cloud computing, it may add an additional risk.

Close Concerns’ Questions

Q: How should companies make the call on when to offer a full-scale alarm? What is the right balance between this situation (not enough alerts) and overly conservative communication (raising false alarm)? 

Q: What implications does this news have for smartphone-based automated insulin delivery, which will use an app as the primary algorithm? For AID apps, technical issues will arguably be a much bigger concern than for CGM itself. How can systems “fail” safely? Will this encourage more to rely on AID systems that work independently of the smartphone?

Q: How will the diabetes community prevent “de-skilling” and teach backup plans for when technology inevitably fails? Who will take care of this education? What responsibilities should patients always take on? What training can there be?

Q: How will companies address technical glitches, especially with convoluted cloud and app services? When multiple organizations are involved in one product, who should a user call? What are the implications for interoperability? Which player is responsible for fixing it? Who provides support and communications, and will that support be offered 24/7 or limited to a company’s corporate hours in one time zone? We imagine organizations are working with each other, but it’s impossible to avoid what is unanticipated. Some may believe they are working in “lockstep”; others may see this is easier said than done.

Q: What is reasonable to expect from companies on social media platforms? How will companies navigate rapidly moving platforms like Twitter? How will Dexcom move forward in its decision-making regarding adding a Twitter presence? Does every company need to do the communications work separately?

Dexcom “Apology” Video and Transcript

Today, Dexcom Executive Chairman, President, and CEO Kevin Sayer released a message to Dexcom customers addressing the outage. The video can be viewed here and was posted on Facebook, Instagram, and YouTube . A transcript of the video is enclosed below. The Facebook thread currently has over 2,100 reactions, 30 comments, and 24 shares.

“Hello, my name is Kevin Sayer. I am the chairman, president and CEO of Dexcom.

Thanksgiving weekend, we experienced an unprecedented outage of the Dexcom Follow function, which enables our users to share their CGM data with caregivers and loved ones.

Let me start with a personal apology on behalf of myself and our entire team. Everyone at Dexcom is disappointed about what happened. It is not what you expect from us and certainly not what we expect from ourselves. We can and we will do better.

Now that we have restored service fully, I’m able to shed some light on what took place.

Earlier this year, we moved our Share/Follow platform to a new cloud service provider in an effort to modernize our data center and allow greater scalability of our systems. During that move, we introduced new components to our platform that weren’t configured for optimal performance. Those components failed, and when they did, there was disruption to core processes within the platform. When this occurred, our Follow app was impacted.

We are reviewing our entire system architecture and operational support to determine and make the necessary changes. Dexcom is working around the clock to ensure this doesn’t happen again.

Furthermore, the hours that passed between the Follow feature going down and our first communication to you were unacceptable. The engineers are accelerating a plan to install an in-app messaging system that would alert users and followers immediately if the system is not working as expected.

We are also working on several additional communication redundancies. For example, the team is developing a dedicated page on the Dexcom website that will report system functionality 24/7 and serve as a central hub for real-time updates.

We created our CGM system and Follow app so you could depend on it. We know you do and that has never been clearer to us.

Although we now understand the reason why this outage occurred, there are no excuses. As I said before, we can do better. And, as the leader of this company, you have my personal promise that we will.

Thank you for communicating with us so we understand your needs. We intend to regain your trust and keep it.”

Dexcom Facebook Thread – Play-by-Play Updates to Dexcom customers on Facebook

12/4/2019, 5:13PM PST: The Dexcom Follow services have been restored to full service levels. We are monitoring the system and closely reviewing customer feedback. 

If you continue to experience any issues with your Dexcom Follow app, please contact Technical Support at 1-844-607-8398 or https://www.dexcom.com/priority-support

12/3/2019, 6:50PM PST: At this time, a small number of Android Follow users are encountering a persistent flickering upon launch of the Follow app. The team has determined root cause and an updated version of the app to remove the issue will be made available in the Google Play Store. We’ll continue to post updates here. 

Users who experience this issue are advised to contact Tech Support at https://www.dexcom.com/priority-support or 1-844-607-8398.

12/3/2019, 9:30AM PST: At this time, the Dexcom Follow services continue to perform at near normal performance.

While the outage primarily impacted users of the Follow app, some G6 and G5 users may still experience issues logging into their app. In the event of a login error, users are advised to retry the login process.

If you are still experiencing issues, please submit a request through our Self-Service Patient Support Portal located at: https://www.dexcom.com/priority-support.

As our teams work to increase system performance, we will continue to post regular updates here.

12/2/2019, 10:50PM PST: We recently experienced a brief service disruption, which the team addressed quickly and the Dexcom Follow services have been restored to near normal performance. 

Our team is continuing to monitor the system closely and we will post any new updates here.

12/2/2019, 4:52PM PST: At this time, the Dexcom Follow services have been restored to near normal performance. 

While the outage primarily impacted users of the Follow app, some G6 and G5 users may still experience issues logging into their app. In the event of a login error, users are advised to retry the login process. 

If you are still experiencing issues, please submit a request through our Self-Service Patient Support Portal located at: https://www.dexcom.com/priority-support.

As our teams work to increase system performance, we will continue to post regular updates here.

12/2/2019, 10:25AM PST: At this time, the Dexcom Follow services have not yet been fully restored but we are still seeing improvement in system performance. 

While the outage is primarily impacting users of the follow app, there are some instances where users are experiencing issues logging into the Dexcom G5 and G6 apps. 

If users are experiencing login issues, they should swipe close the app, restart it and chose “login later” when prompted. Additionally, in response to these issues please do not delete and reinstall the Follow or CGM app as they may not function properly. Keep the apps open and installed.

As our teams work to solve this issue, we will continue to post regular updates here.

12/2/2019, 5:25AM PST: At this time, the Dexcom Follow services have not yet been fully restored but we have seen significant improvement in system performance. Our teams continue to work around the clock on a resolution.

We are still investigating official root cause. However, we have determined that a server overload occurred due to an unexpected system issue that generated a massive backlog, which our system was unable to sufficiently handle.

While the outage is primarily impacting users of the follow app, there are some limited instances where users are experiencing issues logging into the Dexcom G5 and G6 apps. If users are experiencing login issues, they should swipe close the app, restart it and chose “login later” when prompted.

Additionally, in response to these issues please do not delete and reinstall the CGM app as it may not function properly. Keep the app open and installed.

As our teams work to solve this issue, we will continue to post regular updates here.
12/1/2019, 10:30PM PST: At this time, the Dexcom Follow services have not yet been fully restored. Our teams continue to work around the clock on a resolution. Our latest understanding is that we are experiencing a server overload and we’re evaluating a number of potential fixes including increasing server capacity. 

While the outage is primarily impacting users of the follow app, there are some limited instances where users are experiencing issues logging into the Dexcom G5 and G6 apps. If users are experiencing login issues, they should swipe close the app, restart it and chose “login later” when prompted. 

Additionally, in response to these issues please do not delete and reinstall the CGM app as it may not function properly. Keep the app open and installed. 

As our teams work to solve this issue, we will continue to post regular updates here.

12/1/2019, 2:10PM PST: As of right now, the Dexcom Follow services have not yet been fully restored. Additionally, you may be receiving a notice “server unavailable” in your G5 or G6 CGM app. In response to this notice, please do not delete and reinstall your CGM app as it may not function properly. Our teams continue to work on a resolution and we will continue to post updates here.

12/1/2019, 1:00PM PST: As of right now, the Dexcom Follow services have not yet been fully restored. Our teams continue to work on a resolution and we will continue to post updates here.

12/1/2019, 8:43AM: As of this morning, the Dexcom Follow services have not yet been fully restored. There continues to be a 24/7, large cross functional effort with both Google and Microsoft to diagnose and address the issue. Dexcom did not release any updates or changes over the holiday to create this issue, which has created a more complex investigation.

We read every comment and understand your disappointment in both the system challenges and communication shortfalls. First and foremost we are 100% focused on solving the issue at hand. Additionally, we are committed to creating a better customer communication experience moving forward. You deserve nothing less.

We will continue to post updates here throughout the day.

12/1/2019, 1:46AM: As of now, the Dexcom Follow services have not yet been fully restored. Our teams continue to work on a resolution. We will continue to post updates here.

11/30/2019, 8:00PM: The Dexcom Follow services have not yet been fully restored, our teams are actively working on a resolution, and we will continue to post updates here. 

11/30/19, 3:02PM: As of now, full Dexcom Follow services have not been restored. Customers should expect continued intermittent availability of Follow services only. Our teams continue to work vigorously on a resolution. 

11/30/19, 12:29PM: We continue to experience issues with Dexcom Follow that may cause users to not receive CGM data or alerts. The team has been working around the clock on a resolution and will continue to post updates here.

11/30/19, 8:03AM: We're aware of an issue with Dexcom Follow that may cause users to not receive CGM data or alerts. We are actively working on a resolution and will continue to post updates here.

In-App G6 Notification of Issue

--by Ani Gururaj, Adam Brown, and Kelly Close