DTS’ BGM Surveillance Program releases accuracy results: only six of 18 tested BGMs receive seal of approval, including devices from Bayer (Ascensia), Accu-Chek, Arkray, AgaMatrix, and Abbott – July 31, 2017

In a win for transparency, the Diabetes Technology Society’s BGM Surveillance Program recently released long-anticipated results – get them on the DTS website here. The three-site, n=1,035 (!) evaluation found that out of 18 BGMS representing ~90% of the commercially available systems used in the US (2013-2015), only six received the “seal of approval.”

Passing meters included Bayer’s (now Ascensia) Contour Next (100% compliant), Roche’s Accu-Chek Aviva Plus (98%), Arkray’s Walmart ReliOn Confirm/Micro (97%), Agamatrix’s CVS Advanced (97%), Abbott’s FreeStyle Lite (96%), and Roche’s Accu-Chek Smart View (95%) – all had >95% of trials within ±15% (≥100 mg/dl) or ±15 mg/dl (<100 mg/dl) of YSI (see Table 1 below). It was not a surprise to see the Contour Next, Accu-Chek, and FreeStyle Lite systems receive passing marks, as these are generally considered very accurate meters – particularly Ascensia, which had an impressive 97% within ±10%, a full ten percentage points ahead of Roche. We were also glad to see great accuracy for store-brand value meters from Arkray (Walmart ReliOn) and Agamatrix (CVS).

On the other hand, it was disturbing to see that two-thirds of these FDA-cleared meters did not pass the DTS evaluation, with some reporting dismal scores. Those that barely missed the cut included Arkray’s Walmart ReliOn Prime (92% compliant), LifeScan’s OneTouch Verio (92%) and OneTouch Ultra2 (90%), Abbott’s Walmart ReliOn Ultima (89%), Bayer’s Contour Classic (89%), and Prodigy’s Auto Code (90%). Those with even lower scores included Omnis Health’s Embrace (88%), Nipro’s True Results (88%) and True Track (81%), Biosense Medical’s SolusV2 (76%), Suncoast’s Adovcate Redi-Code+ (76%), and Philosys’ Gmate Smart (71%). It was notable to see both LifeScan meters missed the passing threshold, as well as one each from Arkray, Abbott, and Bayer (since all three also had passing meters). Most of the non-passing meters, however, have previously been criticized for inaccuracy, especially those from Prodigy, Suncoast, and Philosys. These results are extremely concerning in light of the latest market share data (April-June 2016) for Medicare BGM claims – 28% for Prodigy, 23% for J&J’s OneTouch Ultra, and 12% for OmnisHealth’s Embrace. In other words, ~68% (!) of the Medicare mail order BGM market in 2016 was for meters that did NOT pass the accuracy bar in DTS’ evaluation. This is very disappointing.

Overall, these initial surveillance results confirm suspicions raised at DTS’ May 2013 and September 2013 meetings – many FDA-cleared meters actually don’t meet the accuracy bar when purchased through retail sales channels and tested rigorously in the post-market setting. And of course, these are likely best-case evaluations, since they were done with trained professionals at three clinical sites: Rainier Clinical Research Center (n=353), Diablo Clinical Research (n=335), and AMCR Institute (n=347). In real-world use where handwashing and testing conditions are far more variable, the data would certainly look more concerning.

The triple-blinded, three-site study included over 1,000 participants (81% with type 1 or type 2 diabetes) and sourced meters from retail pharmacies and online retailers from different parts of the country. As we reported back in 2014, it was made possible through a grant from Abbott – quite interesting considering it is the BGM company that has most moved away from traditional fingerstick monitoring. The authors were a well-respected group: Drs. David Klonoff, Joan Parkes, Boris Kovatchev, David Kerr, Wendy Bevier, Ronald Brazg, Mark Christiansen, Timothy Bailey, Jim Nichols, and Michael Kohn. View the full data here; we’ve copied some of the key results below.

• A major question with this data is what the FDA can/might do about it. At DTM 2016 last fall, Dr. Courtney Lias said she was “extremely excited” for this Surveillance Program, as the data can help inform directed FDA inspections – “This is meant for us as a signal and another tool to prioritize where quality needs to be investigated,” she said. We wonder if these results will stimulate such inspections … As a reminder, the FDA relies on company-reported data to clear BGMs, but isn’t able to rigorously test systems in the post-market setting, and adverse event reports are historically unreliable and challenging to sift through. Perhaps this DTS program can fill an important gap, assuming it stays up to date with systems and continues to be funded. At DTM, study co-author Dr. Joan Parkes (of error grid fame) envisioned that manufacturers might approach DTS in the future and request studies, and she hoped that all devices would be tested “at least once every two years.” We’d expect to hear an update at DTM this fall.
• We’re glad to see the results are freely available on the DTS website and very nicely summarized for a lay audience (good use of colors to indicate passing/not passing). We’re not sure if the data will be published in a peer-reviewed journal, though JDST seems like an obvious contender. We imagine people with diabetes and healthcare providers would be very interested in this data, since it’s extraordinarily difficult to understand from strip labels how accurate systems are – the key accuracy data is typically buried within tables in folded-up strip labels and often very hard to find. These data also reinforce the need for significant patient and HCP education that not all meters are created equal, even if they are all cleared by the FDA.   
• The triple-blinded protocol (previously published by a steering committee in JDST in 2016) was quite rigorous, comprised of 1,035 participants aged 18 and older with type 1 diabetes (n=370), type 2 diabetes (n=470), pre-diabetes (n=4), and no diabetes (n=187). Each clinical site conducted three sub-studies of different sets of six BGMs so that all 18 BGMs were tested at all three clinical sites. Results across the sites were compiled into three over-arching studies for each BGM – in order for a BGM to receive the seal of approval, it had to pass at all three sites. A tube of capillary blood was also collected from each participant to assay on YSI at The William Sansum Diabetes Center for comparison.

Table 1: Overall accuracy results for 18 BGMs (adapted from DTS’ Website)

Table 2: Number and percent of values within specified error limits (adapted from DTS’ website)

The modified Bland-Altman comparison showed quite a bit of bias in certain meters, as well as striking inaccuracy in the worst performing devices. Of those meters that passed, we noted a fairly large -6% bias in Abbott’s FreeStyle Lite and a -5% bias in Roche’s Accu-Chek Smart View; conversely, Contour Next had just a -1.2% bias, and Agamatrix had only a -0.3% bias. Of those that didn’t pass, BioSense Medical’s SolusV2 had a striking -10% bias, followed closely by Suncoast’s Advocate Redi-Code+ (-9% bias) and Nipro’s True Result (-8.9%).

Bland-Altman Plots for Contour Next (highest accuracy) vs. Philosys Gmate Smart (lowest accuracy)

• To assess the accuracy of each BGMS at extremely low glucose values, an additional tube of capillary blood was collected from roughly 20 participants at each of the three sites. The samples were glycolyzed to achieve sufficiently low glucose levels and then tested only on BGMs utilizing glucose dehydrogenase chemistry, which is oxygen insensitive. Interestingly, this qualification excluded two of the six BGMs that received the seal of approval (Arkray Walmart ReliOn Confirm (Micro) and Agamatrix CVS Advanced were not tested). The remaining four systems that passed demonstrated 100% of trials within the protocol limits. 
• After publication, LifeScan shared the following with us: "At LifeScan, Inc. we stand by the accuracy and the efficacy of our OneTouch® products – used by more than 20 million people around the world and recommended by more endocrinologists, primary care physicians, and pharmacists than any other brand. These products are continually assessed in our OneTouch® post-market surveillance program, which includes clinical testing as well as complaint capture, to ensure that all products – including those in the DTS study – continue to meet FDA performance requirements and specifications. Both products meet the International Organization for Standardization (ISO) standard for accuracy, with the OneTouch Verio® BGMS designed to meet the requirements of ISO 15197:2013 and the OneTouch® Ultra®2 designed to meet ISO 15197:2003. In fact, the April issue of the Journal of Diabetes Science and Technology features a study detailing the OneTouch Verio® BGMS platform’s seven years of proven accuracy across more than 70,000 clinical data points.  As we are always evaluating ways to further ensure product quality, LifeScan’s clinical and R&D teams are working to evaluate and understand the DTS results. Patients who have questions are encouraged to get in touch with OneTouch® Customer Care at 1-800-227-8862 so we can work with them to provide answers."

Close Concerns Questions

Q: How has the FDA reacted to these data? What actions are within its authority and bandwidth? For example, will it move to inspect the low-performing meter companies? Could these meters be recalled? Could the FDA impose sanctions on these companies?

Q: Will these results be published in a peer-reviewed journal?

Q: Will these data affect patient, HCP, and payer choice of BGMs? In particular, how will payers view these data, especially Medicare (given that ~68% of the 2016 mail order market was for meters that did not pass)?

Q: Will this Surveillance program continue to re-test meters in the future? Who will provide funding and how much would it cost to repeat this every year or two?  

Q: Will a similar program be needed to assess CGM accuracy in the post-market setting?

-- by Maeve Serino, Adam Brown, and Kelly Close