Certification Retrospective Part 4 – Lessons Learned

Note: this blog post concludes my discussion of experiences in CCHIT. The opinions are mine, and do not necessarily reflect the views of CCHIT or Siemens.

In life in general, and in development of software, standards, and certification criteria, we learn and grow more by trial and error than we do when everything goes smoothly. Frederic Brooks’ classic software engineering book The Mythical Man-Month said “Plan to throw one away; you will anyhow.“ Hence we need prototyping and iterative refinement. The CDA Consolidation Project is fresh on my mind. I voted negative in its HL7 ballot, but that doesn’t mean it won’t succeed, only that it has issues to fix. It would have been a miracle to get that complex task right the first time.

So it is with certification. The CCHIT Interoperability Workgroup (IOWG) experienced growing pains in its early years. 2006 produced only a roadmap, 2007 produced the first (modest) interoperability criteria. 2008 had stronger criteria and roadmap, and 2009 was poised to make big strides toward semantic interoperability. What were my main lessons learned through these years?

What We Got Right (IMHO):

• The “methodical march” sequence of building blocks. Putting the horse before the cart sure helps! As described in Method to our Madness?, we proposed a logical sequence for interoperability:

1. Data Capture
2. Liquidity
3. Standardized Content
4. Data Consumption

Our views crystallized especially in 2008-2009. CCHIT’s criteria included many of the standards that were eventually endorsed by ONC. For example, Lab Results, CCD, e-prescribing, vocabularies (LOINC, RxNorm, SNOMED CT), and leveraging HITSP.

• The Roadmap (“Glide Path”) Concept. I love maps (online, GPS, paper) and want to know ahead of time where I’m going and the milestones along the way. CCHIT published roadmaps for the next two years beyond each certification year. Roadmaps aimed to provide adequate lead time and to encourage developers to plan and head in the right direction. While the HIT Standards Committee did a great job proposing a “Glide Path” to more robust standards in their proposals in late 2009 (see Jamie Ferguson’s August 20th 2009 Clinical Operations Workgroup presentation), that concept was mostly lost when the ONC regulations were published (see comments on John Halamka’s Feb. 10, 2010 blog). For interoperability, developers need to know the target standards, more than a high-level meaningful use matrix, to move forward.
• Staying a Course. Once we defined the roadmap, we might adjust timing and provide more detailed criteria in our next development cycle, but overall there was continuity and stability with a minimum of “surprises.”
• Transparency. CCHIT posted responses to every public comment on its website. Of course not everyone would agree, but we thought everyone deserved to know their comments were considered and our rationale for decisions.
• Teamwork. The IOWG built relationships and trust among multiple stakeholders within CCHIT, as mentioned in Part 1. That enabled us to work through differences and get things done, despite conflicting opinions. In contrast, it’s harder to capture the “team” element in groups where 50-100 people are nominally on a project but attendance is sporadic, yet I understand the need for openness and transparency. Ultimately, a small group forms a cohesive team within the larger group. For example, the Documentation and Testing workgroup within the Direct Project gradually developed a “team” spirit similar to what the IOWG had.

Regrets:

• Lack of Opportunity to Finish What We Started. Last week I used football, now here’s my first baseball analogy. CCHIT, as the “starting pitcher” for EHR certification was relieved by ONC in the 6^th inning – enough to get a quality start, but not enough to win the game. Our methodical march didn’t reach its goal, but I hope that ONC will be a good “closer” to save the win for interoperable EHRs, coordination of care, and the good of the patient.
• Lack of End-to-End Coverage. Interoperability involves senders, receivers/requesters, and perhaps intermediaries. To succeed, it must encompass all participants (e.g., EHRs, HIEs, PHRs, labs, pharmacies, public agencies), because so much exchange goes beyond EHRs. We didn’t achieve this in CCHIT, but not for lack of trying! We envisioned covering some end-to-end interoperability through certification programs for HIEs and PHRs in addition to EHRs. Excellent people were in the HIE and PHR workgroups, and the IOWG had good dialogues with both. The HIE program was launched with poor uptake followed by suspension in light of impending ONC program changes. The PHR program was not launched.
• All-or-Nothing Approach. CCHIT pondered a modular certification approach early on, especially for inpatient EHRs, but didn’t implement it until ONC mandated it in the Certification Final Rule. In hindsight, I wish we had done that sooner. I don’t think we needed an “all or nothing” approach. Truth in labeling would have disclosed each product’s gaps, to help people make informed decisions. Vendors with complete EHR functionality could still differentiate themselves as a “one-stop shop” but others (“little guys”) would not be excluded from certification just because they didn’t provide 100% of all capabilities.
• A year in limbo. From mid-2009 through mid-2010, the industry was paralyzed by uncertainty and lost momentum on interoperability. It was clear that CCHIT criteria wouldn’t be adopted as is by ONC, but no one knew what would replace them. There was no assurance of staying a course or a clear glide path, and people who had done their best to follow the CCHIT roadmap were concerned and confused. Interoperability features (such as standardized coding and discrete data consumption) that the IOWG expected to be in products in 2010 have been delayed, possibly to MU Stages 2 or 3.

Whether right or regret, there are lessons learned that can be applied going forward.

• Follow the methodical march – a logical sequence of capture, liquidity, standardization, and consumption.
• Define a roadmap (glide path) with ample lead time.
• Stay the course.
• Take an end-to-end perspective.
• Be transparent.
• Be flexible, not “all or nothing.”

That’s it! This has been a long blog series, and if you made it to the end, thanks!