DoD, VA and Coast Guard experience EHR outage nationwide.

DoD, VA and Coast Guard experience EHR outage nationwide

April 12, 2022
by John R. Fischer, Senior Reporter
Over 95,000 clinicians were unable to access or update patient medical data on April 6th due to a nationwide outage that took down the EHR systems used by the U.S. Department of Defense, Veterans Affairs and the Coast Guard for three hours.

Both systems are developed and supported by Cerner, with the VA’s known as Millennium and the DOD’s and Coast Guard’s as MHS Genesis. Three Oracle databases supporting the systems went offline after 5 p.m. ET and prevented clinicians at 66 DoD sites, 109 Coast Guard sites and three VA sites from accessing medical records, according to Nextgov.

The systems were fully restored shortly before 8 p.m. the same day. No evidence suggests any harm to patients was done. The VA is looking into the source. “We are in close contact with Cerner, and we are exercising all of our authority to make sure that we can position ourselves for success and assure that when these types of things happen, we get to the root cause. We conduct a root cause analysis, we identify how the problem arose, and we make sure that we don’t see the same thing happen in the future,” Deputy VA Secretary Donald Remy told Federal News Network.

Terry Adirim, executive director for the VA EHRM Integration office said a bug in the Oracle databases was responsible for the outage and not ongoing deployment efforts.

While the systems were down, clinicians were able to review patient data but could not update information, with backup “read-only” systems taking over. They were able to continue “most clinical operations,” according to Adirim, but had to record patient information by pen and paper and later update the EHR systems.

No specific disruptions to care or patient safety incidents were recorded during the outage. Cerner, Leidos, VA, DOD and Coast Guard technicians worked together to restore the system.

In early March, administrators at Mann-Grandstaff in Spokane took down their EHR system temporarily after an update led to potential data corruption. Staff were told not to admit new patients and “provide only those healthcare services you are comfortable providing, assuming all electronic sources of data are unreliable.”

The hospital was the site of the first EHR go-live. The cause of the issue was a defect in an interface between the systems that provide demographic information to Spokane. “That defect was fixed in short order. However, [it] went through some testing to be 100% certain that it was safe to turn the system back on,” said Adirim.

Spokespeople for Cerner and Leidos did not immediately respond to Nexgov’s request for comment and did not disclose if other government and non-government systems were affected.

The DOD EHR system has been in the works since 2015, with the Coast Guard joining in 2018. The VA also agreed in 2018 to upgrade its own EHR system as part of the combined Electronic Health Record Modernization Program. Initially estimated to cost $10 billion, the cost for the project was later upgraded to $16 billion.

The schedule for full deployment of the system has been hit with a number of setbacks. The COVID-19 pandemic delayed the project in early 2020, with work resuming that August. Another six-month delay in July 2021 was put in place after training failures, data migration problems, and concerns over patient safety were discovered at the site in Spokane.

Additionally, a report in May estimated that the VA may have been short by as much as $2.6 billion for the infrastructure upgrades that the system required and was followed by another in July that said an additional $2.5 billion was necessary for IT infrastructure needs. These concerns led to an independent cost estimate in October to determine how much the VA expected to spend on the project.

Another report in November by the VA Office of Inspector General showed that the EHR scheduling platform “reduced the system’s effectiveness and risked delays in patient care” and said that the VA was aware of these “significant” limitations prior to installing the system last year in Spokane, Washington and Columbus, Ohio.

Among the problems was the inability to change appointment types, with schedulers having to manually create a new one or ask the provider to submit a new order. They also could not mail appointment letter reminders automatically, a feature that was available on the old system. Additionally, the new system did not have oversight reports that were previously accessible to track and monitor patient wait times.

Remy says the agency is using the reports and its experiences with the go-live site in Spokane to improve the system, and has already made over 2,000 content changes in training, expanded the training course catalog and virtual training options.

The VA plans to launch the system on April 30 in Columbus, Ohio.