Another in the "Health IT Crashed, But Patient Care Was Not Compromised" Series - NHS Greater Glasgow and Clyde

We must have health IT to prevent stupid doctors and nurses from making mistakes.  Without it, patients are at the mercy of dreaded Paper Records, which by definition cannot ever be used to provide quality care.

But when the IT goes down, Patient Care Has Not Been Compromised.  This line should be trademarked, as it's seen so often.  (I even have an indexing tag for it, see this query link: http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised.)

Care is never, never compromised when the IT goes belly up en masse due to information technology malpractice.  Care is only compromised by paper, no matter how good the paper records and its human stewards are.

Here's the latest example that made it to the news, in Scotland:

1 October 2013
BBC News
Appointments postponed after major IT failure at NHSGGC (NHS Greater Glasgow and Clyde)

Hundreds of outpatient appointments and a number of operations had to be postponed after computer systems failed at Scotland's biggest health board.

NHS Greater Glasgow and Clyde said technicians were working through the night to fix a "major IT problem" which occurred on Tuesday morning.

It affected staff access to clinical and administrative systems.

Delaying hundreds of appointments and delaying surgeries at up to 10 major hospitals seems on its face to represent "compromised care."

The health board apologised to patients and said all appointments would be rescheduled.

In total, 288 outpatient appointments, four planned inpatient procedures, 23 day surgery cases and 40 chemotherapy sessions were postponed.

There was also some delay in calls to the switchboard being answered.

The problem may have affected up to 10 major hospitals across the health board area.

One wonders how these appointments and surgeries were triaged for delay.  Clearly the downed computer was of no help.

Here's that wonderful line:

But emergency operations were not compromised - neither were community services.

So when does computer failure actually compromise patient care?  I'd like to see some hospital executive with a spine for once admit that IT malpractice does disrupt patient care, create distractions, and thus create safety risk.  Considering the domain, however, I doubt I'll ever see that.

A spokeswoman said: "Our technical staff are working flat out to resolve this.

It should never have happened in the first place.

"The problem relates to our networks and the way staff can connect to some of our clinical and administrative systems.

Well, sick patients really appreciate that explanation.

It was not clear how long the disruption would last.

NHSGGC said if it did continue, people who were scheduled for treatment would be contacted directly.

Per a computer guru from my past:  "Either you're in control of your information systems, or they're in control of you."

In this instance, the latter clearly applies.

In healthcare, having your information systems in control of you is, sooner or later, going to be deadly.

-- SS

Setback for Sutter after $1B EHR crashes (in followup to post "RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals")

At my July 12, 2013 post "RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals" (http://hcrenewal.blogspot.com/2013/07/rns-say-sutters-new-electronic-system.html) I reproduced a California Nurses Association warning about rollout of an EHR at Sutter:

RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals

Introduction of a new electronic medical records system at Sutter corporation East Bay hospitals has produced multiple problems with safe care delivery that has put patients at risk, charged the California Nurses Association today.

Problems with technology are not unique to health care ...  [What is unique to healthcare IT is the complete lack of regulation - ed.]

In over 100 reports submitted by RNs at Alta Bates Summit Medical Center facilities in Berkeley and Oakland, nurses cited a variety of serious problems with the new system, known as Epic. The reports are in union forms RNs submit to management documenting assignments they believe to be unsafe.

Patient care concerns included computerized delays in timely administration of medications and contact with physicians, ability to properly monitor patients, and other delays in treatment.  Many noted that the excessive amount of time required to interact with the computer system, inputting and accessing data, sharply cuts down on time they can spend with patients with frequent complaints from patients about not seeing their RN.  [Note: patients are not given the opportunity for informed consent about the risks, nor opt-out of EHR use in their care - ed.]

In related posts I'd observed such concerns being ignored by hospital management.  See header of the aforementioned post.

Now we have this:  a major system crash.

Healthcare IT News
Setback for Sutter after $1B EHR crashes

'No access to medication orders, patient allergies and other information puts patients at serious risk'

SACRAMENTO, CA | August 28, 2013

The nearly $1 billion electronic health record system at Sutter Health in Northern California crashed early this week, leaving nurses and clinical staff unable to access any patient information for a full day.
he Epic EHR was down Aug. 23 due to a planned upgrade for eight hours during which nurses and other clinical users could read medication orders and patient histories but had to record new data on paper to then be re-entered into the system later, according to reports from the California Nurses Association, part of National Nurses United, the largest nurses union in the U.S.

Then, on the morning of Aug. 26 at approximately 8 a.m., the system crashed completely, at which time nurses, physicians and hospital staff had no access to patient information, including what medications patients were taking or required to take and all vital patient history data. 
 
The Epic EHR went dark at several Sutter Health locations, including Alta Bates Summit Medical Center, Eden Medical Center, Mills-Peninsula Hospital, Sutter Delta, Sutter Tracy, Sutter Modesto and numerous affiliated clinics.

"This incident is especially worrisome," said Bonnie Castilla, RN, CNA legislative director, in a news release. "No access to medication orders, patient allergies and other information puts patients at serious risk.  These systems should never be relied upon for protecting patients or assuring the delivery of the safest care."  [This is no exaggeration.  My mother was nearly killed by such a problem when caused by an EHR defect (link) that I averted at the last minute by happenstance of showing up at the right time, while she recuperating from another EHR-related disaster - ed.]   

Some RNs also reported that Sutter failed to provide adequate support during the blackout and didn't implement adequate backup planning.
 
"All information such as medication administration records and patient histories was outdated by two to three days," said Mike Hill, Alta Bates Summit RN, in a news release. "There were no orders that could be seen of any kind through the day so nurses called for what they needed."

Kevin Sweat, an emergency room RN at Eden Medical Center, a Sutter affiliate, also weighed in on the blackout, saying that in his 12 years as an RN he has never seen a system crash like this. 
  

Worse, clinicians must now serve their Cybernetic Master to perfection, or be whipped (apparently to improve morale):

... "We have been on Epic for 5 months now, and we can no longer have incorrect orders, missing information or incorrect or missing charges. Starting on September 1st, errors made in any of the above will result in progressive discipline," according to another hospital memo sent to staff.

In the setting of dire warnings by the nurses of EHR dangers several months back that were likely largely ignored, if any patient was harmed or killed as a result of this latest fiasco, the corporate leadership has literally begged to be sued for negligence, in my view.

However I'm sure a press release soon will claim that "patient care has not been compromised."

Of course this includes now and moving forward, even with informational gaps all over the place.

-- SS

Aug. 29, 2013 additional thought:

The punishment for not being a 'perfect' user of this EHR is the ultimate "blame the user" (blame the victim?) game, considering the pressures of patient care in hospitals in lean times - partly due to EHR expense! - and EHRs that have not been formally studied for usability and are poorly designed causing "use error" (that is, a poor user experience promotes even careful users to make errors).  Cf. definition of bad health IT:

Bad Health IT ("BHIT") is defined as IT that is ill-suited to purpose, hard to use, unreliable, loses data or provides incorrect data, causes cognitive overload, slows rather than facilitates users, lacks appropriate alerts, creates the need for hypervigilance (i.e., towards avoiding IT-related mishaps) that increases stress, is lacking in security, compromises patient privacy or otherwise demonstrates suboptimal design and/or implementation.

The study of usability is getting underway only now via NIST but will likely be done in an industry-friendly way due to health IT politics.

-- SS

Aug. 29, 2013 addendum

There have been numerous comments over at HisTalk (at http://histalk2.com/2013/08/27/news-82813/) defending the outage as not EPIC's fault.   From the point of view of clinicians - and more importantly, patients - it doesn't matter what component of the hospital's entire "EHR" (an anachronistic term used for what is now a complex enterprise clinical resource and clinician command-and-control system) went down. 

Aside from all the EPIC issues the nurses have been complaining about (see earlier July 12, 2013 post linked above), the larger problem is that IT malpractice occurred.  The term "malpractice" is used in medical mishaps; I see no reason why it does not apply to major outages of mission critical healthcare information technology systems.

IT malpractice in healthcare kills.

These are the types of nurses I'd want caring for me and mine.  Letting this kind of snafu go "anechoic" does not promote proper management remedial education on Safety 101 and on health IT risk, two areas of education that management appears to desperately need in hospitals.

-- SS

Boulder Community Hospital computer records back on line - but something does not add up

This post is in followup to my March 20, 2013 post "Boulder Community Hospital computer system crash: Either you're in control of your information systems, or they're in control of you".

At a March 24, 2013 Denver Post article "Boulder Community Hospital computer records back on line" the following statements are made:

The computer system that Boulder Community Hospital uses to manage patient records, which had been down for almost two weeks, is now up and running again, hospital officials said Saturday.

Meditech, the system used by the hospital to manage patient records, went down March 12 and affected the hospital, its Foothills campus, eight laboratories and six imaging centers. It was put back into full service at about 3 p.m. Friday, according to hospital spokesman Rich Sheehan.

Sheehan said an investigation showed the outage was a result of a malfunction in one of the main computer servers ... the hospital has replaced the hard drives for the server that failed and are inspecting the remaining servers ... [the failure] resulted in the system being unable to access patient information. The malfunction affected both the primary server and a backup server kept off-site.

A hard drive failure led to a two-week outage of an entire EHR system and its offsite backup server?  A mission-critical system in a hospital is so fragile that a hard drive failure caused a two week outage?

If so, that itself shows, at best, poor overall system design with regard to reliability and redundancy (any server worth its salt has hard drives in a failure-tolerant configuration e.g., RAID), but also is not quite credible on its face.  A remote server should not be taken down by the failure of a local server.  I suspect the failure was more than just a hard drive failure, including software bugs or configuration errors, mass hardware and/or network failure, or even sabotage.

The following statement also lacks believability on its face:

... All patient data was recovered except for an eight-hour period the day of the outage. Sheehan said the hospital had to re-create, re-enter and validate the patient information for that eight-hour period before the system could resume normal operations.

If an information system is down for two weeks, there's two weeks worth of data lost.

... Sheehan said the hospital has replaced the hard drives for the server that failed and are inspecting the remaining servers. The hospital is also now doing data backups every four hours as opposed to every six hours, and is planning on doing hourly backups by the end of the week.

Replacing a failed hard drive is an inadequate precaution.  A 'system redundancy makeover' seems in order for when the next hard drive fails.   Hard drives have a very well known MTBF (mean time between failure) and annual failure rate.  (The very Seagate ST3750528AS hard drive in the PC I am typing this blog post on has an Annualized Failure Rate of 0.34%, per the manufacturer's publicly-available literature.) 

... An independent consulting firm also has been hired to conduct an investigation. The hospital said it expects a report within a few weeks. 

As other organizations are using Meditech products, Joint Commission Safety Standards (as I wrote in a 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards" available at this link) call for sharing the results of that report with other organizations.  I had discussed this letter numerous times with senior Joint Commission leadership.

Will sharing of the independent consultant firm's report happen?  Probably not.

However, rest assured the Plaintiff's attorneys of Colorado will request it in malpractice suits that arose during the time period of outage.

-- SS

Boulder Community Hospital computer system crash: Either you're in control of your information systems, or they're in control of you

Yet another health IT crash, "prolonged" this time, from some unspecified "glitch":

Boulder Community Hospital computer system crash frustrates patients
Officials say it could take until Friday for outage to be resolved
By Brittany Anas
Camera Staff Writer
Posted:   03/18/2013 07:23:23 PM MDT
Updated:   03/18/2013 07:24:16 PM MDT

A prolonged computer system outage is preventing Boulder Community Hospital from accessing patient records -- making it difficult for people to schedule surgeries, get test results and make appointments for routine blood work.

Meditech, the system used by the hospital to manage patient records, went down in the middle of last week. It could take the hospital until Friday to get the system back up, said Rich Sheehan, spokesman for Boulder Community.

That fits my definition of "prolonged."

While information technology officials are investigating what caused the outage, Sheehan said patient records are protected and hospital officials don't believe they've been hacked. 

That's not very reassuring, considering the length of the outage.

The outage affects the hospital, its Foothills campus, eight laboratories and six imaging centers. 

Patients are on its face put at-risk ... for example, I know of several deaths of infants and adults from delayed x-ray reports alone ... but the clinicians, not the IT seller or hospital IT staff, are liable.

"We know medical care is important to people, so we understand the concerns those in the community have," Sheehan said. "We have a lot of people working on this, doing the best they can to solve this problem in a safe manner and as quickly as possible." 

"We know medical care is important to people?"  No, really?

In the meantime, the hospital is using manual paper record-keeping systems and traditional paper charts for its inpatients. Hospital officials say the system allows them to continue treating patients, provide diagnostic services and collect important clinical information that will be entered later into each patient's electronic health record.

But that concerns Eroca Lowe, whose mother was in the hospital Thursday through Sunday with gallbladder pain.

Lowe said the outage made it extremely difficult for doctors and nurses to do their jobs while hunting down lab results. She criticizes the hospital for not having a backup computer system and resorting to paper records.

"That's not a hospital in 2013," she said.

It's a good bet the paper records and HIM personnel managing them are not what they used to be pre-computer.

... Dina Huber said it took her and her significant other six days to schedule an appointment for a hernia surgery because the system used for scheduling is down.

"If they can't keep their computer system running, how can we trust them to perform surgery?" Huber said.

Fortunately, surgeons perform surgery ... not computers, IT staff or management.  Doctors, as the enablers of healthcare, don't need computers to save lives.

However, making their job harder is not a good idea.

A physician who works at Boulder Community Hospital, speaking on the condition of anonymity, said he doesn't think the outage is compromising the health or safety of patients. But, he said, the backup response "seems a little haphazard, and it's not an organized plan." He said physicians are left chasing down records.

If a prolonged outage "is not compromising safety", then why did the hospital spend tens of millions on computers?

Sheehan said the hospital is prioritizing accuracy and patient safety while getting the records system up and running. 

Once it's running again, there is significant risk of data now recorded manually being lost, thus again increasing error risk.

"We apologize for the delays, but this was an unavoidable situation," Sheehan said.

("We apologize for the chilly water, but this was an unavoidable situation." - Captain of the HMS Titanic?)

If an injury occurs, how will that sound to a jury?

Let me answer that:  like bull***.  

My response to Mr. Sheehan and Meditech, and the IT personnel involved:  "Either you're in control of your information systems, or they're in control of you."

It seems the latter clearly applies here.

I pray nobody gets injured ... and that the principals don't end up before plaintiff attorneys I've educated on the issues of bad health IT.

-- SS

When "Human Error" Causes EHR Downtime, Who is Liable For Patient Injuries That Result?

In the Pittsburgh Post-Gazette was this story of yet another EHR "glitch":

March 9, 2013 12:17 am 

Human error the cause of UPMC electronic issue

A systemwide problem with UPMC's information systems Wednesday left electronic patient records and other data inaccessible for about three hours. A UPMC spokeswoman said the hospitals "immediately went to manual backup systems, and we quickly identified and fixed the problem." She said there was no indication that patient care was compromised by the incident, which was due to human error.

I will presume the "human error" was not a physician or nurse pressing the wrong button, but a "human error" involving the servers or IT infrastructure such as a botched system upgrade, action that caused a server room power fault, etc.

UMPC is a very large system as their webpage shows, showing approximately fifteen major facilities.

The now-expected "patient care has not been compromised" line was provided to the Gazette, a line so commonly heard after EHR outages that I  use it as a Healthcare Renewal indexing tag (see this query link).

The following questions arise:

• What, exactly, was the "human error" and why was there no fault tolerance built into these mission-critical systems to account and compensate for it, such as via redundancy?

• If paper is so bad as a record-keeping medium that hundreds of billions of dollars are being spent to replace it, then how can patient care not be compromised, especially when multiple hospitals unexpectedly and without warning have to return to its use? 

• How can a very large hospital system rapidly declare that "patient care was not compromised" without a thorough and comprehensive patient review, accounting for possible delayed negative outcomes (by way of just a few simple examples, due to medication or imaging delays?)

•  Who is liable for any adverse patient outcomes that occurred related to the sudden unavailability of past records:  the clinicians?  The "human" who committed the computer-related error?  The corporation, either for direct negligence in implementing and mandating use of a system prone to mass outage by human error, or vicariously for the negligence and/or misconduct of its information technology employees and/or agents?

•  How many "outages" will it take before some patient is outright, no-doubt-about-it harmed or killed?  Do we want to find out, or is a priority to have redundancies so these systems don't crash?

I, for one, would not want to have a family member be "crashing" at the time of a sudden, unexpected system-wide EHR outage.

-- SS