Sickness in the Information Technology Sector: Technical problems, discord plagued Maryland health care site

Eye-opening, ground-level, no-holds-barred accounts of IT mismanagement and mayhem are too rare, considering the stakes in 2013.  The behind-the-scenes crap that goes on needs a great deal of sunlight. Below is such an account of great merit, in the Baltimore Sun.

First, I note physicians and nurses are generally able to collaborate to take care of sick patients.  They are natural "team players"; taking care of patients is their professional calling.  I did it all the time in my patient care years, especially in critical care settings, where lack of collaboration with colleagues could lead to dead patients.

IT personnel are another species entirely.  It's all about territory, competition, self-promotion, empire-building, drinking from the ever-flowing fountain of money, etc.; process matters more than results; and to hell with the end users, let alone patients.

Here is, in macro form, the Fifty Shades of Mass Dysfunction I've encountered any time my professional work intersected with IT personnel, whether in hospitals or industry.  And I was supposed to simply "shut up" about this crap - 

Technical problems, discord plagued Maryland health care site
http://www.baltimoresun.com/health/bs-hs-exchange-woes-20131207,0,6559272.story 

By Meredith Cohn and Andrea K. Walker, The Baltimore Sun 10:50 p.m. EST, December 7, 2013

Although state officials have provided the public scant detail about the troubled launch of Maryland's version of Obamacare, emails and documents show that the project was beset behind the scenes for months by an array of technical issues, warring contractors and other problems.

Since Maryland's online health exchange opened Oct. 1 for people to buy insurance under the Affordable Care Act — and immediately crashed — the two main companies in charge of the website have taken their fight to court, a corporate project manager was replaced and a high-powered consulting firm was quietly brought in to restore order. Though state officials initially said the crash of the online exchange was an unexpected and fixable problem, emails and documents obtained by The Baltimore Sun through state open-records laws outline serious issues before and after the launch.

The revelations came just days before Rebecca Pearce, the head of the exchange, resigned. State officials announced that move Friday night and pulled Carolyn Quattrocki from the governor's health reform office to serve as an interim replacement

I am going to reproduce some paragraphs that show just how chaotic are the processes - in any health IT endeavor, whether they be for the exchanges that are essential to getting insured, or to the hospital IT systems that are essential to getting out of the hospital in something other than a pine box.

Emphases mine:

Just two weeks before the launch, Pearce visited the prime contractor's Linthicum headquarters and found a room of empty seats. She fired off an email questioning the company's commitment to resolve problems and reminding the contractors of what was at stake: "Tonight, I am begging. I don't know how else to say it: we have got to make this a reality."

Despite her proddings, in-fighting between contractor Noridian Healthcare Solutions and a key subcontractor, EngagePoint Inc., disintegrated amid finger-pointing and accusations in court papers. At one point, after Noridian severed contractual ties between the companies but continued to ask for help, EngagePoint CEO Pradeep Goel emailed Noridian officials: "Are you people on crack cocaine?"

Contractors, subcontractors, all offering "solutions" in healthcare and all likely to have not a single soul on board with anything beyond a merchant-computing background.

And nice, friendly questions amongst the friendly, collaborative finger-pointing and accusations typical of the IT world when mass mayhem occurs.

And a begging boss?  How lovely.

... Pearce, who was hired in 2011 at a salary of $175,000, declined to comment on problems surrounding the exchange or her resignation.  [If I were her I would have resigned out of disgust with these idiots.  Oh wait - I did leave IT positions for reasons, in part, like that  - ed.]

The emails provided by the state covered the two weeks before and after the website launch. They give only a limited behind-the-scenes view of creating and launching the exchange. Officials withheld an unknown number of emails, saying state law exempts them from disclosure because they involved the decision-making process of high-ranking executive officials.

The troubles in Maryland mirror problems faced by other state exchanges, as well as the federal portal providing insurance options to consumers in 36 states.

 Emails withheld under claims of legal privilege means we'll never know all the dirt.

... In Maryland, Sharfstein said the complexity was compounded because of an aging state Medicaid computer system that needed to be integrated into the exchange. Officials also chose to customize existing technology that proved tougher to retrofit than expected, he said.

"Unlike buying a book online from Amazon, this process is more akin to applying for a passport, buying a home, and receiving an individually calculated tax credit all through a single web portal," O'Malley said Friday night. "We had more user glitches and user problems than we had hoped.

"A longer testing period might have allowed us to prioritize and address more of these problems before the launch date. Time and ultimate success will tell whether the decision to purchase off-the-shelf software and employ multiple contractor entities were good or bad decisions."

"Hope" is not a proper project management technique of which I am aware, especially in a project of the admitted complexity as this one.

"Tougher to retrofit than expected?" Expected by whom?  IT "experts" who believe in unicorns and the tooth fairy?  I point out that in medicine, such mistakes are called "malpractice."

A longer testing period might have allowed us to address more problems?  But they went live anyway?

... In early 2012, the state gave a $71 million contract to develop the website to a Noridian-led team that included Curam Software, IBM and Connecture. To save time in creating the exchange, the Maryland legislature exempted the contract from the normal procurement process, and North Dakota-based Noridian outscored three other bidders.

Sharfstein said Noridian will likely remain at work in its Linthicum offices beyond its contract's year-end expiration. The company has already been paid about $57 million but the state contract allows penalties for delays. State officials declined to comment on whether any penalties will be sought.

Noridian is ultimately responsible for delivering the system, Sharfstein said. EngagePoint, which is based in Calverton, was not included in the original contract and appeared to have been hired without the exchange's knowledge, officials said.

The state first learned of the companies' "deep strains" in the three months before the website launched, according to documents in U.S. District Court in Baltimore. The issues disputed included accounting, project management, intellectual property and payment.

Emails offer a glimpse at how their differences affected efforts to build the site and then fix post-launch problems. Pearce repeatedly questioned the contractors' commitment to the project after Gov. Martin O'Malley announced on national TV that Maryland's site would go live on time.

On Sept. 22, after Sen. Barbara Mikulski echoed the governor in publicly applauding Maryland's readiness, Pearce wrote the contractors: "It's time to get this right. Now. Period."

Noridian was also criticizing the subcontractor it hired. On Sept. 25, Noridian's project manager wrote to Goel, complaining that EngagePoint refused to perform critical work: "EngagePoint is responsible for 'designing and implementing [an exchange] system,'" the project manager wrote.

The 8 a.m. launch was supposed to allow the estimated 800,000 uninsured Marylanders to sign in and browse 45 plans from six insurers. Officials had warned of "bumps in the road," but the site crashed in minutes.

... "As the executives in charge of this program, I would like to understand from you exactly what is happening with the project and what you are doing to address the issues," she wrote to the contractors at 7:56 a.m. on Oct. 2. By 4:10 p.m., she questioned why 85,000 people had hit the "get started" button, but there fewer than 500 accounts had been created.

About a half-hour later, she wrote to the contractors, "Can you please provide an update on what is going on right now? Who is on site? What has anyone learned?"

Some of the companies' emails focused on achievements rather than dwelling on worsening problems.

Noridian CEO Tom McGraw wrote to state officials on Oct. 4, "We have seen increases in all aspects of the system performance over the last several hours and anticipate that these will start showing in the next report."

But four days later McGraw notified state officials that the project manager was being replaced.

Conflict, favoritism, fighting, "it's not my job", lies, spin, stonewalling, strife ... chaos.

"Efforts to build the site and then fix post-launch problems" is also typical of the IT world.  In health IT, especially clinical IT, those harmed or killed during the "fixing" process are considered a necessary sacrifice, a "bump in the road."

... Paul von Ebers, CEO of Noridian Mutual Insurance Co., Noridian's parent company, wrote on Oct. 10 that the consultants "expressed concern with ongoing coordination issues between the Noridian and EngagePoint teams." He requested a meeting to resolve "working differences" between the companies.

This was days before Noridian fired EngagePoint, sparking the angry email exchanges and dueling lawsuits between the companies. Noridian then sought to hire EngagePoint workers; EngagePoint sued and was met with a counter-suit.

"We are expected to do piecemeal work for Noridian after contract termination because you just woke up and decided you don't know what you are doing?" Goel wrote Oct. 26. "We are not going to respond to ridiculous emails from Noridian demanding our team members show up for work after being escorted out of the office."

That's the path to progress:  firing subcontractors and attempting to steal their employees, and lawsuits and countersuits.

I could go on, but read the entire postmortem account (itself rare in large IT projects) at the Baltimore Sun link above.

Some of the de-identified cases at my academic site on health IT difficulties reflect this type of discord; some are mine personally such as at http://cci.drexel.edu/faculty/ssilverstein/cases/?loc=cases&sloc=clinical%20computing%20problems%20in%20ICU , http://cci.drexel.edu/faculty/ssilverstein/cases/?loc=cases&sloc=Cardiology%20story and http://cci.drexel.edu/faculty/ssilverstein/cases/?loc=cases&sloc=Cultures%20of%20mismanagement%20toxic%20to%20healthcare%20quality.

I was expected to "go along to get along" with this BS, and was deemed "not a team player" when I spoke up for enduser (clinician) and ultimate customer (patient) rights.

In summary, the IT world is demonstrably dysfunctional, with the types of conflict as in the Baltimore Sun story more the rule than the exception, especially where healthcare is concerned.  Good, relatively brief resources for better understanding these issues are the following papers:

Social Informatics.  An introductory essay entitled “Learning from Social Informatics” by R. Kling at the University of Indiana can be found at this link (PDF).  The book “Understanding And Communicating Social Informatics” by Kling, Rosenbaum & Sawyer, Information Today, 2005 (Amazon.com link here) was based on this essay.

Pessimism, Computer Failure, and Information Systems Development in the Public Sector.  (Public Administration Review 67;5:917-929, Sept/Oct. 2007, Shaun Goldfinch, University of Otago, New Zealand).  Cautionary article on IT that should be read by every healthcare executive documenting the widespread nature of IT difficulties and failure, the lack of attention to the issues responsible, and recommending much more critical attitudes towards IT.  linkto pdf

“Defensive climate in the computer science classroom” by Barker et al., Univ. of Denver.  Link here(subscription required).  May help explain the control-seeking culture of IT personnel.   As part of an NSF-funded IT workforce grant, the authors conducted ethnographic research to provide deep understanding of the learning environment of computer science classrooms. Categories emerging from data analysis included 1) impersonal environment and guarded behavior; and 2) the creation and maintenance of informal hierarchy resulting in competitive behaviors. These communication patterns lead to a defensive climate, characterized by competitiveness rather cooperation, judgments about others, superiority, and neutrality rather than empathy.

If a person thinks this technology will "revolutionize" healthcare anytime soon, considering the "people issues" involved let alone the technical ones, then that person is either hopelessly naïve, or needs a mental health evaluation (colloquially, "needs their head examined").

-- SS

Over At The Health Care Blog, Aneesh Chopra Distorts IT Failure Reality

I feel sorry for President Obama.  I really do.  He's been deceived by IT hyperenthusiasts who, in addition to their hyperenthusiasm, probably heaped him a big helping of plain old lies.

I warned of this in my Feb. 18, 2009 Letter to the Editor published in the Wall Street Journal (at http://online.wsj.com/news/articles/SB123492035330205101, third letter):

Dear Wall Street Journal:

You observe that the true political goal is socialized medicine facilitated by health care information technology. You note that the public is being deceived, as the rules behind this takeover were stealthily inserted in the stimulus bill.

I have a different view on who is deceiving whom. In fact, it is the government that has been deceived by the HIT industry and its pundits. Stated directly, the administration is deluded about the true difficulty of making large-scale health IT work. The beneficiaries will largely be the IT industry and IT management consultants.

For £12.7 billion the U.K., which already has socialized medicine, still does not have a working national HIT system, but instead has a major IT quagmire, some of it caused by U.S. HIT vendors. [That project, the National Programme for IT in the NHS or NPfIT, was since abandoned - ed.]

HIT (with a few exceptions) is largely a disaster. I'm far more concerned about a mega-expensive IT misadventure than an IT-empowered takeover of medicine.

The stimulus bill, to its credit, recognizes the need for research on improving HIT. However this is a tool to facilitate clinical care, not a cybernetic miracle to revolutionize medicine. The government has bought the IT magic bullet exuberance hook, line and sinker.

I can only hope patients get something worthwhile for the $20 billion.

Now, more spin and misinformation by Aneesh Chopra, senior fellow at the Center for American Progress, and the former U.S. chief technology officer in the Obama Administration over at the Health Care Blog at a piece "What's Next For Healthcare.gov?" (http://thehealthcareblog.com/blog/2013/10/25/whats-next-for-healthcare-gov/).

Like a cybernetic Rasputin, Chopra writes:

The launch of HealthCare.gov certainly didn’t go as planned. Due to technical errors, millions of Americans were sent to the functional equivalent of a waiting room before they could enter the shopping portion of the site.

Historically, projects of such complexity and demand have encountered early problems yet still often achieve great success. While much of the commentary has focused on coding problems, the site still has the potential to spur innovation — be it public or private —  that will result in quality improvement and lower costs.

(Note the definitive "will", without evidence.  Mr. Obama's probably been hearing a lot of non-evidence-based wishful thinking about health IT in recent years.)

The statement about future success is belied, for example, by the National Programme for IT in the NHS (NPfIT) failing and going "pfffft" as just one example (http://hcrenewal.blogspot.com/2011/09/npfit-programme-going-pffft.html and other links).

More generally, there's this article by Shaun Goldfinch, formerly at the U. Otago in New Zealand:

"Pessimism, Computer Failure, and Information Systems Development in the Public Sector."  (Public Administration Review 67;5:917-929, Sept/Oct. 2007, Shaun Goldfinch, University of Otago, New Zealand). 

Unfortunately it's not freely available, but a free first-page preview is at http://www.jstor.org/discover/10.2307/4624644?uid=3739864&uid=2129&uid=2&uid=70&uid=4&uid=3739256&sid=21102828298677:

Summary:

The majority of information systems developments are unsuccessful. The larger the development, the more likely it will be unsuccessful. Despite the persistence of this problem for decades and the expenditure of vast sums of money, computer failure has received surprisingly little attention in the public administration literature. This article outlines the problems of enthusiasm and the problems of control, as well as the overwhelming complexity, that make the failure of large developments almost inevitable. Rather than the positive view found in much of the public administration literature, the author suggests a pessimism when it comes to information systems development. Aims for information technology should be modest ones, and in many cases, the risks, uncertainties, and probability of failure mean that new investments in technology are not justified. The author argues for a public official as a recalcitrant, suspicious, and skeptical adopter of IT.

Article start:

The majority of information systems (IS) developments are unsuccessfu1. The larger the development, the more likely it will be unsuccessful.

Though the exact numbers are uncertain and depend to some extent on how success is measured, something like 20 percent to 30 percent of all developments are total failures in which projects are abandoned. Around 30 percent to 60 percent are partial failures in which there are time and cost overruns or other problems. The minority are those counted as successes (Collins and Bicknell 1997; Corner and Hinton 2002; Georgiadou 2003; Heeks and Bhatnagar 1999; Heeks 2002, 2004 ; Iacovou 1999; James 1997; Korac-Boisvert and Kouzmin 1995 ; Standish Group 2001, 2004).

A U.S. survey of IS projects conducted by the Standish Group in 2001 found that success rates varied from 59 percent in the retail sector to 32 percent in the financial sector, 27 percent in manufacturing, and 18 percent in government. Overall, the success rate was 26 percent. In all, 46 percent of projects had problems, including being over budget and behind schedule or being delivered with fewer functions and features than originally specified. Another 28 percent failed altogether or were cancelled. Cost overruns averaged nearly 200 percent. Th is success rate varied dramatically by total project budget: For projects under US$750,000 the success rate was 55 percent; for those with budgets over US$10 million, no projects were successful (SIMPL/NZIER 2000).

More recent Standish Group (2004) estimates saw a success rate of 29 percent, but 53 percent of projects had problems and 18 percent failed. A New Zealand government study judged 38 percent of government projects a success, while 59 percent involved problems and 3 percent were a complete failure or were cancelled. Government success rates, at 31 percent, were slightly higher than private sector success rates. Above the NZ$10 million mark, however, the success rate for both was zero (SIMPL/ NZIER 2000). One study of hundreds of corporate software developments found that five out of six projects were considered unsuccessful, with one-third cancelled outright. Of the two-thirds that were not cancelled, price and completion times were almost twice what had originally been planned (Georgiadou 2003). The Royal Academy of Engineering and the British Computer Society (2004) found that 84 percent of public sector projects resulted in failure of some sort.

The sums involved in such projects can be staggering.  A study of IS developments in the British public sector estimated that 20 percent of expenditures were wasted, and a further 30 percent to 40 percent led to no perceivable benefits (Wilcocks 1994). In 1994, the U.S. General Accounting Office reported that spending of more than US$200 billion in the previous 12 years had led to few meaningful returns.

The article is extensively referenced, and nothing has changed since it was published.  While the article's cost from the jstor.org site is $25 (US I think), it is well worth reading for anyone involved in large public sector IT projects.

Especially, the President of the United States and his staff.

It is my belief the same actors who've misled him about health IT and the supposed ease of building a national insurance portal for 300 million+ people are going to mislead him about remediation of the latter, resulting in yet more embarrassment, and perhaps eventually patient harm and death when someone, somewhere, is denied care or has delayed care due to insurance loss.

Finally, I note this in Chopra's bio at the Health Care Blog Piece:

Aneesh Chopra is a senior advisor at The Advisory Board Company

The Advisory Board Company (http://www.advisory.com/About-Us) is:

... a performance improvement partner for 165,000+ leaders in 4,100+ organizations across health care and higher education.

The Advisory Board Company has been an advisor on hospital management and technology for decades.

Perhaps they're another piece in the puzzle as to why hospital executives are implementing bad health IT (http://cci.drexel.edu/faculty/ssilverstein/cases/) technology in droves, such as in the ED where many underprivileged people get primary and emergency care.  (See "Quality and Safety Implications of Emergency Department Information Systems: ED EHR Systems Pose Serious Concerns, Report Says" at http://hcrenewal.blogspot.com/2013/10/quality-and-safety-implications-of.html, for example.)

After all, with senior advisors to a major healthcare organization consultant in denialist roles, if the IT's bad now, it will be just great in ver. 2.0.

(See "The Denialists' Deck of Cards: An Illustrated Taxonomy of Rhetoric Used to Frustrate Consumer Protection Efforts" by Chris Jay Hoofnagle, available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=962462.)

The same Denialist Deck of Cards is being played, I fear, to frustrate taxpayer attempts at hemming in waste ... and to frustrate the President of the United States' efforts to leave a good legacy.

-- SS

Drudge Report, Oct. 10, 2013, 9 AM EST: All that needs to be said about government, computing and healthcare

Per Drudge Report. Oct. 10, 2013, 9 AM EST:

From the same people who brought us HITECH, the stimulus bill for rapid rollout of commercial electronic medical records, order entry, results reporting and other components of enterprise clinical "command and control" software for hospitals through which every transaction of care must pass.

More IT malpractice.  The Drudge links, as they appear on the page:

Obamacare website cost more than FACEBOOK, TWITTER, LINKEDIN, INSTAGRAM...
'How can we tax people for not buying a product from a website that doesn't work?'
Major insurers, Dem allies repeatedly warned Obama admin...
REPORT: WH knew site might not be ready...
POLL: Just 1 in 10 report success...
DNC head says site designed for 50,000 max...
Once you get in, you can't get out...
Crazzzzzzzy code...
'It looks like nobody tested it'...
WASHPOST: Not code, but 'outdated, costly, buggy technology'...
CARNEY: 'I Don’t Know' If Obama Has Tried Website...
Hawaii forced to relaunch after zero sign-ups...

I won't comment any further; I don't think I need to.

Drudge Report, Oct. 10, 2013, 9 AM EST.  Click to enlarge.

Of course, the Anecdotalists [1] and Denialists [2] will probably say this is all a "glitch" and that things will be great in ver. 2.0.

Fools all.

Oh, and the cost, via Drudge, per the linked story.  A mere:

$634,320,919

-- SS

[1]  See "Health IT: On Anecdotalism and Totalitarianism" at  http://hcrenewal.blogspot.com/2010/09/health-it-on-anecdotalism-and.html)

[2]  See "The Denialists' Deck of Cards: An Illustrated Taxonomy of Rhetoric Used to Frustrate Consumer Protection Efforts" by Chris Jay Hoofnagle, available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=962462)

Oct. 10, 2013 addendum:

Also see "Analysis: IT experts question architecture of Obamacare website" at http://uk.reuters.com/article/2013/10/05/us-usa-healthcare-technology-analysis-idUKBRE99407T20131005.  If the allegations here are even partially true, every programmer and manager who ever worked on this system should be summarily fired and never permitted to touch another computer involved in healthcare - ever.

-- SS