The Risk Factor

I have long argued that the IT community needs to separate IT failures from blunders.

Most organizations do not have enough IT project failures. The reason I say this is that, in my experience, most project cancellations (or escalations for that matter) are not true failures but instead represent blunders. There is a big difference. A project failure is one in which most project decisions and actions were correct at the time, but for some reason the project didn't work out. It is a professional project -- the project risks were assessed, managed, and accepted where required; the assumptions were checked; success criteria were defined; the plan was estimated and funded well; the stakeholders participated; and so on.

Project blunders, which I contend most project overruns and cancellations are, arise from Dilbert-like approaches to project management and implementation. There is little or no risk management, the project plan is a fantasy, stakeholder concerns are given short shrift, and on and on.

Well, in a distressingly familiar story in today's LA Times, yet another IT blunder is described. The lede paragraph reads as follows:

Since launching a $95-million computer system six months ago, the Los Angeles Unified School District has been beset by programming glitches, hardware crashes and mistakes by hurriedly trained clerical staff. The result: tens of thousands of teachers, cafeteria workers, classroom aides and others have been underpaid, overpaid or not paid at all.

Sounds like a blunder to me.

While the LA Unified School District payroll mess is one sorry affair, what is even worse is what has happened in Philadelphia. This from a 20 August 2007 press release from Philadelphia's City Controller Alan Butkovitz:

Since the late 1980’s the City of Philadelphia has spent an estimated $35 -$40 million on four separate attempts to replace its 30 year old Customer Billing Information System used for generating monthly water bills. All of these attempts have failed. The City is currently in the process of its fifth attempt, the “new” Project Ocean, at an additional cost of another 6.7 million dollars.

For a full report on the situation, you can go here.

ComputerWorld has good historical coverage of the issue beginning with a recent story posted here.

Controller Butkovitz did say that:

I want to put the City on notice that any sign of failure in the future, will trigger an immediate hold by me on future payments to this and any vendor involved in this project.

One can only hope - but given past failures I wouldn't bet on it.

The Department of Homeland Security (DHS), after spending $42 million, has shut down its anti-terrorism data-mining tool Analysis, Dissemination, Visualization, Insight and Semantic Enhancement (ADVISE). Seems that it was being tested with information on real people rather than made-up data, which was against policy and probably the law.

According to the AP story, "ADVISE is not expected to be restarted," DHS spokesman Russ Knocke said. DHS' Science and Technology directorate "determined that new commercial products now offer similar functionality while costing significantly less to maintain than ADVISE."

ADVISE (I wonder how long and how much it cost to come up with that acronym) was supposed to, among other things, report on suspicious people going through customs. In a bit of multiple ironies, the London Guardian disclosed just the day before ADVISE was being closed, that the Metropolitan Police's Special Branch had been spying on George Orwell.

One report from 1942 noted that Orwell was a suspicious character because he dressed "in a bohemian fashion both at his office and in his leisure hours."

Hmm, I wonder if ADVISE was also data mining for people who fit the profile "bohemian fashion," "work hours"and "leisure hours" as a match for "suspicious person." If not, maybe the new, commercial data mining products can be set to be lookout for these characteristics - never know who you might catch.

And just think what Special Branch could have done with ADVISE back then.

Dr. Brian Randell, Emeritus Professor, and Senior Research Investigator, School of Computing Science, University of Newcastle upon Tyne, was kind enough to let me know that the Journal of Information Technology has just released an issue focused on the UK National Health Service's (NHS) National Program for IT (NPfIT), its electronic health record initiative.

I think you'll find the articles very informative.

"We don’t need hackers to break the systems because they’re falling apart by themselves,” said Dr. Peter Neumann in an New York Times article, "Who Needs Hackers?" discussing how IT systems are falling apart. Peter and several others discuss the increasing complexity of IT systems today, and how system design and development haven't been keeping up, often as a matter of convenience more than lack of knowledge (which I also argue in my IEEE Spectrum article on "Why Software Fails.")

Some 19 years ago to almost the day (11 September 1988), the NY Times published a story titled, "In Computer Behavior, Elements of Chaos." In this article, the late Dr. Alan Perlis postulated that the break down in networks that were occurring with greater regularity during the late 1980s, "lies in the inevitable disparity between the real world and the models used to simulate it. Even the finest computer simulation is only an approximation. At some point that cannot be determined in advance, the discrepancies between reality and the computer's simplified world view will lead to a chaotic breakdown."

"The only way we can improve our systems is to be prepared to continually redesign them when they fail - which they almost certainly will."

Some things never seem to change, eh?

"Unfathomable."

That's how Gov. M. Jodi Rell of Connecticut described the incident involving a computer backup tape that was stolen in June from a car in Ohio that contained bank account among other financial data for nearly all Connecticut state agencies as well as sensitive information on 1.3 million Ohio residents, according to an article in the New York Times.

The tape was in a car of an intern working for Accenture, which was hired by both Connecticut and Ohio to develop a computer systems integrating payroll, accounting, personnel and other fiscal functions. According to the Times report, "Rich Harris, a spokesman for Governor Rell, said yesterday that Accenture seemed to have used the program it created for Connecticut as a template for its project in Ohio, 'and it’s our understanding that this is how the data got mixed up' on the tape."

Los Angeles Federal Judge Audrey B. Collins issued a preliminary injunction yesterday against RMG Technologies, Inc., of Pittsburgh, Pennsylvania ordering the company "to stop creating, trafficking in, or facilitating the use of computer programs that allow its clients to circumvent the protection systems in the ticketmaster.com web site." Users of RMG software, typically ticket brokers and some ticket scalpers, have used it to flood Ticketmaster to obtain large blocks of tickets, denying consumers an opportunity to buy tickets.

According to a Wall Street Journal article, a recent Hannah Montana concert, the retail price of a ticket was $63, but were being sold for an average of $237. For some shows, according to the New York Times, the show's tickets were sold out in 12 minutes, and then appeared on sale for on the internet up to 10 times their face value. Ticketmaster said that for some shows, software "bots" were responsible for as much as 80% of all ticket requests.

Last May, you may recall, TB patient Mr. Andrew Speaker flew back to the US from Europe over his doctors’ objections, and was able to enter the US even though he was on a travelers’ watch list. To reduce the possibility of something like this happening again, US Custom and Border Protection officials said that they were putting new procedures in place.

Well, last week it was disclosed that a Mexican national with multi-drug-resistant tuberculosis boarded 11 flights, at least one to the United States and crossed the US border a total of 76 times. Customs and Border Protection (CBP) officials were warned on April 16 that this person was infected, but it took the Department of Homeland Security until June 7 to warn the inspectors on the border and the Transportation Security Administration to add this traveler to the travelers' watch list.

So there were actually two incidents, one highly publicized and one not, happening simultaneously. During the Speaker incident, DHS said that it was inexcusable what happened.

However, it is very clear that given the bad publicity of the Speaker case, senior DHS officials deliberately tried to keep this other traveler off the watch list until things quieted down a bit. The DHS, surprise, surprise, is not commenting on this latest "oops".

As I wrote before, I was skeptical that the Speaker incident would trigger a wider review of the limitations of the Custom and Border automated travelers' watch system as well as its systemic role in being able to manage the risks of travelers having infectious diseases. I guess I was more correct than I knew, unfortunately.

An interesting article was published in today's LA Times on a federally-funded identity-theft study performed by the Center for Identity Management and Information Protection (CIMIP) located at Utica College in New York. The study says that contrary to popular belief, about half of identity theft is performed by strangers, not family or acquaintances, as reported by others like Javelin Strategy & Research and ID Analytics. Both have strongly suggested (here and here) that on-line id theft was overblown, and that consumers shouldn't be worried about it.

Javelin said that the CIMIP study didn't contradict their work (which is funded by Visa USA, Wells Fargo & Co., and others with a vested interest in promoting on-line transactions) because the CIMIP study focused "on high-dollar cases" which would "more likely to involve businesses, strangers and technology" than their broad base of consumer victims reached through telephone surveys.

Okay, sure.

Anyway, I think it is going to take some time sorting out who is at risk by whom, but regardless, on-line or off, it isn't getting any safer out there.

WellCare Health Plans Inc. of Connecticut appears to be too busy to fix a software bug that is harming low-income adult and children Medicaid patients, the Hartford Courant is reporting today. During the summer, WellCare and two other insurance companies, Anthem and HealthNet, were discovered "accidentally" sending pharmacists computer messages saying that a prescription was not covered when in fact what should have been sent was that the prescription required prior authorization from the insurer. By law, managed care organizations are required to cover all drugs that are approved by the federal Food and Drug Administration.

Anthem and HealthNet have already fixed the problem, but Wellcare says it can't do so until December 1st. Must be part of a larger WellCare software maintenance build, I guess. It may also be because the FBI, Department of Health and Human Services, and Florida Medicaid Fraud Control Unit raided the company's Tampa, Fla., headquarters last Wednesday.

Wellcare says "to the best of its knowledge" it knows of no one who has been denied coverage, but Connecticut Attorney General Richard Blumenthal said his office has credible and plausible reports that prescriptions have been denied.

As reported in the Palm Beach Post, the Palm Beach County courts are trying to determine whether they should scrap their computer system that had a $13.6 million upgrade last year. The upgrade got them off their old mainframe onto a newer platform, and it was slated to give the court some functional upgrades as well.

Unfortunately, things haven't turned out too well. For example, when the court's computer system electronically alerts the Florida Department of Motor Vehicles of license suspensions, court staff have to telephone the DMV to ensure the information was not only received but received correctly. Another example was that before sending out 40,000 letters ordering people who owed the court money for unpaid fines, court staff had to manually check to ensure that they were mailing letters to the people who actually owed the court money and that the amounts stated in the letters were correct.

As a result of the problems, the courts have had to hire an additional 29 staff at a cost of $1.4 million per annum to try to keep the court system operating to some level of normality.

The upgrade, which was originally estimated to take six months to a year to convert all the data stored in the mainframe into the new system actually took 3 1/2 years. When the initial schedule estimate was made, court officials figured they would encounter three or four different methods of inputting data into the system. However, over a 150 different were actually being used.

No one seemed to checked this "minor" assumption before the contract was let. It gets better.

The Rocky Mountain News had an article today on Denver International Airport's (DIA) final agreement with two companies to demolish and cart away the remnants of its infamous automated baggage system. The cost for its removal is not known exactly, but it will be in the millions.

A fitting comment on was made by Denver City Council President Michael Hancock, "This thing never dies."

An interesting point in the story is that there may be a final, accurate accounting of the costs for the automated baggage system, something that has never been fully determined.

The article also said that a new baggage system will be in place soon.

Tony Collins, over at ComputerWeekly, has written a fascinating story about the secret (until now) political decisions to create the UK National Program for IT (NPfIT), the UK's attempt at creating a national electronic health record (EHR) system, similar to what Sen. Hillary Clinton (my apologies for not using her proper title before) is currently advocating, and what President Bush wants in place by 2014.

In papers obtained by the UK Freedom of Information Act, it appears that former Prime Minister Tony Blair in 2002 wanted a full fledged EHR system by early in the year 2005, before the next general election he would have to call. Even an EHR system operational by the 2005 date was seen by Blair as taking too long!

It is apparent that the potential for improved patient health care that EHRs promise was cavalierly traded off for immediate political gain - not a big surprise, of course. The haste and lack of concern for the technological implications in which the NPfIT decision was made is still breath-taking, nevertheless.

Best guess is that it will be 2013 before NPfIT is fully up and running; however, doctors aren't particularly supportive of it; nine out of ten doctors don't believe that the UK government can protect patient data; many doctors and privacy advocates are suggesting patients opt of of it; and support contractors are thinking of pulling out.

As I have mentioned, politicians seem to believe that they are the most brilliant and clever IT system architects that exist.