The Risk Factor

There are reports tonight of a bridge collapse in Minneapolis, Minnesota. As I write this, the number of dead and injured is unknown.

The reason I add it to this to a blog on IS&T failure and success is that recently I spoke with Dr. Henry Petroski, professor of civil engineering at Duke University on success and failure of design, as articulated in his recent book, Success Through Failure. Dr. Petroski has written extensively on the history of bridge failure, and one of his predictions using historical evidence is that about every 30 or years or so, there is a major bridge collapse that surprises everyone. We are/were overdue for one.

It is too early to tell yet why this bridge collapsed, which is about 40 years old from news reports. But we shouldn't be surprised if it turns out that it was because of a design flaw hidden in plain sight.

California Secretary of State Debra Bowen must decide by this Friday whether to decertify any or all electronic voting machines used in California. A recent test of three popular voting machines showed that they were vulnerable to various forms of hacking.

There is some controversy about whether the tests were realistic - the "red" hacking team from the University of California had unfettered access to the machines - and now that the vulnerabilities/threats have been exposed, whether they can be defended against by officials at state polling locations. Before the decision is made, a risk assessment of these factors, as well as whether the magnitude of any voter fraud or lost votes that could occur in comparison to paper ballots needs to be done. Given the time, I doubt a thorough risk assessment is possible.

The day before California Secretary of State Debra Bowen decided to pull the plug on e-voting machines, across the country another electronic voting problem was causing fits. Seems that the US Congress electronic voting system went down during a extremely politically charged and extremely close vote dealing with agriculture and immigration. Republicans claimed they had won the vote 215 to 213, but Democrats claimed they won 216 to 212. Since they control the House, they indeed did win.

This has led to the creation of a special select committee that has subpoena powers to see if there was any "skulduggery" afoot. I doubt this action would have been taken if the vote was 400 to 28 for or against.

The lesson to be taken away: computer cock-ups only appear when consequences don't matter - it is always conspiracy when they do.

As I noted a few days ago, the Wall Street Journal published an article on how to get around your IT Department's security restrictions. As I think I made clear, I didn't think it was a particularly well-thought out article.

The WSJ finally published one letter yesterday questioning it that was written by Dr. William Hery from the Department of Computer and Information Science & Department of Management, Polytechnic University in Brooklyn. I thought I would reproduce it here, since I think it is spot on:

Your article "Ten Things Your IT Department Won't Tell You" (The Journal Report on Technology, July 30) is irresponsible. The article encourages and abets the circumvention of security controls developed by corporate information-technology departments. These controls are typically carefully thought out and based on a corporate-level risk analysis, with confidential corporate information, private employee information, corporate reputation and even the ability of the corporation to conduct business all at risk. It is unlikely that all employees who use the methods you suggest, even those who "play it safe," as described in your article, are knowledgeable enough to provide the level of protection the corporation needs.

The article also left out, except indirectly in one comment, a critical risk to the employee: Even though they are circumventing the technology, they are still bound by the policies that led to the controls. The resources the employees are using belong to the corporation for appropriate business use. By actively circumventing the policy, the employees are admitting that they know the policies. By violating the policies, they are subject to any penalties defined in the policies, including reprimand, poor performance appraisal and potential dismissal.

As reported in the London Guardian Unlimited, the UK Custody-National Offender Management Service Information System (C-NOMIS) that was intended to keep close track of the 330,000 prisoners and those serving their probation, is in deep trouble. The cost of the development, originally estimated at ₤240 million has jumped to an estimated ₤950 million.

An "urgent review" is now being ordered to see in anything can be salvaged from the effort, which has been halted. About ₤155 million has been spent so far, and cancellation the program would cost ₤5o million in cancellation fees.

A government official said without irony that the original cost and schedule were "optimistic." I guess so.

The review will be published this autumn - we'll keep you informed of the outcome.

The UK government has decided against suggestions made earlier this year by the Commons Public Accounts Committee to conduct an independent assessment of the NHS's electronic medical records project's business case.

The NPfIT program has been plagued by uncertainty since its inception four years ago. The government insists that everything is fine, but as the testimony taken by Commons Public Accounts Committee suggests, there are plenty of reasons to be concerned.

It is more than likely that in a few years, some government spokesperson will, like in the C-NOMIS situation, call for an urgent review because the NPfIT will also have been seen in retrospect as being a wee bit optimistic.

According to news reports, a computer problem with a U.S. Customs and Border Protection immigration system delayed more than 20,000 arriving passengers into LAX for hours. The problem began at 1330 local time yesterday, and a back-up system wasn't operational until 2145. The last person finally cleared customs at 0350 this morning. A couple of flights diverted to other airports to avoid the problem.

What is interesting in reading the various news stories was that U.S. Customs and Border Protection authorities were telling the press that initially it was 2,500 and then later 6,000 passengers were affected. In the LA Times article I linked to, you'll see that the airport said that it was 11,000 passengers that were affected but Customs finally ended up with a 20,000 passenger count.

I don't know what is more troubling - the computer glitch or the "lost 9,000."

The US Customs and Border computer malfunction at LAX on Saturday was blamed on a hardware fault. Once the fault occurred, the back-up system didn't immediately takeover, and once it did, surprise, surprise, the back-up system lacked needed capacity.

US Customs said that 17,398 passengers on 73 flights were affected. So, I guess that the "over 20,000 passengers affected" count given out by Customs yesterday was an over-estimate, while the 11,000 passenger number put out by LAX management was an undercount. I think LAX management may want to go back and see how they missed 6,000 incoming passengers.

The mayor of Los Angeles, Antonio Villaraigosa, reportedly called for - what else - but an investigation into what happened, and said that he was working with Customs officials to prevent another such a failure, which Villaraigosa called "troubling and unacceptable."

Wonder how the mayor plans to prevent computer systems from malfunctioning, or their back ups from being inadequately scaled. I think we would all like to know.

It is bad enough that US Customs did a poor job of systems design and contingency management, but may we all be saved from politicians who think they are instant computer system experts.

October 19, 1987 has become known as one of the world's stock markets Black Mondays. Part of the reason blamed for the rapid market fall-offs was automated trading.

By 1989, there were calls by Congress and others to stop the use of automated trading; one firm, Dean Witter Reynolds said they were going to stop using it as it "threatened the integrity of the market in customers' minds."

This attitude didn't last long. Program trading volume now runs about 30 - 35% of the New York Stock Exchange on a weekly basis, and occasionally as high as 50%.

Well, here we are twenty-years later, and program trading surfaces as a cause again in the current world's stock markets' wild gyrations, which as of today, has wiped out all this year's market gains. This time, it appears that the algorithms used by the trading programs underestimated (i.e., missed) the risks that the sub-prime mortgage meltdown implied.

In an interesting article in the Wall Street Journal (subscription required), there were many excuses given for why the trading programs failed this time: "A unique combination of factors," " A perfect negative storm," etc., etc.

For more on the problem, just Google the word "quant," which is the modern slang for quantitative automated trading programs.

Using computer models for market prediction is great as long as the current reality meets the model reality. Once they diverge, then they don't work very well if at all. This has been known and warned about for over forty years - yet it is a lesson that people just keep insisting they want to painfully relearn, which is why I call it the déjà vu to the nth power problem.

There is a report in Dark Reading that the IT company Verus, Inc. (the link to their site is dead) has gone out of business. Verus built websites for hospitals across the company, but its work was cited in at least five security breaches where confidential patient information was exposed.

Not only is this a warning to IS&T suppliers about taking security seriously, but also for those in the medical community about ensuring that their suppliers can handle the security & privacy requirements. It also points out a warning to those who want to place electronic health records on the web.

Yesterday morning (about 0900 EDT in the US) the internet phone and messaging company Skype acknowledged that its users were experiencing log-on problems due to a software problem. The problem shut down the service to an unknown number of customers around the world, but it was likely in the millions.

According to the Financial Times of London late this afternoon, the problem has been fixed. Now let the debate begin as to whether this will harm Skype in particular or internet calling in general.

It will be interesting to see how similar this "software problem" will be to the one that happened on 15 January 1990 when AT&T suffered a massive failure of its long-distance service due to a elementary programming error.

Skype disclosed what happened to its services last week. Turns out the problem was related to its new software update and the reboot process.

On Thursday, 16th August 2007, the Skype peer-to-peer network became unstable and suffered a critical disruption. The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update.

The high number of restarts affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources, prompted a chain reaction that had a critical impact.

Normally Skype’s peer-to-peer network has an inbuilt ability to self-heal, however, this event revealed a previously unseen software bug within the network resource allocation algorithm which prevented the self-healing function from working quickly. Regrettably, as a result of this disruption, Skype was unavailable to the majority of its users for approximately two days.

SKype went on to say that it was their fault, and not due to hacking. They also pointed out that, in effect, to expect more of the same in the future.

This disruption was unprecedented in terms of its impact and scope. We would like to point out that very few technologies or communications networks today are guaranteed to operate without interruptions.

eWeek posted an on-line slide show listing the "Most Disastrous Data Breaches" since February 2005. They list 17 of them: 5 caused by outside hacking, 1 by insider theft, 5 by inadvertent posting of information, 5 by devices (laptop, memory stick) being stolen, and 1 caused by data being lost.

One of the seventeen listed was the discount retailer TJX. The company announced last week that the cost of its data breach last year that affected 45.8 million of its customers was likely to exceed $150 million, although given its previous estimates this is probably an underestimate of at least 100% or more. To quote TJX's press release:

In the second quarter of fiscal 2008, the Company recorded an after-tax cash charge of approximately $118 million, or $.25 per share, with respect to the previously announced computer intrusion(s). This charge includes $11 million (after tax), or $.02 per share, for costs incurred during the quarter, as well as a reserve of $107 million (after tax), or $.23 per share, for the Company's exposure to potential losses. This reserve reflects the Company’s estimation of probable losses, in accordance with generally accepted accounting principles, based on the information available to the Company as of August 14, 2007, and includes an estimation of total, potential cash liabilities from pending litigation, proceedings, investigations and other claims, as well as legal and other costs and expenses, arising from the intrusion(s). In addition, TJX expects to incur future non-cash charges of approximately $21 million (after tax), or $.05 per share, that are not included in this reserve and could be recorded in fiscal year 2009. Together, these cash and non-cash charges represent the Company’s best estimate of the total losses the Company expects to incur as a result of the computer intrusion(s).

And people still argue that organizational IT security rules are meant to be broken.

A computer failure at Wells Fargo, the fifth-largest bank in the US, that knocked out its Internet access, telephones and ATMs over the weekend, has been fixed. The bank had to revert to its back-up systems until the issue was cleared up.

However, as reported in ComputerWorld, phishers are rapidly gearing up to exploit the event. According to the article, on-line scammers have been waiting for a problem to crop up at a large bank or financial institution which will help add legitimacy to their message.

So if you get something purporting to be from Wells Fargo, the best course is to ignore it.

The US Department of Defense announced that it was shutting down its controversial Talon data gathering program.

Talon was established in 2002 by then-Deputy Defense Secretary Paul D. Wolfowitz as a way to collect and evaluate information about possible threats to U.S. servicemembers and defense civilians at stateside and overseas military installations. It is being closed because reporting to the system had declined significantly, and it was determined to no longer be of analytical value, said Army Col. Gary Keck, a Pentagon spokesman.

A reason for its shut down was noted in an article in Government Executive,

A June 2007 report by the Defense Department's inspector general found that counterintelligence officials "maintained TALON reports without determining whether information on organizations and individuals should be retained for law enforcement and force-protection purposes."

In addition, the article notes that:

To ensure a mechanism to document and examine potential threats, Assistant Defense Secretary Paul McHale plans to propose a new, streamlined reporting system that can better meet the Pentagon's needs, an agency press release said. In the interim, Defense Department officials will send information pertaining to protection concerns to the FBI's Web-based threat tracking system.

What a "streamlined reporting system" means hasn't been explained, but past history says don't place bets that it isn't going to resemble a data vacuum cleaner.

Two anniversary's were observed in past two weeks. Twenty-five years ago, on 17 August 1982, the first compact discs containing Richard Strauss' Alpine Symphony were mass produced. By 1986, CD players outsold record players, and by 1988, there were more CD sales then record sales. CD sales reached their peak in 2001 with 712 million sold, but pressure from other formats like MP3 has cut CD sales by 25% by 2007.

Then thirty years ago, on 24 August 1977, RCA announced a suggested retail price of $1,000 for its VBT200 VHS VCR to be marketed in the US. This price point - and a fortuitous set of circumstances that created the slogan "Four hours, $1,000, SelectaVision" explained here and here helped propel the VHS format tapes into the lead over Betamax.

I don't know how I missed it, but the Humane Society of the United Sates has been extremely successful at getting states, and now it looks like the Federal government, to outlaw Internet hunting via H.R. 2711, the Computer-Assisted Remote Hunting Act.

According to a recent story in the Wall Street Journal (subscription may be required), the Humane Society has been mailing people "an urgent message, underlined and in bold type":

Such horrific cruelty must stop and stop now.

No debate there - except, as the WSJ article points out, no one is actually hunting using the Net, even though the Humane Society's site implies that it is rampant.

Better safe than sorry, I guess.

I have long argued that the IT community needs to separate IT failures from blunders.

Most organizations do not have enough IT project failures. The reason I say this is that, in my experience, most project cancellations (or escalations for that matter) are not true failures but instead represent blunders. There is a big difference. A project failure is one in which most project decisions and actions were correct at the time, but for some reason the project didn't work out. It is a professional project -- the project risks were assessed, managed, and accepted where required; the assumptions were checked; success criteria were defined; the plan was estimated and funded well; the stakeholders participated; and so on.

Project blunders, which I contend most project overruns and cancellations are, arise from Dilbert-like approaches to project management and implementation. There is little or no risk management, the project plan is a fantasy, stakeholder concerns are given short shrift, and on and on.

Well, in a distressingly familiar story in today's LA Times, yet another IT blunder is described. The lede paragraph reads as follows:

Since launching a $95-million computer system six months ago, the Los Angeles Unified School District has been beset by programming glitches, hardware crashes and mistakes by hurriedly trained clerical staff. The result: tens of thousands of teachers, cafeteria workers, classroom aides and others have been underpaid, overpaid or not paid at all.

Sounds like a blunder to me.

I received a short note from Prof. Brian Randell, Emeritus Professor, and Senior Research Investigator, School of Computer Science at New Castle University that the "Gang of 23" have been given permission to publish the supplementary evidence on Independent Reviews that was provided at their request to the House of Commons Select Committee on Health's inquiry into Electronic Patient Records.

You can find the new information here.

You can find a lot more background information on the UK National Programme for IT initiative pulled together by Dr. Randell and his colleagues here.

It would be nice to have a similar broad and deep information clearing house regarding the pros and cons of the US electronic health record initiative. Any takers?

While the LA Unified School District payroll mess is one sorry affair, what is even worse is what has happened in Philadelphia. This from a 20 August 2007 press release from Philadelphia's City Controller Alan Butkovitz:

Since the late 1980’s the City of Philadelphia has spent an estimated $35 -$40 million on four separate attempts to replace its 30 year old Customer Billing Information System used for generating monthly water bills. All of these attempts have failed. The City is currently in the process of its fifth attempt, the “new” Project Ocean, at an additional cost of another 6.7 million dollars.

For a full report on the situation, you can go here.

ComputerWorld has good historical coverage of the issue beginning with a recent story posted here.

Controller Butkovitz did say that:

I want to put the City on notice that any sign of failure in the future, will trigger an immediate hold by me on future payments to this and any vendor involved in this project.

One can only hope - but given past failures I wouldn't bet on it.

USA Today ran a small story last week on Nissan Motors plans to equip all of its cars and trucks with a dashboard gauge showing the fuel-efficiency of one's driving. The gauge displays your instantaneously computed miles per gallon as a bar graph - the more fuel efficient you drive, the longer the bar displayed.

Nissan claims that based on its in-house testing drivers will cut their fuel by 10%.

I bet if the price per gallon of gasoline was also displayed, or maybe the IRS standard cost per mile reimbursement rate (currently 48.5 cents per mile) used instead, people would drive even less. Seeing that the drive to the local store ten miles away cost you $9.70 might give you incentive to do it less.

Maybe Nissan will add in a costing feature as well in the future. The average cost per gallon gasoline or a total cost of driving per mile could be broadcast over a preset radio frequency, which then could be used to compute the cost per trip.

Given that Nissan's gauge looks software driven, this shouldn't be too difficult too add.

While not an IS&T related story, it is interesting from a speculative risk perspective. The London Telegraph had a nice little story on the auction of the key "believed to have fitted the locker that contained the binoculars for the crow’s nest."

As the story notes,

It is thought to have fitted the locker that contained the crow's nest binoculars, vital in detecting threats to the liner lurking in the sea in the pre-sonar days of 1912.

Catastrophically for the Titanic and the 1,522 lives lost with her, the key's owner, Second Officer David Blair, was removed from the crew at the last minute and in his haste forgot to hand it to his replacement.

This story should be a reminder that a small event in conjunction with a series of other improbable events can easily lead to disaster.

Without access to the glasses, the lookouts in the crow's nest were forced to rely on their eyes and only saw the iceberg when it was too late to take action.

BTW, the key is expected to bring between $125 K to $150K at auction.

Over the past decade or so, the UK certainty seems to been in a hurry to implement a 1984 society. It already has the largest DNA registry in the world, the UK National Criminal Intelligence DNA Database, which contains the records of over 4 million individuals. The fact that 1 in 8 records is faulty doesn't seem to be a deterrent to the police or government officials (like former PM Tony Blair) wanting everyone's DNA on file.

Then there is the children's' national registry, which by next year will have details of every child under 18 (all 11 million of them), including, "the country, listing their name, address and gender, as well as contact details for their GP, school and parents and other carers. The record will also include contacts with hospital consultants and other professionals, and could show whether the child has been the subject of a formal assessment on whether he or she needs extra help."

Of course, certain children 's records will be excluded (like those of politicians and celebrities), but for everyone else, some 330,000 "vetted" others will have access to them. The government has promised tight security over the records, but then why are some records being excluded? I wonder if celebrity and politician children have their DNA kept off the DNA registry as well - since there are over 100,000 innocent children DNA records on file.

Of course, the UK has a big lead in security and CCTV cameras as well, with an estimated 4.2 million in operation. There are red light enforcement and speed cameras throughout the country as well. This spring, new "talking" CCTV cameras were being installed in 20 areas across England that will inform individuals that they are engaged in littering or other anti-social behaviors.

And to add a bit more emphasis to the idea that we're from the government and we are here to help is a plan to implant microchips in trash containers as a means to encourage people to throw out less rubbish.

The UK does seem on the cutting edge of using IS&T to shape a different - if not necessarily - better society.

The FAA announced today that the team lead by ITT Corporation has been selected as "the prime contractor for Automatic Dependent Surveillance-Broadcast (ADS-B), the keystone technology to the Next Generation Air Transportation System. The new system promises to significantly reduce delays and enhance safety by using precise signals from the Global Navigation Satellite System instead of those from traditional radar to pinpoint aircraft locations."

"The contract is worth approximately $1.8 billion from 2007 to 2025. ITT Corporation will build the ADS-B ground stations and own and operate the equipment. The FAA will pay subscription charges for ADS-B broadcasts transmitted to properly equipped aircraft and air traffic control facilities."

Hopefully, ITT will be able to implement the ADS-B on time, on budget and to specification, and doesn't end up like the last major air traffic control upgrade effort called the Advanced Automation System (AAS) project.