The Risk Factor

It looks like the six German-made, Russian programmed computers on the International Space Station (ISS) are back up and running after a few days of tense troubleshooting trying to discover the reason why they wouldn’t reboot properly. The computers which control the ISS’s navigation and command and control systems shut down last Wednesday, and there was trouble rebooting them. ( A good time line and incident details can be found at CBS News Space Place.)

These problems had been preceded by problems on Tuesday, where a computer crash prevented the ISS from immediately taking over gyroscopic control as planned from the docked shuttle Atlantis. During the computer rebooting sequence, a false fire alarm on the Russian segment of the ISS was sounded. Later Tuesday night, gyroscopic control was handed back to the ISS computers, although the reason for the computer crash on was not understood. However, only one out of the three navigation and one out of the three command and control computers were working after the successful reboot.

Early Wednesday morning, while astronauts were outside working on retracting a solar array wing, the two remaining computers crashed, and none of the computers would reboot – a first in ISS history. If the computers could not be rebooted, the ISS would potentially have to be abandoned. Making the troubleshooting a bit harder was that the Russian Federal Space Agency Roskosmos does not have its own satellites which can communicate with the ISS, forcing Russian space engineers to wait until the ISS is within line-of-site of Russian ground stations to downlink the needed telemetry to perform troubleshooting.

By yesterday afternoon, the computers were back up and working. There was a belief that there was a problem with the quality of power supply to the computers, possibly caused by the addition of new solar arrays. Russian astronauts used jumper cables to by-pass the computers’ surge protectors, and lo and behold, the computers booted up as normal. While this solution points to the source of the problem, the reasons why remain a mystery. NASA’s space station program manager Michael Suffredini probably summed it up best when he said, “As the station gets bigger, this potential [for problems] continues to grow. I think we’re going to find system sensitivities as we change the space station.”

There are a number of interesting aspects to this story. First, while the computers (and software) were designed to be redundant and independent, the power supplies to them don’t appear to be so. I bet that this issue is going to get a hard look in the next few weeks by NASA and Roskosmos.

Second, this episode will likely mean more consideration for possible unintended consequences to not only the computers but other systems and their interfaces aboard the ISS as it continues to be constructed. Even after all these years in space, surprises can still occur and nothing up there can be seen as ever being easy.

Third, the folks who are working on the Mars program are likely trying to figure out whether there is something they now need to be worried about. A mission to Mars could last well over two years, and any computer problems on that little voyage could spell big trouble.

Fourth, reliable computers are really, really important in space. This crash was not by any means the first, nor will it likely be the last. In 2001, during the shuttle Endeavour’s visit to the ISS, all three of the ISS command and control computers shut down, which was apparently caused by a bad hard drive.

Finally, having a really good tool kit around with lots of patch, jumper cables and spare parts about is priceless. While it often appears to be, not every computer problem is a software problem

As I wrote about in my previous post, world-wide efforts are underway to replace the paper-based medical record with electronic medical records (EHRs). For information on the US effort, you can visit the White House website to get some background information of the US effort, as well as the US Department of Health & Human Services (HHS) website to see current status information.

Something that has gone surprisingly unnoticed is that this month marks the hundredth anniversary of the modern paper medical record. This innovation, which we all take for granted, can trace its origins to Dr. Henry Plummer, a partner at the Mayo Clinic, in the year 1907. Plummer recognized that each patient’s medical history needed to be recorded, stored and retrieved in a different manner than was the current practice if the quality of patient care were to improve.

eWeek posted an on-line slide show listing the "Most Disastrous Data Breaches" since February 2005. They list 17 of them: 5 caused by outside hacking, 1 by insider theft, 5 by inadvertent posting of information, 5 by devices (laptop, memory stick) being stolen, and 1 caused by data being lost.

One of the seventeen listed was the discount retailer TJX. The company announced last week that the cost of its data breach last year that affected 45.8 million of its customers was likely to exceed $150 million, although given its previous estimates this is probably an underestimate of at least 100% or more. To quote TJX's press release:

In the second quarter of fiscal 2008, the Company recorded an after-tax cash charge of approximately $118 million, or $.25 per share, with respect to the previously announced computer intrusion(s). This charge includes $11 million (after tax), or $.02 per share, for costs incurred during the quarter, as well as a reserve of $107 million (after tax), or $.23 per share, for the Company's exposure to potential losses. This reserve reflects the Company’s estimation of probable losses, in accordance with generally accepted accounting principles, based on the information available to the Company as of August 14, 2007, and includes an estimation of total, potential cash liabilities from pending litigation, proceedings, investigations and other claims, as well as legal and other costs and expenses, arising from the intrusion(s). In addition, TJX expects to incur future non-cash charges of approximately $21 million (after tax), or $.05 per share, that are not included in this reserve and could be recorded in fiscal year 2009. Together, these cash and non-cash charges represent the Company’s best estimate of the total losses the Company expects to incur as a result of the computer intrusion(s).

And people still argue that organizational IT security rules are meant to be broken.

While the LA Unified School District payroll mess is one sorry affair, what is even worse is what has happened in Philadelphia. This from a 20 August 2007 press release from Philadelphia's City Controller Alan Butkovitz:

Since the late 1980’s the City of Philadelphia has spent an estimated $35 -$40 million on four separate attempts to replace its 30 year old Customer Billing Information System used for generating monthly water bills. All of these attempts have failed. The City is currently in the process of its fifth attempt, the “new” Project Ocean, at an additional cost of another 6.7 million dollars.

For a full report on the situation, you can go here.

ComputerWorld has good historical coverage of the issue beginning with a recent story posted here.

Controller Butkovitz did say that:

I want to put the City on notice that any sign of failure in the future, will trigger an immediate hold by me on future payments to this and any vendor involved in this project.

One can only hope - but given past failures I wouldn't bet on it.

As I wrote a a couple of days ago, the UK seems determined on making 1984 a reality. A senior UK judge, Lord Justice Sedley, in the name of fairness, called for everyone in the UK including visitors to have their DNA captured in a database. He objects that only those who come in contact with the criminal justice system have their DNA captured.

According to the London Guardian, Sedley said that "disproportionate numbers of people from ethnic minorities were on the database. 'It also means that a great many people who are walking the streets, and whose DNA would show them guilty of crimes, go free,' he said."

If George Orwell were alive today and updating 1984, I wonder how IS&T would influence the story line.

Homer Simpson: Facts are meaningless. You could use facts to prove anything that's even remotely true!

Last week, Sir Derek Wanless delivered his second review in the past five years on the UK National Health Service's efforts at modernization. According to the London Times, Wanless found that even after spending an additional £43 billion:

The money poured into the NHS has failed to produce a more efficient service, or to reduce unhealthy lifestyles.

As a result, more money will be needed.

The Guardian newspaper reported that Sir Derek's report included, " a warning that slow progress on introducing new IT systems could seriously undermine the productivity gains envisaged in 2002." He recommend that, ".. the £12bn programme run by the NHS agency Connecting for Health should undergo detailed external scrutiny to ensure the benefits will outweigh the costs."

This coming Thursday, the 4th of October, will be the 50th anniversary of the launching of Prosteishiy Sputnik (or the Simplest Satellite) and the beginnings of the Space Age and Space Race. Only now is the fascinating back story detailing the events leading up to the launch coming out in the open.

For instance, the public was told that the object they were seeing as it twinkled across the night sky was Sputnik itself. However, the satellite weighing in at 184 pounds was too small to be seen with the naked eye. What people actually were looking at was the second stage of the booster rocket used to lift Sputnik into orbit. Interestingly, the Soviet leadership at the time did not at first realize the magnitude of their achievement until the Western governments and press made a big deal out of it.

Yesterday, Fairchild Semiconductor celebrated its 50th anniversary as well. Founded by Gordon Moore, Robert Noyce, C. Sheldon Roberts, Victor Grinich, Eugene Kleiner, Jean Hoerni and Julius Blank, and Jay Last with $3,500 of their own money, the company helped make Silicon Valley. Fairchild perfected the capability to mass produce transistors from a single wafer, whereas up to this point only one transistor could be produced per wafer. The company also created the monolithic integrated circuit and the planar transistor, which is still the the primary method for producing transistors today.

Moore and Noyce left 11 years later to start another company in the Valley, something called Intel.

According to today's Boston Globe (registration may be required), the Massachusetts Appeals Court upheld the accuracy of information received from automobile event data recorders (EDR) for use in court cases. Event data recorders, sometimes called car "black boxes," are devices installed in a motor vehicle to record technical vehicle and occupant information for a brief period of time (seconds, not minutes) before, during and after a crash, according to the National Highway Transportation Safety Association website.

An EDR may record (1) pre-crash vehicle dynamics and system status (e.g., wheel speed, engine rpm), (2) driver inputs (e.g., braking, acceleration), (3) vehicle crash signature, (4) restraint usage/deployment status, and (5) post-crash data such as the activation of an automatic collision notification (ACN) system. According to an article in Time magazine, some 64% of cars made today have EDRs, and about 33% of all cars on the road today have them installed.

In the Massachusetts case, a woman was sentenced to two years in prison after her GMC Yukon skidded on ice and hit a tree, killing her passenger in 2003. The woman claimed that she was traveling only 20 to 30 miles per hour when she lost control, but the car's recorder showed that she was traveling 58 m.p.h. in a 40 m.p.h. zone. Her lawyer appealed her case arguing that the EDR's information was not reliable or accurate.

Consumer and privacy advocates have been opposite sides of the debate. According to the Time article, Public Citizen's Joan Claybrook "wants tougher rules compelling automakers to install EDRs in every car because objective crash data will lead to the design of safer cars and highways. Privacy activists want the government to prevent police and insurance companies from checking drivers' black boxes without permission. 'We have a surveillance monster growing in our midst," says Barry Steinhardt of the American Civil Liberties Union. 'These black boxes are going to get more sophisticated and take on new capabilities.' "

Like most technologies, once out of the bottle, they can't be put back in. And when its a question of public safety or privacy, privacy usually loses. The same will likely be true in the case of electronic medical records.

James Oberg reports in an IEEE Spectrum webcast a very important story on the background to the NASA computer failure that occurred in June. Oberg stories states that, "The critical computer systems ... had been designed, built, and operated incorrectly—and the failure was inevitable. Only being so relatively close to Earth, in range of resupply and support missions, saved the spacecraft from catastrophe."

The problem was a cable short-circuit caused by moisture build-up, likely itself caused by a malfunctioning dehumidifier. But as Oberg writes, the short-circuit should not have caused the problems it did. "..in a shocking design flaw, there was a “power off” command leading to all three of the supposedly redundant processing units. The line was designed to protect the main computers, which are downstream of the power monitor, from power glitches too great for normal power filters to protect against. It does so by turning the computers off when it senses trouble. But in a failure unanticipated by its designers, this one command path itself was able to kill all three processing units due to a single corrosion-induced short."

As Oberg noted, if this happened on the way to Mars, it would likely have resulted in loss of the crew. What's worse, was the instinctive reaction of those involved to look for assigning blame instead of looking for the root cause of the problem, or a means to mitigate it.

Everyone interested in risk assessments, communication and management should read it.

As many as 40 employees at Palisades Medical Center in North Bergen where actor George Clooney and a companion was taken after his motorcycle accident a few weeks back are being investigated for looking at his medical records, with over two dozen suspended without pay so far. It is probably a safe guess that at least one leaked Clooney's records to the press, since the media reported in detail on his injuries within "minutes" of his admittance.

The employees got to Clooney's medical records by accessing the hospital's computers. Let's hear it for computerized medical records - makes spying so easy.

As I noted a few weeks back, a celebrity's (reported to be ex-English football coach Sir Bobby Robson) medical records were looked at in a UK hospital.

A Palisade's hospital workers union spokesperson said, "It was inappropriate but they [the employees who sneaked a peak] are paying a steep price. But I don't even think George Clooney would want people to pay. Again, the apology to him for his privacy rights [is necessary], but I think in fact the hospital is overreacting."

"There are hospital obligations to have security systems so that a breach can't occur -- obviously that failed," she added. The spokesperson also tried to argue that since the employees (for the most part) only looked at Clooney's medical record and didn't disclose it (what, other than to friends and relatives?), it was a "no harm, no foul situation."

I hate to differ - I think they all need to be terminated. Or how about this as a compromise: a full public disclosure of the medical records (or better tax records - what's the difference?) of all those who sneaked a peak, and for fairness, let's include the union spokesperson since she thinks snooping does not rate a suspension, let alone a firing. That's a fair trade, right?

Furthermore to say that it's the hospital's fault for not having technology to keep prying eyes out is more than a bit self serving. In the UK incident, for example, those authorized to look at Robson's medical records simply gave access to those who did not. Technology doesn't prevent bad behavior or a lack of personal responsibility.

With attitudes expressed by this spokesperson, I would say that ensuring the privacy of electronic health records still have a long way to go.

Last week, ComputerWorld published a lengthy story about the disruption of the US Department of Veterans Affairs' VistA electronic health record (EHR) system in Northern California last August. According to the story, the outage was caused by "a simple change management procedure that wasn't properly followed."

It turns out that one group of maintainers asked another to make a change to a network port configuration without having the proper authorization to do so, which the second team did. In other words, the system was done in by poor configuration management.

For reasons left better explained by the ComputerWorld article, the VistA back-up systems that were supposed to kick in, didn't.

The outage caused the VistA system to be down for a good part of a day, which caused healthcare workers to revert to paper and pencil. Patient safety was increasingly put at risk, because the VA health system is almost completely electronic. In the VA's words, the outage was "the most significant technological threat to patient safety (the) VA has ever had.” It has taken months to put all the paper-based information created that day back into electronic format.

The VA experience provided a glimpse of what may happen if a major outage and back-up systems fail once EHR systems are fully up and running. System designers of EHR systems need to think a bit harder about what happens when the "unthinkable" does indeed happen.