The Risk Factor

It looks like the six German-made, Russian programmed computers on the International Space Station (ISS) are back up and running after a few days of tense troubleshooting trying to discover the reason why they wouldn’t reboot properly. The computers which control the ISS’s navigation and command and control systems shut down last Wednesday, and there was trouble rebooting them. ( A good time line and incident details can be found at CBS News Space Place.)

These problems had been preceded by problems on Tuesday, where a computer crash prevented the ISS from immediately taking over gyroscopic control as planned from the docked shuttle Atlantis. During the computer rebooting sequence, a false fire alarm on the Russian segment of the ISS was sounded. Later Tuesday night, gyroscopic control was handed back to the ISS computers, although the reason for the computer crash on was not understood. However, only one out of the three navigation and one out of the three command and control computers were working after the successful reboot.

Early Wednesday morning, while astronauts were outside working on retracting a solar array wing, the two remaining computers crashed, and none of the computers would reboot – a first in ISS history. If the computers could not be rebooted, the ISS would potentially have to be abandoned. Making the troubleshooting a bit harder was that the Russian Federal Space Agency Roskosmos does not have its own satellites which can communicate with the ISS, forcing Russian space engineers to wait until the ISS is within line-of-site of Russian ground stations to downlink the needed telemetry to perform troubleshooting.

By yesterday afternoon, the computers were back up and working. There was a belief that there was a problem with the quality of power supply to the computers, possibly caused by the addition of new solar arrays. Russian astronauts used jumper cables to by-pass the computers’ surge protectors, and lo and behold, the computers booted up as normal. While this solution points to the source of the problem, the reasons why remain a mystery. NASA’s space station program manager Michael Suffredini probably summed it up best when he said, “As the station gets bigger, this potential [for problems] continues to grow. I think we’re going to find system sensitivities as we change the space station.”

There are a number of interesting aspects to this story. First, while the computers (and software) were designed to be redundant and independent, the power supplies to them don’t appear to be so. I bet that this issue is going to get a hard look in the next few weeks by NASA and Roskosmos.

Second, this episode will likely mean more consideration for possible unintended consequences to not only the computers but other systems and their interfaces aboard the ISS as it continues to be constructed. Even after all these years in space, surprises can still occur and nothing up there can be seen as ever being easy.

Third, the folks who are working on the Mars program are likely trying to figure out whether there is something they now need to be worried about. A mission to Mars could last well over two years, and any computer problems on that little voyage could spell big trouble.

Fourth, reliable computers are really, really important in space. This crash was not by any means the first, nor will it likely be the last. In 2001, during the shuttle Endeavour’s visit to the ISS, all three of the ISS command and control computers shut down, which was apparently caused by a bad hard drive.

Finally, having a really good tool kit around with lots of patch, jumper cables and spare parts about is priceless. While it often appears to be, not every computer problem is a software problem

This coming Thursday, the 4th of October, will be the 50th anniversary of the launching of Prosteishiy Sputnik (or the Simplest Satellite) and the beginnings of the Space Age and Space Race. Only now is the fascinating back story detailing the events leading up to the launch coming out in the open.

For instance, the public was told that the object they were seeing as it twinkled across the night sky was Sputnik itself. However, the satellite weighing in at 184 pounds was too small to be seen with the naked eye. What people actually were looking at was the second stage of the booster rocket used to lift Sputnik into orbit. Interestingly, the Soviet leadership at the time did not at first realize the magnitude of their achievement until the Western governments and press made a big deal out of it.

Yesterday, Fairchild Semiconductor celebrated its 50th anniversary as well. Founded by Gordon Moore, Robert Noyce, C. Sheldon Roberts, Victor Grinich, Eugene Kleiner, Jean Hoerni and Julius Blank, and Jay Last with $3,500 of their own money, the company helped make Silicon Valley. Fairchild perfected the capability to mass produce transistors from a single wafer, whereas up to this point only one transistor could be produced per wafer. The company also created the monolithic integrated circuit and the planar transistor, which is still the the primary method for producing transistors today.

Moore and Noyce left 11 years later to start another company in the Valley, something called Intel.

James Oberg reports in an IEEE Spectrum webcast a very important story on the background to the NASA computer failure that occurred in June. Oberg stories states that, "The critical computer systems ... had been designed, built, and operated incorrectly—and the failure was inevitable. Only being so relatively close to Earth, in range of resupply and support missions, saved the spacecraft from catastrophe."

The problem was a cable short-circuit caused by moisture build-up, likely itself caused by a malfunctioning dehumidifier. But as Oberg writes, the short-circuit should not have caused the problems it did. "..in a shocking design flaw, there was a “power off” command leading to all three of the supposedly redundant processing units. The line was designed to protect the main computers, which are downstream of the power monitor, from power glitches too great for normal power filters to protect against. It does so by turning the computers off when it senses trouble. But in a failure unanticipated by its designers, this one command path itself was able to kill all three processing units due to a single corrosion-induced short."

As Oberg noted, if this happened on the way to Mars, it would likely have resulted in loss of the crew. What's worse, was the instinctive reaction of those involved to look for assigning blame instead of looking for the root cause of the problem, or a means to mitigate it.

Everyone interested in risk assessments, communication and management should read it.

If you haven't seen them yet, the Japan Aerospace Exploration Agency (JAXA) released some great HD pictures of earth rise and earth set as seen from the moon.

The Canadian Space Agency's Dextre (Special Purpose Dexterous Manipulator) robot apparently needs a software patch to fix a timing problem that is preventing Dextre from fully powering up.

According to CSA's website, "Dextre is an essential tool for maintaining and servicing the space station. With its dual-arm design providing added flexibility, Dextre will remove and replace smaller components on the Station’s exterior, where precise handling is required. It will be equipped with lights, video equipment, a tool platform and four tool holders."

"Dextre is a complex robot designed to perform intricate maintenance and servicing tasks on the outside of the ISS. Dextre will carry out delicate work that, so far, could only be accomplished by astronauts during spacewalk activities. In other words, Dextre will provide an alternative to astronauts, considerably reducing the amount of time that they have to venture out of the ISS to perform demanding spacewalks and providing more time for them to perform science on the ISS."

Engineers are "confident" that the patch will fix the problem (I noticed they didn't say software programmers were confident, though).

I wonder if the engineers are as confident as I am during every Microsoft Patch and Pray Tuesday?

Update: At the end of the day, it wasn't a software problem as first thought. The patch was uploaded, but nothing happen. Engineers next thought it was a faulty cable. This was by-passed, and power was restored. You can read more about it here at ComputerWorld.

A report in the London Times says that the high cost of satellite launches are making communication companies "flinch" at investing in new satellites. New, larger satellites are required to handle the increasing volume of mobile traffic especially in Asia and India.

The report says that the new generation of communication satellites (which cost $650 million and up) weigh up to 8 tons, and only the Ariane 5 rocket is currently commercially available to carry the satellites up into high orbit. With a virtual stranglehold on the market, Ariane is demanding $120 million per launch.

There is concern that the high launch and development costs will begin to slow down the introduction of new or upgraded communication services. Satellite makers like at least two launch suppliers, and until there is a competitor to Ariane, they are reluctant to move ahead.

As explained in the report by Jean-Marie Robert, the head of telecom satellites at Thales Alenia Space, “The way this industry works is that we build the satellite and the buyer then chooses the launcher they want based on price and reliability. But we need at least two launchers to have a competitive industry and to avoid expensive launches."

The high costs involved may also force space insurance rates to rise, further increasing the reluctance of communication companies to send up new satellites. Insurance costs have been rising, and the recent loss of the $150 million AMC -14 satellite which was to deliver television services to the US won't help.