Public domain photo courtesy of Yinan Chen/Wikimedia CC
Fictional stories about putting too much trust in technology often involve armies of killer robots. But what if some of today’s real threats of improperly checked technology are less thrilling but nevertheless harmful or even deadly?
On the bit-player blog, Brian Hayes writes about computation and mathematics. He also has a self-described “morbid fascination with stories of technological disaster.” He has written about a few disasters that tragically illustrate the results of putting too much trust in technology.
Two Boeing 737 MAX 8 airplane crashes
Earlier this month, Hayes wrote an in-depth post about ongoing investigations of two tragedies involving new Boeing 737 MAX 8 airplanes. The first is the crash of Lion Air Flight 610. That plane departed Jakarta, Indonesia on October 29, 2018. After a flight control problem, that plane crashed into the sea, killing 189 people. The second is the crash of Ethiopian Airlines Flight 302, which departed Addis Ababa (the capital of Ethiopia) on March 8, 2019. Just six minutes after takeoff, the plane crashed, killing 157 people.
“The pilots reported control problems, and data from a satellite tracking service showed sharp fluctuations in altitude. The similarities to the Lion Air crash set off alarm bells: If the same malfunction or design flaw caused both accidents, it might also cause more. Within days, the worldwide fleet of 737 MAX aircraft was grounded. Data recovered since then from the Flight 302 wreckage has reinforced the suspicion that the two accidents are closely related,” Hayes wrote.
Preliminary analysis suggests that problems occurred with the a type of onboard software called the Maneuvering Characteristics Augmentation System, according to Hayes’s post.
Pressure surge and a series of fires and explosions
Last October, Hayes wrote about the role that technology played in the Merrimack Valley gas explosions and fires in Massachusetts on September 13, 2018. “By the end of the day 131 buildings were damaged or destroyed, one person was killed, and more than 20 were injured. Suspicion focused immediately on the natural gas system. It looked like a pressure surge in the pipelines had driven gas into homes where stoves, heaters, and other appliances were not equipped to handle the excess pressure,” Hayes wrote. He added that the National Transportation Safety Board (NTSB) had released a preliminary report supporting that hypothesis.
Hayes wrote about a disturbing finding:
“The cause of the accident was not a leak or an equipment failure or a design flaw or a worker turning the wrong valve. The pressure didn’t just creep up beyond safe limits while no one was paying attention; the pressure was driven up by the automatic control system meant to keep it in bounds. The pressure regulators were “trying” to do the right thing. Sensor readings told them the pressure was falling, and so the controllers took corrective action to keep the gas flowing to customers. But the feedback loop the regulators relied on was not in fact a loop. They were measuring pressure in one pipe and pumping gas into another.”
That safety board subsequently released a safety recommendation report about the product development and review of natural gas distribution systems. For NiSource, Inc., the company whose subsidiary (Columbia Gas of Massachusetts) owned and operated the distribution system involved in the disaster, the report’s recommendations include such measures as applying “management of change process to all changes to adequately identify system threats that could result in a common mode failure” and developing and implementing “control procedures during modifications to gas mains to mitigate the risks identified during management of change operations. Gas main pressures should be continually monitored during these modifications and assets should be placed at critical locations to immediately shut down the system if abnormal operations are detected.”
Cathy O’Neil, who is author of Weapons of Math Destruction, blogger at mathbabe.org and a Bloomberg opinion columnist, has written extensively about algorithms and the issues that accompany them. You can find her posts about algorithms here. (Check out the other posts on this blog about O’Neil and mathbabe.org, along with Anna’s coverage of O’Neil’s MAA-AMS-SIAM Gerald and Judith Porter Public Lecture, “Big data, inequality, and democracy” on the 2019 JMM blog.)