In the days after the first crash of Boeing's 737 Max, engineers at the Federal Aviation Administration came to a troubling realisation: They didn't fully understand the automated system that helped send the plane into a nose-dive, killing everyone on board.
Engineers at the agency scoured their files for information about the system designed to help avoid stalls. They didn't find much. Regulators had never independently assessed the risks of the dangerous software known as MCAS when they approved the plane in 2017.
More than a dozen current and former employees at the FAA and Boeing who spoke with The New York Times described a broken regulatory process that effectively neutered the oversight authority of the agency.
The regulator had been passing off routine tasks to manufacturers for years, with the goal of freeing up specialists to focus on the most important safety concerns. But on the Max, the regulator handed nearly complete control to Boeing, leaving some key agency officials in the dark about important systems like MCAS, according to the current and former employees.
While the agency's flawed oversight of the Boeing 737 Max has attracted much scrutiny since the first crash in October and a second one in March, a Times investigation revealed previously unreported details about weaknesses in the regulatory process that compromised the safety of the plane.
The company performed its own assessments of the system, which were not stress-tested by the regulator. Turnover at the agency left two relatively inexperienced engineers overseeing Boeing's early work on the system.
The FAA eventually handed over responsibility for approval of MCAS to the manufacturer. After that, Boeing didn't have to share the details of the system with the two agency engineers. They weren't aware of its intricacies, according to two people with knowledge of the matter.
Late in the development of the Max, Boeing decided to expand the use of MCAS, to ensure the plane flew smoothly. The new, riskier version relied on a single sensor and could push down the nose of the plane by a much larger amount.
Boeing did not submit a formal review of MCAS after the overhaul. It wasn't required by FAA rules. An engineering test pilot at the regulator knew about the changes, according to an agency official. But his job was to evaluate the way the plane flew, not to determine the safety of the system.
The agency ultimately certified the jet as safe, required little training for pilots and allowed the plane to keep flying until a second deadly Max crash, less than five months after the first.
The plane remains grounded as regulators await a fix from Boeing. If the ban persists much longer, Boeing said this past week that it could be forced to halt production.
The FAA and Boeing have defended the plane's certification, saying they followed proper procedures and adhered to the highest standards.
"The agency's certification processes are well-established and have consistently produced safe aircraft designs," the regulator said in a statement Friday. "The 737 Max certification program involved 110,000 hours of work on the part of FAA personnel, including flying or supporting 297 test flights."
Boeing said "the FAA's rigor and regulatory leadership has driven ever-increasing levels of safety over the decades," adding that "the 737 Max met the FAA's stringent standards and requirements as it was certified through the FAA's processes."
Federal prosecutors and lawmakers are now investigating whether the regulatory process is fundamentally flawed. As planes become more technologically advanced, the rules, even when they are followed, may not be enough to ensure safety. The new software played a role in both disasters, involving Lion Air and Ethiopian Airlines, which together killed 346 people.
"Did MCAS get the attention it needed? That's one of the things we're looking at," said Chris Hart, former chairman of the National Transportation Safety Board, who is now leading a multiagency task force investigating how the Max was approved. "As it evolved from a less robust system to a more powerful system, were the certifiers aware of the changes?"
Boeing needed the approval process on the Max to go swiftly. Months behind its rival Airbus, the company was racing to finish the plane, a more fuel-efficient version of its best-selling 737.
The regulator's hands-off approach was pivotal. At crucial moments in the Max's development, the agency operated in the background, mainly monitoring Boeing's progress and checking paperwork. The nation's largest aerospace manufacturer, Boeing was treated as a client, with FAA officials making decisions based on the company's deadlines and budget.
It has long been a cosy relationship. Top agency officials have shuffled between the government and the industry.
During the Max certification, senior leaders at the FAA sometimes overruled their own staff members' recommendations after Boeing pushed back. For safety reasons, many agency engineers wanted Boeing to redesign a pair of cables, part of a major system unrelated to MCAS. The company resisted, and FAA managers took Boeing's side, according to internal agency documents.
After the crash of the Lion Air plane in October, FAA engineers were shocked to discover they didn't have a complete analysis of MCAS. The safety review in their files didn't mention that the system could aggressively push down the nose of the plane and trigger repeatedly, making it difficult to regain control of the aircraft, as it did on the doomed Lion Air flight.
Despite their hazy understanding of the system, FAA officials decided against grounding the 737 Max. Instead, they published a notice reminding pilots of existing emergency procedures.
The notice didn't describe how MCAS worked. At the last minute, an FAA manager told agency engineers to remove the only mention of the system, according to internal agency documents and two people with knowledge of the matter. Instead, airlines learned about it from Boeing.
Delegating and deferring
The FAA department that oversaw the Max development had such a singular focus that it was named after the company: The Boeing Aviation Safety Oversight Office.
Many FAA veterans came to see the department, created in 2009, as a symbol of the agency's close relationship with the manufacturer. The top official in Seattle at the time, Ali Bahrami, had a tough time persuading employees to join, according to three current and former employees.
Some engineers believed that Bahrami had installed managers in the office who would defer to Boeing. "He didn't put enough checks and balances in the system," Mike McRae, a former FAA engineer, said of Bahrami. "He really wanted abdication. He didn't want delegation."
Before the certification of the Max began, Bahrami called a group of FAA engineers into his office, the current and former employees said, and asked some of them to join the group. Many didn't want to change jobs, according to a complaint filed by the National Air Traffic Controllers Association, the union representing FAA engineers.
"I got dragged kicking and screaming," said Richard Reed, a former systems engineer at the FAA. Reed said he had just left surgery when agency officials called to ask whether he would work in the office. "I always claimed that I was on drugs when I said 'yes.'"
The FAA said in a statement that Bahrami "dedicated his career to the advancement of aviation safety in both the private and public sectors."
For decades, the FAA relied on engineers inside Boeing to help certify aircraft. But after intense lobbying by industry, the agency adopted rules in 2005 that would give manufacturers like Boeing even more control. Previously, the agency selected the company engineers to work on its behalf; under the new regulations, Boeing could choose them.
Many of the agency's top leaders embraced the approach. It would allow the FAA to certify planes more efficiently and stretch its limited resources. The regulator had also been finding it harder to compete for talented engineers, their government salaries unable to keep up with the going rates in the industry.
For Boeing, the changes meant shedding a layer of bureaucracy. "The process was working well," said Tom Heineman, a retired Boeing engineer who worked on the Max. "The FAA was delegating more of the work and the review and the oversight to the manufacturers than it used to."
But some FAA engineers were concerned that they were no longer able to effectively monitor what was happening inside Boeing. In a PowerPoint presentation to agency managers in 2016, union representatives raised concerns about a "brain drain" and the "inability to hire and retain qualified personnel."
By 2018, the FAA was letting the company certify 96 per cent of its own work, according to an agency official.
Nicole Potter, an FAA propulsion and fuel systems engineer who worked on the Max, said supervisors repeatedly asked her to give up the right to approve safety documents. She often had to fight to keep the work.
"Leadership was targeting a high level of delegation," Potter said. When FAA employees didn't have time to approve a critical document, she said, "managers could delegate it back to Boeing."
It was a process Bahrami championed to lawmakers. After spending more than two decades at the FAA, he left the agency in 2013 and took a job at the Aerospace Industries Association, a trade group that represents Boeing and other manufacturers.
"We urge the FAA to allow maximum use of delegation," Bahrami told Congress in his new lobbying role, arguing it would help American manufacturers compete.
In 2017, Bahrami returned to the FAA as the head of safety.
Internal battle at the FAA
With Boeing taking more control, FAA engineers found they had little power, even when they did raise concerns.
Early on, engineers at the FAA discovered a problem with one of the most important new features of the Max: its engines. The Max, the latest version of the 50-year-old 737, featured more fuel-efficient engines, with a larger fan and a high-pressure turbine. But the bigger, more complex engines could do more damage if they broke apart midair.
The FAA engineers were particularly concerned about pieces hitting the cables that control the rudder, according to five people with knowledge of the matter and internal agency documents. A cable severed during takeoff would make it difficult for pilots to regain control, potentially bringing down the jet.
The FAA engineers suggested a couple solutions, three of the people said. The company could add a second set of cables or install a computerised system for controlling the rudder.
Boeing did not want to make a change, according to internal FAA documents reviewed by The Times. A redesign could have caused delays. Company engineers argued that it was unlikely that an engine would break apart and shrapnel would hit the rudder cable.
Most of the FAA engineers working on the issue insisted the change was necessary for safety reasons, according to internal agency emails and documents. But their supervisors balked. In a July 2015 meeting, Jeff Duven, who replaced Bahrami as the head of the FAA's Seattle operation, sided with Boeing, said two current employees at the agency.
FAA managers conceded that the Max "does not meet" agency guidelines "for protecting flight controls," according to an agency document. But in another document, they added that they had to consider whether any requested changes would interfere with Boeing's timeline. The managers wrote that it would be "impractical at this late point in the program," for the company to resolve the issue. Duven at the FAA also said the decision was based on the safety record of the plane.
Engineers at the agency were demoralised, the two agency employees said. One engineer submitted an anonymous complaint to an internal FAA safety board, which was reviewed by The Times.
"During meetings regarding this issue the cost to Boeing to upgrade the design was discussed," the engineer wrote. "The comment was made that there may be better places for Boeing to spend their safety dollars."
An FAA panel investigated the complaint. It found managers siding with Boeing had created "an environment of mistrust that hampers the ability of the agency to work effectively," the panel said in a 2017 report, which was reviewed by The Times. The panel cautioned against allowing Boeing to handle this kind of approval, saying "the company has a vested interest in minimising costs and schedule impact."
By then, the panel's findings were moot. Managers at the agency had already given Boeing the right to approve the cables, and they were installed on the Max.
Playing down risks
In the middle of the Max's development, two of the most seasoned engineers in the FAA's Boeing office left.
The engineers, who had a combined 50 years of experience, had joined the office at its creation, taking on responsibility for flight control systems, including MCAS. But they both grew frustrated with the work, which they saw as mostly paper pushing, according to two people with knowledge of the staff changes.
In their place, the FAA appointed an engineer who had little experience in flight controls, and a new hire who had gotten his master's degree three years earlier. People who worked with the two engineers said they seemed ill-equipped to identify any problems in a complex system like MCAS.
And Boeing played down the importance of MCAS from the outset. An early review by the company didn't consider the system risky, and it didn't prompt additional scrutiny from the FAA engineers, according to two agency officials. The review described a system that would activate only in rare situations, when a plane was making a sharp turn at high speeds.
The FAA engineers who had been overseeing MCAS never received another safety assessment. As Boeing raced to finish the Max in 2016, agency managers gave the company the power to approve a batch of safety assessments — some of the most important documents in any certification. They believed the issues were low risk.
One of the managers, Julie Alger, delegated the review of MCAS. Previously, the FAA had the final say over the system.
The FAA said that decision reflected the consensus of the team.
Boeing was in the middle of overhauling MCAS. To help pilots control the plane and avoid a stall, the company allowed MCAS to trigger at low speeds, rather than just at high speeds. The overhauled version would move the stabiliser by as much as 2.5 degrees each time it triggered, significantly pushing down the nose of the plane. The earlier version moved the stabiliser 0.6 degrees.
When company engineers analyzed the change, they figured the system had not become any riskier, according to two people familiar with Boeing's discussions on the matter. They assumed that pilots would respond to a malfunction in three seconds, quickly bringing the nose of the plane back up. In their view, any problems would be less dangerous at low speeds.
So the company never submitted an updated safety assessment of those changes to the agency. In several briefings in 2016, an FAA test pilot learned the details of the system from Boeing. But the two FAA engineers didn't understand that MCAS could move the tail as much as 2.5 degrees, according to two people familiar with their thinking.
Under the impression the system was insignificant, officials didn't require Boeing to tell pilots about MCAS. When the company asked to remove mention of MCAS from the pilot's manual, the agency agreed. The FAA also did not mention the software in 30 pages of detailed descriptions noting differences between the Max and the previous iteration of the 737.
Days after the Lion Air crash, the agency invited Boeing executives to the FAA's Seattle headquarters, according to two people with knowledge of the matter. The officials sat incredulous as Boeing executives explained details about the system that they didn't know.
In the middle of the conversation, an FAA employee, one of the people said, interrupted to ask a question on the minds of several agency engineers: Why hadn't Boeing updated the safety analysis of a system that had become so dangerous?
Written by: Natalie Kitroeff, David Gelles and Jack Nicas
Photographs by: Ruth Fremson
© 2019 THE NEW YORK TIMES