Years before two Boeing 737 Max jets crashed in Indonesia and Ethiopia, U.S. regulators found a pattern of recurring safety problems with the manufacturing giant.
During a trip to Japan in 2015, an auditor with the Federal Aviation Administration discovered a Boeing subcontractor was falsifying certifications on cargo doors for hundreds of 777s and had been doing so for years, according to interviews and government documents.
Back in the United States, Boeing mechanics were leaving tools inside plane wings, precariously close to the cables that control their movements. Workers also were improperly installing wires in 787s, which could increase the risk of shorts or fires, FAA officials found.
Repeatedly, safety lapses were identified, and Boeing would agree to fix them, then fail to do so, the FAA said. The agency launched or was considering more than a dozen legal enforcement cases against the company for failing to comply with safety regulations, a review of FAA records shows, with fines that could have totalled tens of millions of dollars.
So FAA officials tried a new approach. Rather than pursue each violation separately, agency officials bundled them together and negotiated a broader deal.
"The thinking was, get everything wrapped into one case since we're trying to address a bunch of broader systemic issues anyway," said an FAA official who spoke on the condition of anonymity to discuss internal deliberations.
As Boeing faces intense scrutiny over back-to-back crashes of its 737 Max jet, documents and interviews show that the company had safety problems known to federal regulators for years.
In 2015, the FAA decided to try to get Boeing to meet, then go beyond, federal safety requirements by addressing broader corporate culture and governance issues, including what agency officials considered a lack of transparency.
The week before Christmas of that year, Boeing and the FAA signed a five-year settlement agreement that was unprecedented in scope. The company paid a modest $12 million penalty, but it agreed to make significant changes in its internal safety systems and practices for "ensuring compliance" with regulations.
In the days after the agreement was signed, top US officials cast it as a powerful reminder that every company, no matter its size, must comply with minimum safety standards.
Boeing's profits after signing the deal topped $20 billion by the end of September 2018, making the company's $12 million penalty easy to gloss over despite occasional press reports of the firm's shortcomings.
Then a Lion Air 737 Max plunged into the Java Sea on Oct. 29. It was the first of two tragedies that led many to question the soundness of the Boeing aircraft and the company's approach to safety.
And it injected the largely overlooked government settlement with a new urgency - as a means of tracking company failures identified by the FAA, illuminating Boeing's relationship with its regulators, and potentially forcing new safety improvements as investigators probe what was behind the two Max crashes.
Among its commitments under the deal, Boeing agreed to greatly expand its use of an internal tool meant to help determine the root causes of its safety problems and make sure they get fixed and stay that way.
In response to FAA findings that Boeing was often late or incomplete with its required safety submissions, the company also agreed in 2015 to take various remedial steps and be more responsive.
But over the first 3½ of the agreement, Boeing failed to meet some of its obligations, according to two people who requested anonymity to discuss details of the settlement.
Boeing says it has taken major steps to comply. The FAA, meanwhile, last year chose not to invoke enforcement provisions that could have meant $12 million in additional penalties for the company. The agreement runs through Dec. 31, 2020, and there is the potential for more financial penalties if Boeing fails to meet the requirements.
As investigations continue into Boeing and the FAA over the safety certification of the 737 Max and the two crashes that killed 346 people, regulators and others are scrutinising whether there are echoes of the persistent problems that prompted the 2015 agreement.
An FAA official said it is too early to establish a connection between the probes and specific concerns addressed in the settlement. But the official, who asked not to be named discussing ongoing matters with Boeing, said that "if we learn something, we'll obviously go back and address that."
Boeing's inability to rid its newly built planes of what it calls "Foreign Object Debris," such as tools left behind, was one of the problems that prompted the settlement. But an FAA official said the company is still struggling with the issue, and a top Air Force official told Congress the government temporarily halted deliveries of Boeing tankers earlier this year over "FOD" problems.
The company committed to improving the quality and timeliness of information it provides to the FAA. But in the case of the 737 Max, the FAA said, it took Boeing more than a year to notify it about a software problem that disabled a crucial warning light connected to the automated system at the center of the tragedies.
Most of the cases that sparked the settlement talks involved "apparent failures of corrective action," according to the agreement - meaning Boeing wasn't implementing promised fixes or problems were reoccurring.
For example, before the 2015 agreement, Boeing workers kept failing to insert "lock wires" into holes in bolts used to build planes, according to the FAA. The wires are akin to the twist-ties of industrial manufacturing, and they serve as a backup so bolts or other fasteners can't come loose, allowing critical parts to detach in the rumble of flight. Whether because of problems with training, sloppiness or employees rushing to meet commercial deadlines, the FAA found Boeing failed to sufficiently correct the problem in the period before the settlement.
In addition, special decompression panels - meant to prevent a sudden change in pressure from collapsing the floor beneath passengers - had been improperly installed, and "corrective actions" weren't sufficient.
A source of contention has been whether Boeing has met its commitment to use its own "Boeing Problem Solving Model" to examine the causes of an array of problems and come up with systemic solutions, rather than simply dealing with the symptoms.
"The problem-solving methodology is really important to . . . alleviate the chance for those kinds of events to occur," said one of the FAA officials familiar with the settlement. "Absent doing some of this, you can go off and resolve each issue specifically, but we're trying to push Boeing to that level beyond that."
The company also agreed to provide "on-demand reports" covering "any documentation relating to" Boeing's work implementing the agreement. But a person familiar with the settlement said Boeing has demonstrated "some resistance" to providing that information.
Boeing did not answer questions about how many times the problem-solving model has been used or whether it had been slow to meet its commitments under the agreement.
The FAA said in a statement that it "has not yet assessed any deferred penalties" against the company but "continues to closely monitor and evaluate Boeing's performance."
It added: "When we have had a concern, we have raised it with Boeing for resolution consistent with the agreement. As this settlement remains open, we cannot discuss our current evaluation and potential appropriate actions we may take based on Boeing's performance of the agreement."
Boeing said it "has worked with the FAA and invested significant resources to implement these improvements, further enhancing Boeing's quality and compliance systems" and is "fully committed to meeting the rigorous standards" in the agreement.
"This is precisely how the system is supposed to work: rigorous oversight by the FAA, coupled with significant Boeing investment in process, systems, and people, leading to continuous improvement in safety, quality, and compliance," Boeing said in a statement.
The settlement agreement grew out of long-running concerns among some at the FAA that Boeing was not complying with federal safety standards.
Critics say that because of the firm's importance to the U.S. economy, its confidence in its own technical expertise and its deep ties in Washington, Boeing has a history of shaping the rules it follows - and slow-rolling legal mandates it doesn't consider a priority.
Boeing's delays turned into a major FAA concern after a fuel tank exploded in a Boeing 747 in 1996 and TWA Flight 800 crashed off Long Island, killing 230 people.
In 2008, the FAA gave Boeing and other companies more than two years to provide airlines with the technical information they needed to comply with a post-TWA regulation intended to make fuel tanks less flammable.
Boeing missed the deadline by more than 300 days, and the FAA took legal action in response to Boeing's "tardiness."
That case helped lead to the settlement, as did missed deadlines elsewhere, including for sending airlines data needed for their inspections to make sure structures didn't crack, according to the FAA.
Some at the FAA say Boeing was testing agency rules in other important ways that also have resonance today, citing the Organization Designation Authorization (ODA) program as an example.
The program delegates power to a unit inside Boeing that is supposed to do much of the detailed, technical work involved with finding whether the company is complying with minimum safety standards. The Boeing employees are supposed to act independently of Boeing's corporate interests in judging whether an airplane's designs meet those standards.
But when there were disagreements over whether safety standards were being met, top Boeing employees who were supposed to be representing the FAA at times aggressively championed the company's interests instead, even traveling to Washington to make their case.
Agency leaders, with backing in Congress, have pushed for greater delegation of safety responsibilities to Boeing. Some FAA officials have warned that the agency has been too deferential, and auditors have repeatedly found that the FAA has fallen short in overseeing its own delegation program.
The 2015 agreement said Boeing's top two ODA employees "will not advocate" for the company on specific compliance issues, to avoid potential conflicts. But a recent case under investigation in South Carolina indicates similar conflicts may have continued.
In April, the FAA investigated after a Boeing inspector, working as part of the ODA program in South Carolina, complained about facing undue pressure from company management. The inspector found Boeing failed to comply with a safety regulation and alleged that a manager from outside the ODA unit interfered with the inspector's work.
Earlier this month, the FAA notified Boeing in a letter of investigation that the inspector had indeed faced management interference. The FAA said procedures meant to ensure inspectors remained independent weren't being followed or were being subverted by a dual reporting structure that left inspectors subject to more than one boss.
Boeing declined to comment.
The highly delegated certification process has come under increased scrutiny since the Lion Air and Ethiopian Airlines 737 Max crashes, with questions being raised about how the anti-stall system investigators say was a factor in both crashes was certified as safe.
The Justice Department's criminal division is looking into the 737 Max, and a congressional committee, the Transportation Department's Office of Inspector General and several internal and external panels are investigating the FAA's approval of the plane and the agency's broader certification system.
Plane lost at sea
After a cargo door on a United Airlines 747 broke off near Honolulu in 1989, the Boeing jet faced an "explosive decompression," and nine passengers were "ejected from the airplane and lost at sea," according to the National Transportation Safety Board.
A problem in the door control system - a bad switch or wiring - allowed the locked door to become unlatched and blow open. "Also contributing to the accident was a lack of timely corrective actions by Boeing and the FAA following a 1987 cargo door-opening incident on a Pan Am B-747," the NTSB said.
So FAA officials were alarmed to discover in 2015 that a mechanic at a bus and airplane parts maker north of Nagoya, Japan, Iwado Industry Co., had falsified certifications on hundreds of cargo doors.
Iwado was acting as a supplier to Kawasaki Heavy Industries, which in turn provided parts directly to Boeing.
For years, the Iwado mechanic had been working on part of the door's mechanical system without the correct tool and had been falsely recording that the work had been done properly, according to the FAA. The submissions were also certified as correct by a second layer of quality-control workers.
Safety experts say that type of breach calls into question the trustworthiness of operations.
"It's disturbing that people might think that they can cut corners and lie, if that's happening, when in fact the consequences of those falsehoods could be severe," said Arnold Barnett, an aviation safety researcher and professor of statistics and management at MIT.
Boeing said all the large 777 doors were "re-measured" by both suppliers after the FAA's discovery and "no non-conformances were found."
After a follow-up FAA visit, "all parts were confirmed to be safe and compliant," Boeing said. The mechanic was retrained the day the issue was discovered, and improvements were made to the training curriculum, Boeing said in a statement.
But the episode raised questions about Boeing's oversight of its global network of suppliers and the suppliers' internal inspection regimens.
Kawasaki Heavy Industries said in a statement that "Boeing was the primary examinee" of the FAA audit, so it and Iwado were not in a position to answer questions.
"KHI investigated to confirm that no other similar cases had occurred . . . and took the necessary corrective actions, " which were later verified by the FAA, the statement said.
Iwado did not respond to requests for comment.
The mechanic was one of a number of such examples, according to the FAA.
Boeing's own employees in Renton and Everett, Washington, and elsewhere also recorded false information for certain planes, including on a 787 that was sold to Air Canada and later experienced a fuel leak, the FAA found. A Boeing mechanic and inspector involved in that case faced unspecified "corrective action," the company said.
Boeing audits "concluded that this was an isolated event," the company said in its statement, adding that it launched long-term fixes, "including formal training and communication on personal accountability in the manufacturing process, stressing the importance of complying with all regulatory requirements."
The Seattle Times in 2017 first reported on many of the cases that led to the settlement.
Beyond the issue of providing accurate information, the settlement also describes specific benchmarks Boeing must meet to improve the quality of its safety submissions and hit deadlines.
The goal is to "make sure there's clarity about what right looks like and expect that the first time and avoid our use of resources reworking things to get it to be acceptable," the FAA official said.
A Boeing spokesman did not answer questions about its progress in those and other specific areas. But a person familiar with the company's activities said it made a number of improvements, including to some specifications used by its mechanics in building planes that the FAA said could be confusing. It also audited a sample of its suppliers and found that the cases of "improper acceptance" of parts addressed in the settlement were not typical.
"You've got to give them credit for what they do," an FAA official said.
In the months since the dual disasters overseas, acting FAA administrator Daniel Elwell has repeatedly pointed to the remarkable safety record of the U.S. commercial aviation system, crediting the rigour and shared purpose of government and industry.
"One fatality in 10 years. I'll say it again: 90 million flights, 7 billion passengers. We've had one fatality," Elwell said after a Texas meeting last month with international aviation authorities on the FAA's response to the Max.
Barnett, of MIT, said those numbers represent a "remarkable story of success."
"Of course, everything looks a little different in light of the Max," he said.
In a recent study, Barnett found that about 1 in 8 million passengers worldwide died on a scheduled commercial flight in the decade ending in 2017. But for passengers who flew on the Max, Barnett estimated the risk was "at least 20 times that," or roughly 1 in every 300,000 passengers.
- Washington Post