Facebook's 45,000 employees have been asked to work at home during the pandemic, emptying the offices. Photo / Jason Henry, The New York Times
The social network is straining to deal with skyrocketing usage as its 45,000 employees work from home for the first time.
As the coronavirus spread around the world and people everywhere were ordered to stay home, phone calls over Facebook's apps more than doubled. In many countries, messaging on Instagram and Facebook soared by more than 50 per cent, while group calls in Italy jumped by more than 1,000 per cent. And hungry for information, people clicked repeatedly on virus news stories shown by the social network.
Inside Facebook, that meant the pressure was on.
"We're just trying to keep the lights on over here," Mark Zuckerberg, Facebook's chief executive, said in an interview last week.
READ MORE:
• Covid 19 coronavirus: New Zealanders make last minute scramble for supplies before Covid-19 lockdown
• Covid 19 coronavirus: The Warehouse, liquor stores to close during lockdown
• Covid-19 coronavirus: Essential services include dairies, but only one person in at a time
• Covid-19 coronavirus lockdown: What is an 'essential' service that can stay open?
As airlines, hotels, restaurants and other companies struggle to stay afloat during the pandemic, Facebook is also labouring to cope with the fallout. But unlike those other businesses, the Silicon Valley giant is being strained by the coronavirus in a different way: Its usage is going through the roof.
Skyrocketing traffic and a crush of new users are now stressing Facebook's systems just as its 45,000 employees are dealing with working remotely for the first time. The company is also trying to keep its users' data secure while employees who sift through posts to moderate content do so from home. At the same time, Facebook has added to its workload by promising to do more to limit virus misinformation.
It is a pressure test moment for Facebook, which has for years grappled with a backlash over privacy and toxic content, but now has a chance to change that narrative and be seen as an essential communications and information tool during the outbreak.
"The usage growth from COVID-19 is unprecedented across the industry, and we are experiencing new records in usage almost every day," Alex Schultz and Jay Parikh, two Facebook vice presidents working on infrastructure, said in a blog post on Tuesday. "Maintaining stability throughout these spikes in usage is more challenging than usual now that most of our employees are working from home."
What has saved Facebook's network from crashing altogether, Zuckerberg said, was that the virus and the quarantines have had the largest impact in just a few areas where Facebook operates. Facebook is banned in China, where the virus first appeared, for instance.
• Covid19.govt.nz: The Government's official Covid-19 advisory website
Those areas that have the highest concentration of people using Facebook's services during peak hours from home are also spread out by time zone, Zuckerberg said, which staggers the swell of traffic.
"It really is a big technical challenge," he said. "We're basically trying to ready everything we can." He said Facebook had mobilised its engineers to make sure the company has enough computing capacity and adequate support to handle the surges.
The strain has been compounded by Facebook's workforce adapting to working from home, which had been discouraged in the past.
The company's executives have long preached internally that face-to-face meetings and in-person collaboration were central to Facebook's success. The importance of in-person conversation was so great that employees at offices from Singapore to New York were frequently asked to travel to the company's headquarters in Menlo Park, California, for quarterly meetings.
That has made the transition to working from home especially difficult, said four Facebook employees who spoke on the condition of anonymity because they were not authorized to speak publicly.
To communicate, Facebook employees were told to use BlueJeans, which provides technology for videoconferencing calls, they said. But they quickly found that calls were frozen, or the video quality so bad that it was hard to make out who was speaking. Many employees instead turned to Apple's FaceTime feature, Google video hangouts or Zoom conference calls. Some even built their own version of a video conference call, according to two employees.
Issues quickly piled up.
Two days into working from home, some Facebook managers sent the engineering teams a message: limit idle chat on work message boards.
Facebook employees had been posting on those boards at a record rate, according to one employee. While some workers were sharing tips and best practices of how to set up a home office, others were sharing links to buy heirloom seeds for at-home farming, and instructions on how to sew their own face masks, one employee said.
Other snafus surfaced.
Last week, a bug within Facebook's system began marking thousands of posts by major news outlets like Politico and The Sydney Morning Herald as spam, which resulted in the removal of the posts. It took Facebook a day to correct the mistake, as engineers struggled to communicate remotely with one another over how the bug had been introduced and what it would take to fix it.
While they scrambled, rumours spread among Facebook's users over the source of the bug, with many accusing the company of censoring people's speech. Internally, Facebook managers said that while the bug was routine, the amount of time it took to fix it was not.
We’ve restored all the posts that were incorrectly removed, which included posts on all topics - not just those related to COVID-19. This was an issue with an automated system that removes links to abusive websites, but incorrectly removed a lot of other posts too.
— Guy Rosen (@guyro) March 18, 2020
"This was just a technical error, and we're still doing the post-mortem to understand what happened so we can operationalise any learnings from that," Zuckerberg said last week.
Working from home has also made moderating Facebook's posts more difficult. This month, Facebook put its army of global contractors from outside agencies on paid leave. Those contractors, who number more than 15,000, are responsible for sorting through the posts, images and videos that flow through Facebook's services on a daily basis to weed out sensitive, explicit or hateful material.
As the outbreak spread, contractors were ordered not to come into the office, where they worked on protected networks behind virtual firewalls to maintain user privacy. Many of those contractors do not have the same technology setup at home.
Facebook is still trying to figure out how to let the contractors continue working. For now, it is relying on full-time employees, who do not have the training or the time, to moderate the posts themselves.
Given that, Facebook employees have been asked to remove only the most sensitive, fringe posts, said one employee. The company also told employees that it would rely more heavily on their artificial intelligence systems to flag and remove posts.
"I do think it's reasonable to expect that for some of the other categories where the severity might not be as imminent or extreme, that we may be a little less effective in the near term while we're adjusting to this," Zuckerberg said.
While Facebook's usage is soaring, that may not translate into financial gains. Most of the increase in traffic has occurred on the company's messaging services — like WhatsApp and Messenger — which bring in relatively little revenue. And though more people are using the main News Feed and Facebook Stories in the core app, the company said it wasn't immune to a wider pullback in advertising.
"Our business is being adversely affected like so many others around the world," Schultz and Parikh said in the blog post. "We've seen a weakening in our ads business in countries taking aggressive actions to reduce the spread of COVID-19."
Zuckerberg said Facebook was doing what it could to prepare for the weeks ahead, as it doesn't anticipate the issues to abate anytime soon.
"I've never seen anything like this before," he said.
Written by: Mike Isaac and Sheera Frenkel
Photographs by: Jason Henry, Luke Sharrett, Jessica Chou
© 2020 THE NEW YORK TIMES
The industry faces challenges but hopes to bring newcomers and veterans together.