Does Google feel your pain? Give the search giant points for empathy.
“Imagine the sinking feeling of logging in to your Gmail account and finding it empty.”
So begins a blog post by Ben Treynor, Google’s vice president of engineering, explaining the recent interruption in service that affected a small portion of Gmail users.
Google (NASDAQ: GOOG) pegs the number at 0.02 percent of Gmail users worldwide. While Google hasn’t said how many Gmail users there are worldwide (one estimate pegs the number at 170 million), tens of thousands of Gmail users were likely affected by the service interruption.
“That’s still a lot of people and as we make email a critical communications tool, no outage is acceptable,” Rebecca Wettemann , vice president of research at Nucleus Research, told InternetNews.com. “This incident is a reminder that whatever service or software you’re using, people still need to have a backup and archive strategy.”
Google said no emails were lost in the incident and it’s already restored access to many of those users who lost it. “Though it may take longer than we originally expected, we’re making good progress and things should be back to normal for everyone soon,” Treynor said in his Gmail blog post.
Like other apps in Google’s cloud portfolio, Gmail depends on Google’s highly regarded, widely distributed network of servers that includes multiple layers of redundant backup to greatly reduce the likelihood of any significant disruption.
Update: Google updated the blog post later on Tuesday, indicating it was in the home stretch of restoring all the missing emails.
“Data for the remaining 0.012% of affected users has been successfully restored from tapes and is now being processed. We plan to begin moving data into mailboxes in 2 hours, and in the hours that follow users will regain access to their data. Accounts with more mail will take more time. Thanks for bearing with us.”
A Google spokesperson told InternetNews.com that the company’s expectation is that any remaining issues with missing emails will be completely resolved today.
So what happened?
“Well, in some rare instances software bugs can affect several copies of the data,” Treynor explained in his post. “That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue.”
Google also backs up its servers to tape storage. But while the offline tape backup wasn’t affected by the bugs, Treynor said restoring data from the tapes to another data center takes hours instead of the mere milliseconds an online restore typically entails.
Google also reported that it identified that the source of the bugs was a storage software update that caused the Gmail users to temporarily lose access to their email. Once it identified the problem, Treynor said Google switched back to an older version of the storage software.
“If you were affected by this issue, it’s important to note that email sent to you between 6:00 PM PST on Feb. 27 and 2:00 PM PST on Feb. 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered,” Treynor said.
Google maintains an Apps Status Dashboard that updates the status of all its cloud-based applications. As of 11:30 a.m. Pacific time Tuesday, Gmail was tagged with a “Service disruption” status for Feb. 28, the most recent date listed, while the 12 other services were all marked as having “No Issues.”
David Needle is the West Coast bureau chief at InternetNews.com, the news service of Internet.com, the network for technology professionals.
