While Facebook isn’t exactly a poster child for open source, it’s not quite an orphan either. Facebook like many Web 2.0 sites was built using open source software – and in using that software Facebook has realized that they needed to develop some of their own. Scribe is one such piece of software engineering and it’s now being released outside of Facebook as open source licensed code.
“Scribe is a system for collecting massive amounts of data from a large
number of servers, and we use it for everything from tracking how much
memory a database is using to delivering relationship stories into News
Feed,” Facebook developer Robert Johnson blogged. “Before we wrote scribe, we tried a variety of open source and
proprietary systems, but none of them could keep up with the massive
amount of information our users were generating. So we wrote our own
system, conceived from the start to handle the sorts of problems you
only encounter at such large scale.”
What’s particularly interesting to me about Scribe is the fact that it was built using another open source tool developed by Facebook called Thrift. According to Facebook, Thrift is its a software framework for scalable cross-language services development.That’s a big deal.
Currently Thrift is an incubator project at Apache and it will be interesting to see if/when it gets promoted to full project status and which other vendors might end up participating in the overall Thrift development.
