[san-tech][03236] Realtime Hadoop usage at Facebook, SIGMOD2011

Date: Fri, 01 Jul 2011 16:18:35 +0900
--------------------------------------------------
Facebook技術者による、Hadoopの Peta-byte scale規模での Realtime Workload対応
についての解説 Blogです (ACM SIGMOD 2011で発表)：

"Realtime Hadoop usage at Facebook -- Part 1"
 May 17, 2011, HDFS, Dhruba Borthakur, Facebook
  http://hadoopblog.blogspot.com/2011/05/realtime-hadoop-usage-at-facebook-part.html
"Realtime Hadoop usage at Facebook -- Part 2 - Workload Types"
 May 28, 2011, HDFS, Dhruba Borthakur, Facebook
  http://hadoopblog.blogspot.com/2011/05/realtime-hadoop-usage-at-facebook-part_28.html

  "Facebook recently deployed Facebook Messages, its first ever
   user-facing application built on the Apache Hadoop platform. It uses
   HDFS and HBase as core technologies for this solution. Since then,
   there are many more applications that have started to used HBase.
   We have gained some experience in deploying and operating HDFS and
   HBase at peta-byte scale for realtime-workloads and decided to
   write a paper detailing some of these insights. This paper will be
   published in SIGMOD 2011." (Part 1)

  "You can find the full paper here later, but here are some highlights:"
※現時点 (2011/07/01) ではこちらのサイトでは公開されていません。

OUR WORKLOADS
  "Before deciding on a particular software stack and whether or not to
   move away from our MySQL-based architecture, we looked at a few specific
   applications where existing solutions may be problematic. These use
   cases would have workloads that are challenging to scale because of
   very high write throughput, massive datasets, unpredictable growth,
   or other patterns that may be difficult or suboptimal in a sharded
   RDBMS environment." (Part 2)


SIGMOD 2011での発表ペーパー
"Apache hadoop goes realtime at Facebook"
 Dhruba Borthakur (facebook), et al.
  http://portal.acm.org/citation.cfm?doid=1989323.1989438
※一部単位の購入が可能です。

Proceedings of the 2011 international conference on Management of data
SIGMOD/PODS 2011
  http://portal.acm.org/citation.cfm?id=1989323

2011 ACM SIGMOD/PODS Conference, June 12-16, 2011
  http://www.sigmod2011.org/index.shtml
※講演資料等は公開されていません。

Keynote 1: James Hamilton, Amazon Web Services
"Internet Scale Storage"
  http://www.sigmod2011.org/keynote_1.shtml
の講演資料は、Hamilton氏がご自身の Blogで公開しています。
"SIGMOD 2011 in Athens"
 2011/06/23, Perspectives, James Hamilton's Blog
  http://perspectives.mvdirona.com/2011/06/23/SIGMOD2011InAthens.aspx


ACM SIGMOD (Special Interest Group on Management of Data)
  http://www.sigmod.org/


直接関係ありませんが、Facebookの社内ツールの紹介：

"Visualizing Facebook's PHP Codebase"
 2011/5/31, Greg Schechter, Software engineer at Facebook
  https://www.facebook.com/note.php?note_id=10150187460703920

  "Visualizations often are a great tool for understanding code
   interdependencies, especially when you're facing the challenge of
   determining interdependencies among more than 10,000 code modules.
   That happens to be about the number of modules that comprise the
   Facebook front-end code."
.....
  "We released this visualization internally for the Facebook
   engineering team to consult and gain insight into module
   granularity, dependency management, and the like. As with all
   Facebook code, this is a work in progress, so we hope to add more
   value to it as it evolves."