[san-tech][03466] "The Magellan Report on Cloud Computing for Science", Dec 2011 (Re: Magellan, US DOE Cloud Project)

Date: Wed, 07 Mar 2012 20:32:04 +0900
--------------------------------------------------
2012/04/10
[san-tech][03517] Re: "The Magellan Report on Cloud Computing for Science", Dec 2011 (Re: Magellan, US DOE Cloud Project)
  Date: Tue, 10 Apr 2012 15:59:25 +0900
--------------------------------------------------
ちょっと前の公開ですが、Magellan Projectの最終レポートが公開されました：

"Can Cloud Computing Address the Scientific Computing Requirements
 for DOE Researchers? Well, Yes, No and Maybe Can Cloud Computing
 Address the Scientific Computing Requirements for DOE Researchers?
 Well, Yes, No and Maybe"
 January 30, 2012
  http://www.nersc.gov/news-publications/news/nersc-center-news/2012/can-cloud-computing-address-the-scientific-computing-requirements-for-doe-researchers-well-yes-no-and-maybe/

  "After a two-year study of the feasibility of cloud computing
   systems for meeting the ever-increasing computational needs of
   scientists, Department of Energy researchers have issued
   a report stating that the cloud computing model is useful,
   but should not replace the centralized supercomputing centers
   operated by DOE national laboratories."
.....
  "To thoroughly evaluate the cloud computing model, the team ran
   scientific applications for studying particle physics, climate,
   quantum chemistry, plasma physics and astrophysics on
   the Magellan system, as well as on a Cray XT4 supercomputer and
   a Dell cluster system. The applications were also run on Amazon's
   EC2 commercial cloud offering, including the HPC offering with
   Cluster Compute instances, for comparison purposes."
.....
"In the end, the Magellan research teams came up with a set of
 key findings:

*)Scientific applications have special requirements that require
  cloud solutions that are tailored to these needs.
*)The scientific applications currently best suited for clouds are
  those with minimal communication and I/O (input/output).
*)Clouds can require significant programming and system administration
  support.
*)Significant gaps and challenges exist in current open-source
  virtualized cloud software stacks for production science use.
*)Clouds expose a different risk model, requiring different security
  practices and policies.
*)The MapReduce programming model shows promise in addressing
  scientific needs, but current implementations have gaps and challenges.
*)Public clouds can be more expensive than in-house large systems.
  Many of the cost benefits from clouds result from the increased
  consolidation and higher average utilization.
*)DOE supercomputing centers already achieve energy efficiency levels
  comparable to commercial cloud centers.
*)Cloud is a business model and can be applied at DOE supercomputing
  centers.


"The Magellan Report on Cloud Computing for Science"
 U.S. Department of Energy
 Office of Advanced Scientific Computing Research (ASCR)
 December, 2011 (170 pages)
  http://www.nersc.gov/assets/StaffPublications/2012/MagellanFinalReport.pdf

Key Findings
".. Magellan was charged with answering the following research questions:"
 
*)Are the open source cloud software stacks ready for DOE HPC science?
*)Can DOE cyber security requirements be met within a cloud?
*)Are the new cloud programming models useful for scientific computing?
*)Can DOE HPC applications run efficiently in the cloud? What applications
   are suitable for clouds?
*)How usable are cloud environments for scientific applications?
*)When is it cost effective to run DOE HPC science in a cloud?

==================================================
On Sat, 17 Apr 2010 03:59:46 +0900

> Magellanの実機写真付き解説文です：
>
> "MAGELLAN: Exploring CLOUD Computing for DOE's Scientific Mission"
>  SciDAC Review17, SPRING 2010
>   http://www.scidacreview.org/1002/html/hardware.html
>
>   "A new $32 million program funded by the American Recovery and
>    Reinvestment Act through the U.S. Department of Energy (DOE) will
>    examine cloud computing as a cost-effective and energy-efficient
>    computing paradigm for mid-range science users to accelerate
>    discoveries in a variety of disciplines, including analysis of
>    scientific datasets in biology, climate change, and physics."
>
> Voltaireの巨大 InfiBand Directorと DDNの特徴あるラックが目立ちます。
>
>
> Magellan Hardware
>   http://www.scidacreview.org/1002/html/hardware1.html
>
>   "the system will offer high density with front-access cabling
>    and will be liquid-cooled using rear-door heat exchangers"
> ※液冷背面ドアを採用しています (多分 IBM製)。
>
>
> Data Storage and Networking
>   "The NERSC Global File (NGF) system will provide most storage needs
>    for projects running on the NERSC portion of the Magellan system.
>    Approximately 1 PB of storage and 25 gigabits per second (Gbps) of
>    bandwidth have been added to support use by the test bed."
>   "NERSC .... as well as experiment with Flash memory technology to
>    provide fast random access storage for some of the more data-intensive
>    problems. Approximately 10 TB will be deployed in NGF for high-bandwidth,
>    low-latency storage class and metadata acceleration. Around 16 TB will
>    be deployed as local SSD in one SU for data analytics, local read-only
>    data and local temporary storage. Approximately 2 TB will be deployed
>    in HPSS.
>
>   "ALCF will provide active storage, using HADOOP over PVFS, on
>    approximately 100 compute/storage nodes. This active storage will
>    increase the capacity of the ALCF Magellan system by approximately 30 TF
>    of compute power, along with approximately 500 TB of local disk storage
>    and 10 TB of local SSD."
>
>
> ALCF研究者の Hadoopと PVFSのレポートをこの前見つけたけど、今すぐには出て
> きませんと、思ったら UCSCの学生と Lawrence Livermore と Los Alamosの方々
> でした：
> "Mixing Hadoop and HPC Workloads on Parallel Filesystems"
>  Esteban Molina-Estolano, UCSC, et. al.
>  4th Petascale Data Storage Workshop Supercomputing '09
>   http://www.pdsi-scidac.org/events/PDSW09/resources/pdsw09-final5.pdf
>   http://www.pdsi-scidac.org/events/PDSW09/resources/PDSW%2009%20Slides-molina-estolano.pdf
>
> 4th Petascale Data Storage Workshop Supercomputing '09
>  November 15, 2009
>   http://www.pdsi-scidac.org/events/PDSW09/
>
>
> で、こんなの発見
> Magellan 2010 Workshop, March 23, 2010
>   http://workshops.alcf.anl.gov/magellan10/
> Agenda/Presentations
>   http://workshops.alcf.anl.gov/magellan10/agenda/
>
> Magellan
>   http://magellan.alcf.anl.gov
> Cloud Research
>   http://magellan.alcf.anl.gov/cloud-research/
>
>
> Magellanは ALCF (Argonne Leadership Computing Facility)と NERSCの
> 二ヶ所に設置されます。今日は ANLだけで。

==================================================
> On Fri, 16 Oct 2009 20:30:07 +0100

> >
> > 米エネルギー省の Cloudプロジェクトです：
> > "NATIONAL IMPACT SERIES: SCIENTISTS LOOK TO THE CLOUDS
> >  TO SOLVE COMPLEX QUESTIONS", October 14, 2009
> >   http://www.science.doe.gov/News_Information/News_Room/2009/Oct%2014_ComplexQuestions.html
> >
> > Advanced Scientific Computing Research (ASCR). DOE
> >   http://www.sc.doe.gov/ascr/index.html
> > の見出しの方が短くまとまっています：
> > "DOE is launching a $32 million program to study how scientific codes
> >  can make use of cloud computing technology.  The program, called
> >  Magellan, is funded by the American Recovery and Reinvestment Act
> >  (ARRA), with the money to be split between the Argonne Leadership
> >  Computing Facility (ALCF) and the National Energy Research Scientific
> >  Computing Center (NERSC)."
> >
> >
> > Argonne National Laboratory,
> > NERSC, Lawrence Berkeley National Laboratoryのプレスリリース：
> > "DOE to explore scientific cloud computing at Argonne, Lawrence
> >  Berkeley National Laboratories", October 14, 2009
> >   http://www.anl.gov/Media_Center/News/2009/news091014a.html
> >   http://www.anl.gov/Media_Center/News/2009/news091014a.pdf
> >   http://newscenter.lbl.gov/press-releases/2009/10/14/scientific-cloud-computing/
> > "DOE is funding the project at $32 million, with the money divided
> >  equally between Argonne National Laboratory and Lawrence Berkeley
> >  National Laboratory, where NERSC is located."
> >
> > "In addition, Magellan will provide data storage resources that will
> >  be used to address the challenge of analyzing the massive amounts of
> >  data being produced by scientific instruments ranging from powerful
> >  telescopes photographing the universe to gene sequencers unraveling
> >  the genetic code of life. NERSC will make the Magellan storage
> >  available to science communities using a set of servers and software
> >  called "Science Gateways," as well as experiment with Flash memory
> >  technology to provide fast random access storage for some of the
> >  more data-intensive problems."
> >
> > "The NERSC and ALCF facilities will be linked by a groundbreaking
> >  100 gigabit-per-second network, developed by DOE's ESnet (another
> >  DOE initiative funded by the Recovery Act)."
> > 　[san-tech][01795] 米ネットワーク研究開発投資：$62M ESnet
> >
> >
> > 関連記事
> > "DOE Labs to Build Science Clouds", October 13, 2009
> >   http://www.hpcwire.com/features/DOE-Labs-to-Build-Science-Clouds-64189872.html
> > "What kind of cloud computing project would you build with $32M?"
> >  10/14/2009
> >   http://www.networkworld.com/news/2009/101409-layer8-cloud-computing-doe.html
> > "US boffins use Obama dough to study clouds", 16th October 2009
> >   http://www.theregister.co.uk/2009/10/16/doe_funds_cloud_research/
> >
> >
> > ちょっと関連：
> > "U.S. government to slow data center build-out by leveraging more
> >  cloud services", 9/15/2009
> >   http://www.datacenterdynamics.com/ME2/dirmod.asp?sid=&nm=&type=news&mod=News&mid=9A02E3B96F2A415ABC72CB5F516B4C10&tier=3&nid=B7906333B7B246DCB0BDC5CBACFBED73
> > [san-tech][01790] US Federal Cloud Computing Initiative
> >
> > こちらはいろいろ報道されたと思います：
> > NASA Nebula - Cloud Computing
> >   http://nebula.nasa.gov/
> >
> > "Executive Order 13514 Signed on Federal Leadership in Environmental,
> >  Energy, and Economic Performance", October 5, 2009
> >   http://www1.eere.energy.gov/femp/news/news_detail.html?news_id=15528
> > "Within 90 days, agencies are required to set a target for reduction
> >  of greenhouse gas emissions by the year 2020 and to recycle or
> >  otherwise divert 50% of their waste by 2015."
> > "Obama Orders Federal IT To Get Greener", 10, 6, 2009
> >   http://www.informationweek.com/news/government/policy/showArticle.jhtml?articleID=220301302
> > "Obama Issues Executive Order on Federal Energy Efficiency",
> >  October 13, 2009
> >   http://blog.enterpriseitplanet.com/green/blog/2009/10/obama-issues-ex.html
> >
> > これがそれ：
> > Executive Order 13514--
> > Federal Leadership in Environmental, Energy, and Economic Performance
> > October 8, 2009
> >   http://www.whitehouse.gov/assets/documents/2009fedleader_eo_rel.pdf
> >
> >
> > 一方日本では・・・
> > http://itpro.nikkeibp.co.jp/article/NEWS/20091016/338944/
> > http://mainichi.jp/select/biz/news/20091017k0000m010029000c.html
> > http://www.mext.go.jp/a_menu/shinkou/hojyo/1285813.htm