{"id":518,"date":"2014-07-02T09:08:00","date_gmt":"2014-07-02T09:08:00","guid":{"rendered":"https:\/\/blogs.technet.microsoft.com\/inside_microsoft_research\/2014\/07\/02\/the-code-that-no-one-in-the-cloud-can-live-without\/"},"modified":"2016-07-20T07:29:58","modified_gmt":"2016-07-20T14:29:58","slug":"the-code-that-no-one-in-the-cloud-can-live-without","status":"publish","type":"post","link":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/blog\/the-code-that-no-one-in-the-cloud-can-live-without\/","title":{"rendered":"The Code That No One in the Cloud Can Live Without"},"content":{"rendered":"

Posted by Rob Knies<\/span><\/p>\n

\"Parikshit (opens in new tab)<\/span><\/a><\/p>\n

A couple of years ago, a few Microsoft researchers published a couple of interesting papers on storage efficiencies. Now, with breathtaking speed, the concepts in those papers have been embraced across the cloud-computing world.<\/p>\n

Technological change can occur at lightning speed. Parikshit Gopalan (opens in new tab)<\/span><\/a>, Cheng Huang (opens in new tab)<\/span><\/a>, and Sergey Yekhanin (opens in new tab)<\/span><\/a> can testify to that.<\/p>\n

In November 2012, Gopalan, Huang, and Yekhanin, along with Huseyin Simitci of Windows Azure Storage (now Microsoft Azure (opens in new tab)<\/span><\/a> Storage), had their paper On the Locality of Codeword Symbols (opens in new tab)<\/span><\/a><\/em>, published in IEEE Transactions on Information Theory<\/em>.<\/p>\n

Erasure-Coding Theory Paper Gains Acclaim<\/h2>\n

During ISIT 2014 (opens in new tab)<\/span><\/a>, the IEEE International Symposium on Information Theory, being held June 29-July 4 in Honolulu, the authors of that paper received the IEEE Communications Society & Information Theory Society Joint Paper Award (opens in new tab)<\/span><\/a>. The honor goes to outstanding papers published in a publication of the Communications Society or the Information Theory Society within the previous three calendar years.<\/p>\n

The winning paper is an in-depth theoretical study of relations between code parameters needed for data-storage applications. Erasure Coding in Windows Azure Storage (opens in new tab)<\/span><\/a><\/em>\u2014an earlier systems paper written by Microsoft\u2019s Huang, Simitci, Yikang Xu, Aaron Ogus, Brad Calder, Gopalan, Jin Li (opens in new tab)<\/span><\/a>, and Yekhanin\u2014pointed the way for a new method to achieve more efficient storage in the cloud. It, too, garnered plenty of attention, winning a best-paper award during the 2012 USENIX Annual Technical Conference (opens in new tab)<\/span><\/a>. The co-authors of the Erasure Coding<\/em> paper also earned a Microsoft Technical Community Network Storage Technical Achievement Award in 2013 for outstanding achievement and contribution to Microsoft software technology.<\/p>\n

Code Requirements<\/h2>\n

\u201cThe project started with Cheng and Jin Li having the idea that Azure might benefit from adopting some new kind of erasure codes,\u201d Yekhanin says. \u201cParikshit and I joined the team. Together, we developed an abstract mathematical framework that captures the requirements for codes that arise in distributed storage applications.<\/p>\n

\u201cThe key differences from classical coding-theory setup is that here, we want codes that provide \u2018locality\u2019\u2014the ability to recover lost data quickly in typical failure scenarios. We designed codes, later adopted by Azure, and also proved that our codes are optimal in a certain strict mathematical sense.\u201d<\/p>\n

Gopalan, who will be traveling to Honolulu to accept the Joint Paper Award, provides a bit more detail.<\/p>\n

\u201cThis is very much a theory paper,\u201d he says. \u201cIts main contribution is a lower bound, saying that any code with certain properties needs to have certain length. But it arose from a very practical setting: We were trying to show that a particular code we had suggested to Azure was optimal.<\/p>\n

\u201cIt is gratifying that a lower bound\u2014a result saying that you cannot do any better than something\u2014should have such practical significance. The results there turn out to have real predictive power in telling us what properties are achievable by code and what are not. It certainly helped in our interactions with product groups to be able to say that our constructions are provably optimal.\u201d<\/p>\n

Local Reconstruction Codes<\/h2>\n

Erasure coding (opens in new tab)<\/span><\/a>, a powerful math tool that reduces the space required to store data, relies on shortened descriptions of data for reassembly and delivery to users. Local Reconstruction Codes (opens in new tab)<\/span><\/a> (LRC) enable quicker data reconstructions, and the result is reduced time and costs for data retrieval.<\/p>\n

Local Reconstruction Codes have been adopted throughout all Microsoft storage production lines, from the cloud to enterprise and the desktop. It was first deployed in Azure Storage in 2012, to great acclaim. In 2013, LRC also shipped with Windows Server 2012 R2<\/a> and Windows 8.1 (opens in new tab)<\/span><\/a>.<\/p>\n

Huang cites great collaboration with partners from Windows storage teams.<\/p>\n

\u201cWe are very lucky to work with fantastic business partners, the Azure Storage team and the Windows Storage Spaces team. LRC wouldn\u2019t have gone anywhere without them taking a leap of faith and making their contributions.\u201d<\/p>\n

At first, erasure coding was a solution for a problem that didn\u2019t exist. But when it eventually did \u2026<\/p>\n

\u201cIt took a long time to bear fruit, from research to production,\u201d Huang says. \u201cWhen we started exploring this direction and published earlier papers in 2007, there was very little interest from business groups. Literally, every team we talked to told us that disks were getting bigger and cheaper every day. Redundancy could be easily achieved with replication, and there was no need to bother with erasure coding.<\/p>\n

\u201cIt is very telling to see how cloud computing has completely turned the world around in several years. Now, the industry is at a point that no one in the cloud business\u2014not only Microsoft, but also Amazon, Facebook, Google, and others\u2014can be competitive or even survive without erasure coding.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Posted by Rob Knies A couple of years ago, a few Microsoft researchers published a couple of interesting papers on storage efficiencies. Now, with breathtaking speed, the concepts in those papers have been embraced across the cloud-computing world. Technological change can occur at lightning speed. Parikshit Gopalan, Cheng Huang, and Sergey Yekhanin can testify to […]<\/p>\n","protected":false},"author":30766,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194470,194476,194488,194460],"tags":[200135,200207,200763,200867,186889,200959,200997,201493,201495,201497,201889,201913,201921,201931,201997,202115,202199,202463,202465,202483,202485,193659,202717,203137,203207,203749,204053,204589,204593,204619,204623,204739],"research-area":[13552,13546,13560],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-518","post","type-post","status-publish","format-standard","hentry","category-computer-architecture","category-devices-and-hardware","category-program-languages-and-software-engineering","category-search-and-information-retrieval","tag-2012-usenix-annual-technical-conference","tag-aaron-ogus","tag-brad-calder","tag-cheng-huang","tag-cloud-computing","tag-code-parameters","tag-communications-society","tag-erasure-code","tag-erasure-coding","tag-erasure-coding-in-windows-azure-storage","tag-huseyin-simitci","tag-ieee-communications-society-information-theory-society-joint-paper-award","tag-ieee-international-symposium-on-information-theory","tag-ieee-transactions-on-information-theory","tag-information-theory-society","tag-isit-2014","tag-jin-li","tag-local-reconstruction-codes","tag-locality","tag-lower-bound","tag-lrc","tag-microsoft-azure","tag-microsoft-azure-storage","tag-on-the-locality-of-codeword-symbols","tag-parikshit-gopalan","tag-sergey-yekhanin","tag-storage","tag-windows-8-1","tag-windows-azure-storage","tag-windows-server-2012-r2","tag-windows-storage-spaces","tag-yikang-xu","msr-research-area-hardware-devices","msr-research-area-computational-sciences-mathematics","msr-research-area-programming-languages-software-engineering","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"July 2, 2014","formattedExcerpt":"Posted by Rob Knies A couple of years ago, a few Microsoft researchers published a couple of interesting papers on storage efficiencies. Now, with breathtaking speed, the concepts in those papers have been embraced across the cloud-computing world. Technological change can occur at lightning speed.…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/518","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/users\/30766"}],"replies":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/comments?post=518"}],"version-history":[{"count":1,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/518\/revisions"}],"predecessor-version":[{"id":235591,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/518\/revisions\/235591"}],"wp:attachment":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/media?parent=518"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/categories?post=518"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/tags?post=518"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=518"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=518"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=518"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=518"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=518"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=518"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=518"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=518"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}