{"id":303917,"date":"2012-04-26T09:00:34","date_gmt":"2012-04-26T16:00:34","guid":{"rendered":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/?p=303917"},"modified":"2016-10-11T14:15:54","modified_gmt":"2016-10-11T21:15:54","slug":"deblurring-images-continents","status":"publish","type":"post","link":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/blog\/deblurring-images-continents\/","title":{"rendered":"Deblurring Images for Continents"},"content":{"rendered":"

By Janie Chang, Writer, Microsoft Research<\/em><\/p>\n

In August 2010, Bing Maps (opens in new tab)<\/span><\/a> rolled out the first images from its Global Ortho program (opens in new tab)<\/span><\/a>, a multiyear mapping project launched that spring to provide the Bing Maps platform and consumer website with consistently high-quality aerial-photo images of the continental United States and western Europe.<\/p>\n

The largest imagery project ever undertaken, the project will be completed in 2012 and will cover about 4 million square miles (10.5 million square kilometers) at a resolution significantly higher than that of commercial imaging satellites.<\/p>\n

\"Neel

Neel Joshi<\/p><\/div>\n

Also in August 2010, a team from Microsoft Research Redmond<\/a> presented a paper called Image Deblurring using Inertial Measurement Sensors<\/em><\/a> during the 37th International Conference and Exhibition on Computer Graphics and Interactive Techniques. The paper received extremely positive reviews. Researcher Neel Joshi<\/a> and his co-authors\u2014Sing Bing Kang<\/a>, principal researcher; Larry Zitnick, senior researcher; and Richard Szeliski, Microsoft distinguished scientist\u2014presented a novel way to correct images by measuring camera motion at the time a photo is taken. The team attached inexpensive gyroscopes and accelerometers to a consumer-quality camera to take inertial-measurement data during camera activity and then removed the blur from photos through algorithms that used the data to reverse the effects of motion from the images.<\/p>\n

Image deblurring remains an open problem of perennial interest to researchers. It\u2019s difficult to solve mathematically because the goal is to recover the photo that would have been taken if the camera had not moved. In theory, if the distortion factor of the photo is known, then it can be applied to the photo to recover the true image.<\/p>\n

Some of the work in this field assumes spatially invariant scenarios, where the same kind of blur applies equally to all parts of the photo. In practice, that hardly ever happens. As a simple example, if the camera rotates slightly, the center of the image will not appear to be blurred significantly, but blurring at the edges will be distinctly noticeable. That requires a spatially variant solution.<\/p>\n

\u201cWhen you look at the actual numbers and the math behind it, it feels like you\u2019re trying to do the impossible,\u201d Joshi says. \u201cYou\u2019re trying to come up with an image that\u2019s not blurred, but you don\u2019t know anything about what caused that blurring.\u201d<\/p>\n

Joshi\u2019s idea was to get more information about how the image became distorted by using sensors on the camera to capture motion data and then applying the result to recover the intended photo.<\/p>\n

\"deblurred

This camera’s images can be deblurred.<\/p><\/div>\n

\u201cWe wanted to get information that was realistically available,\u201d he says, \u201cand accelerometers and gyroscopes are now part of every mobile phone, so we\u2019re using data that we know we can get from real products. We did the experiment by adding gear onto a normal camera mostly because we wanted ready access to the data. Hacking into a phone was not the point of the project.\u201d<\/p>\n

The team was aiming its research at consumer photography, so Joshi was surprised after returning home from the conference to find email asking him to contact Wolfgang Schickler, senior program manager at Bing Imagery Technologies (BIT) in Boulder, Colo.<\/p>\n

Photography at a Different Scale<\/h2>\n

Joshi and Schickler discussed possibilities, but it was not until early 2011, when Schickler visited Redmond and was able to meet face-to-face with Joshi, that discussions about the potential for technology transfer took off.<\/p>\n

The Global Ortho program employs several airplanes mounted with computing equipment and highly efficient digital aerial cameras. The planes fly at the same time, about five miles apart, shooting 220 megapixels every two seconds at 17,000 feet.<\/p>\n

\"motion

The motion-blurred image (top) is sharpened dramatically (bottom).<\/p><\/div>\n

\u201cIf any of the photos don\u2019t meet quality thresholds, they have to get the planes to fly that sequence again,\u201d Joshi says. \u201cThere\u2019s a huge cost each time that happens. If we could help them salvage a little bit through image-deblurring technology, even just 10 percent, it would be a worthwhile cost savings.\u201d<\/p>\n

Schickler was interested in the approach Joshi\u2019s team had taken of using sensor data to remove blur because the equipment in each plane also included a tracking platform that collected data from gyroscopes, altimeters, and other sensors. At the time, this data was being used to determine whether there had been excessive motion or whether the plane was flying at the wrong orientation when the camera was taking an exposure.<\/p>\n

\u201cThis data allowed them to detect when an image would have been blurry,\u201d Joshi explains, \u201cand told them they had to send up the planes again. But until they saw our work, I don’t think they had considered taking that next step to use the data to recover the image.\u201d<\/p>\n

Mapping Data for Maps<\/h2>\n

Joshi was excited when he realized how compatible his research was to the project\u2019s aerial-photography process because BIT was already collecting the data he needed to deblur the aerial images.<\/p>\n

\u201cIt was one of those things that happened organically,\u201d he recalls. \u201cWe had some discussions, they sent some sample images and data, we did some initial exploration, and decided it was a good fit. Then we got started.<\/p>\n

\"aerial

The original aerial image (top) shows distinct improvement once the deblurring is applied (bottom).<\/p><\/div>\n

\u201cI saw pretty quickly that if we could map their data directly into our framework, it would work. The key was to figure out how the coordinate systems BIT used were different from those we had used in our project. Once we understood how to map their coordinates to ours, it turned out to be a pretty smooth process of literally mapping A to B and C to D.\u201d<\/p>\n

Joshi wrote additional code to enable use of the original research code. He was able to generate sample images fairly quickly to show to BIT. That team was impressed with the results and soon informed Joshi that it wanted to make image deblurring part of its aerial-imaging pipeline.<\/p>\n

BIT assigned developers Chintan Shah and Yuxiang Liu to help Joshi package the code to meet the needs of the production process.<\/p>\n

\u201cBIT works with really large images,\u201d Joshi says. \u201cThey take chunks of these images, run them through the deblurring code, then stitch them all back together. The volume of data is amazing. Each individual strip of image data represents miles and miles of actual ground in the real world. Even though they chop up a strip into another hundred blocks to process, each one still represents gigabytes of pixel data.\u201d<\/p>\n

From the Lab to the Real World<\/h2>\n

Joshi has been interested in image deblurring for a long time, even when he was an intern with Microsoft Research Redmond in 2007. When he began working full time at that lab in 2008, this image-deblurring project was the first one he tackled, and he\u2019s thrilled that it has led to his first experience in working with a product group.<\/p>\n

\u201cImage deblurring is a very active research area,\u201d he says. \u201cThe fact that this work is being used in an actual product scenario is pretty cool and a big success, because it is still an open problem.\u201d<\/p>\n

He recalls his excitement when Bing Maps finally put some of \u201chis\u201d images online.<\/p>\n

\u201cI was constantly pinging my contacts in Boulder during the summer of 2011,\u201d he recalls. \u201cThere\u2019s so much processing they have to do before the images go online, not just the deblurring. One day, I got email that said, \u2018OK, go to Bing Maps, and enter these coordinates,\u2019 and I finally got to see an area where the images had been processed using my algorithm.\u201d<\/p>\n

What has been the most interesting thing about this technology transfer?<\/p>\n

\u201cThe fact that when we worked on this paper, it was aimed at consumer photography,\u201d Joshi says. \u201cI never expected technology transfer to an aerial-mapping application. In a sense, the hard part was in connecting the two dots, when someone at BIT who knew about the gyroscopic data they were collecting thought there might be a way to apply our research to their situation. From there on, it was a matter of getting everyone on the same page.\u201d<\/p>\n

Joshi muses that image deblurring is only one of many Microsoft Research tech transfers that have enhanced Microsoft products behind the scenes.<\/p>\n

\u201cThere are many research-to-product transfers that take place behind the scenes at Microsoft that are not apparent to consumers,\u201d he says. \u201cI wish people had more visibility into these collaborations, which take place more often than people might expect, because it makes the product story more interesting.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

By Janie Chang, Writer, Microsoft Research In August 2010, Bing Maps rolled out the first images from its Global Ortho program, a multiyear mapping project launched that spring to provide the Bing Maps platform and consumer website with consistently high-quality aerial-photo images of the continental United States and western Europe. The largest imagery project ever […]<\/p>\n","protected":false},"author":39507,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194480],"tags":[213797,213806,213803,193563,213794,213785,213800,213788,211838,213791],"research-area":[13551],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-303917","post","type-post","status-publish","format-standard","hentry","category-graphics-and-multimedia","tag-accelerometers","tag-aerial-photography","tag-bing-imagery-technologies-bit","tag-bing-maps","tag-camera-motion","tag-global-ortho-program","tag-gyroscopes","tag-high-quality-aerial-photo-images","tag-image-correction","tag-image-deblurring","msr-research-area-graphics-and-multimedia","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199565],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"April 26, 2012","formattedExcerpt":"By Janie Chang, Writer, Microsoft Research In August 2010, Bing Maps rolled out the first images from its Global Ortho program, a multiyear mapping project launched that spring to provide the Bing Maps platform and consumer website with consistently high-quality aerial-photo images of the continental…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/303917","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/comments?post=303917"}],"version-history":[{"count":3,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/303917\/revisions"}],"predecessor-version":[{"id":304130,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/303917\/revisions\/304130"}],"wp:attachment":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/media?parent=303917"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/categories?post=303917"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/tags?post=303917"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=303917"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=303917"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=303917"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=303917"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=303917"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=303917"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=303917"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=303917"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}