{"id":6652,"date":"2015-11-15T07:43:24","date_gmt":"2015-11-15T07:43:24","guid":{"rendered":"http:\/\/revoscience.com\/en\/?p=6652"},"modified":"2015-11-15T07:43:24","modified_gmt":"2015-11-15T07:43:24","slug":"streamlining-mobile-image-processing","status":"publish","type":"post","link":"https:\/\/www.revoscience.com\/en\/streamlining-mobile-image-processing\/","title":{"rendered":"Streamlining mobile image processing"},"content":{"rendered":"

Technique for mobile image processing in the cloud cuts bandwidth use by more than 98 percent.\u00a0<\/strong><\/em><\/span><\/p>\n

\"Credit:<\/a>
Credit: MIT<\/figcaption><\/figure>\n

CAMBRIDGE, Mass.<\/strong> —\u00a0As smartphones become people\u2019s primary computers and their primary cameras, there is growing demand for mobile versions of image-processing applications.<\/span><\/p>\n

Image processing, however, can be computationally intensive and could quickly drain a cellphone\u2019s battery. Some mobile applications try to solve this problem by sending image files to a central server, which processes the images and sends them back. But with large images, this introduces significant delays and could incur costs for increased data usage.<\/span><\/p>\n

At the Siggraph Asia conference last week, researchers from MIT, Stanford University, and Adobe Systems presented a system that, in experiments, reduced the bandwidth consumed by server-based image processing by as much as 98.5 percent, and the power consumption by as much as 85 percent.<\/span><\/p>\n

The system sends the server a highly compressed version of an image, and the server sends back an even smaller file, which contains simple instructions for modifying the original image.<\/span><\/p>\n

Micha\u00ebl Gharbi, a graduate student in electrical engineering and computer science at MIT and first author on the Siggraph paper, says that the technique could become more useful as image-processing algorithms become more sophisticated.<\/span><\/p>\n

\u201cWe see more and more new algorithms that leverage large databases to take a decision on the pixel,\u201d Gharbi says. \u201cThese kinds of algorithm don\u2019t do a very complex transform if you go to a local scale on the image, but they still require a lot of computation and access to the data. So that\u2019s the kind of operation you would need to do on the cloud.\u201d<\/span><\/p>\n

One example, Gharbi says, is\u00a0recent work<\/span><\/a>\u00a0at MIT that transfers the visual styles of famous portrait photographers to cellphone snapshots. Other researchers, he says, have experimented with algorithms for changing the apparent time of day at which photos were taken.<\/span><\/p>\n

[pullquote]The system performs the desired manipulation of the image \u2014 heightening contrast, shifting the color spectrum, sharpening edges, or the like.[\/pullquote]<\/p>\n

Joining Gharbi on the new paper are his thesis advisor, Fr\u00e9do Durand, a professor of computer science and engineering; YiChang Shih, who received his PhD in electrical engineering and computer science from MIT in March; Gaurav Chaurasia, a former postdoc in Durand\u2019s group who\u2019s now at Disney Research;\u00a0Jonathan<\/span><\/a>\u00a0Ragan-Kelley<\/span><\/a>, who has been a postdoc at Stanford since graduating from MIT in 2014; and Sylvain Paris, who was a postdoc with Durand before joining Adobe.<\/span><\/p>\n

Bring the noise<\/strong><\/span><\/p>\n

The researchers\u2019 system works with any alteration to the style of an image, like the types of \u201cfilters\u201d popular on Instagram. It\u2019s less effective with edits that change the image content \u2014 deleting a figure and then filling in the background, for instance.<\/span><\/p>\n

To save bandwidth while uploading a file, the researchers\u2019 system simply sends it as a very low-quality JPEG, the most common file format for digital images. All the cleverness is in the way the server processes the image.<\/span><\/p>\n

The transmitted JPEG has a much lower resolution than the source image, which could lead to problems. A single reddish pixel in the JPEG, for instance, could stand in for a patch of pixels that in fact depict a subtle texture of red and purple bands. So the first thing the system does is introduce some high-frequency noise into the image, which effectively increases its resolution.<\/span><\/p>\n

That extra resolution is basically meaningless \u2014 just some small, random, local variation of the pixel color in the compressed file. But it prevents the system from relying too heavily on color consistency in particular regions of the image when determining how to characterize its image transformations.<\/span><\/p>\n

Patch work<\/strong><\/span><\/p>\n

Next, the system performs the desired manipulation of the image \u2014 heightening contrast, shifting the color spectrum, sharpening edges, or the like.<\/span><\/p>\n

Then the system breaks the image into chunks \u2014 of, say, 64 by 64 pixels. For each chunk, it uses a machine-learning algorithm to characterize the effects of the manipulation according to a few basic parameters, most of which concern variations in the luminance, or brightness, of the pixels in the patch. The researchers\u2019 best results came when they used about 25 parameters. So for each 64-by-64-pixel patch of the uploaded image, each pixel of which could have one of three values, the server sends back just 25 numbers.<\/span><\/p>\n

The phone then performs the modifications described by those 25 numbers on its local, high-resolution copy of the image. To the naked eye, the results are virtually indistinguishable from direct manipulation of the high-resolution image. The bandwidth consumption, however, is only 1 to 2 percent of what it would have been.<\/span><\/p>\n

Applying the modifications to the original image does require some extra computation on the phone, but that consumes neither as much time nor as much energy as uploading and downloading high-resolution files would. In the researchers\u2019 experiments, the energy savings were generally between 50 and 85 percent, and the time savings between 50 and 70 percent.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

As smartphones become people\u2019s primary computers and their primary cameras, there is growing demand for mobile versions of image-processing applications.<\/p>\n","protected":false},"author":6,"featured_media":6653,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[28],"tags":[],"class_list":["post-6652","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-techbiz"],"featured_image_urls":{"full":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"thumbnail":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0-150x150.jpg",150,150,true],"medium":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0-300x200.jpg",300,200,true],"medium_large":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"large":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"1536x1536":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"2048x2048":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"ultp_layout_landscape_large":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"ultp_layout_landscape":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"ultp_layout_portrait":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",600,400,false],"ultp_layout_square":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",600,400,false],"newspaper-x-single-post":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"newspaper-x-recent-post-big":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",540,360,false],"newspaper-x-recent-post-list-image":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",95,63,false],"web-stories-poster-portrait":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",639,426,false],"web-stories-publisher-logo":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",96,64,false],"web-stories-thumbnail":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/11\/RD_MIT-cellphone-photo_0.jpg",150,100,false]},"author_info":{"info":["Amrita Tuladhar"]},"category_info":"Tech<\/a>","tag_info":"Tech","comment_count":"0","_links":{"self":[{"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/posts\/6652","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/comments?post=6652"}],"version-history":[{"count":0,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/posts\/6652\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/media\/6653"}],"wp:attachment":[{"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/media?parent=6652"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/categories?post=6652"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/tags?post=6652"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}