Ask a Techspert: What is on-device processing

Every time a new Pixel phone comes out, you might hear that "on-device processing" makes its cool new features possible. Just take a look at the new Pixel 9 phones - things like Pixel Studio and Call Notes run "on device." And it's not just phones: Nest cameras, Pixel smartwatches and Fitbit devices also use this whole "on-device processing" thing. Given the devices that use it and the features it's powering, it sounds pretty important.

It's safe to assume that the, er, processing, is happening on the, uh…well, the device. But to get a better understanding of what that means, we talked to Trystan Upstill, who has been at Google for nearly 20 years working on engineering teams across Android, Google News and Search.

You were on a team that helped develop some of the exciting features that shipped with our new Pixel devices - can you tell me a little about what you worked on?

Most recently, I worked within Android where I led a team that focuses on melding Google's various technology stack into an amazing experience that's meaningful to the user. Then figuring out how to build it and ship it.

Since we're improving technologies and introducing new ones quite often, it seems like that would be a never-ending job.

Exactly! Within recent years, there's been this explosion in generative AI capabilities. At first when we started thinking about running large language models on devices, we thought it was kind of a joke - like, "Sure we can do that, but maybe by 2026." But then we began scoping it out, and the technology performance evolved so quickly that we were able to launch features using Gemini Nano, our on-device model, on Pixel 8 Pro in December 2023.

That's what I want to know more about: "on-device processing." Let's break it down and start with what exactly "processing" means.

The main processor, or system-on-a-chip (SoC), in your devices, has a number of what are called Processing Units designed specifically to handle the tasks you want to do with that device. That's why you'll see the chip (like the Tensor chip found in Pixels) referred to as a "system-on-a-chip: There's not just one processor, but several processing units, memory, interfaces and much more, all together on one piece of silicon.

Let's use Pixel smartphones as an example: The processing units include a Central Processing Unit, or CPU, as the main "engine" of sorts; a Graphics Processing Unit, or GPU, which renders visuals; and now today we have a Tensor Processing Unit, or TPU, specially designed by Google to run AI/ML workloads on a device. These all work together to help your phone get things done - aka, processing.

For example, when you take photos, you're often using all elements of your phone's processing power to good effect. The CPU will be busy running core tasks that control what the phone is doing, the GPU will be helping render what the lens is seeing and, on a premium Android device like a Pixel, there's also a lot of work happening on the TPU to process what the optical lens sees to make your photos look awesome.

Got it. "On-device" processing implies there's off-device. Where is "off-device processing" happening, exactly?

Off-device processing happens in the cloud. Your device connects to the internet and sends your request to servers elsewhere, which perform the task, and then send the output back to your phone. So if we wanted to take that process and make it happen on device, we'd take the large machine learning model that powered that task in the cloud and make it smaller and more efficient so it can run on your device's operating system and hardware.

What hardware makes that possible?

New, more powerful chipsets. For example, with the Pixel 9 Pro, that's happening thanks to our SoC called Tensor G4. Tensor G4 enables these phones to run models like Gemini Nano - it's able to handle these high-performance computations.

So basically, Tensor is designed specifically to run Google AI, which isalsowhat powers a lot of Pixel's new gen AI capabilities.

Right! And the generative AI features are definitely part of it, but there are lots of other things on-device processing makes possible, too. Rendering video, playing games, HDR photo editing, language translation - most everything you do with your phone. These are all happening on your phone, not being sent up to a server for processing.

Public link

Please use the above public link if you want to share this noodl on another website.