Qualcomm is announcing three new camera/sensor bundles and a new Spectra ISP to rule them all. Sure SemiAccurate likes to abuse quotes but a ground up new ISP will run all three bundles even without bad analogies.
Today the Qualcomm camera modules have a name, Saur… Spectra Module Program and it is a really good idea. Take a camera module and add a second camera, depth sensor, or something else, and have Qualcomm do all the tuning for it. Bake that tuning into a binary file that the Snapdragon ISP and other relevant hardware understands and you have something close to an actual plug and play camera module.
If this seems rather pointless, it isn’t. The fine tuning of a camera is hard work and takes time, something between a black art and science. I am sure you can recall a few stories about the headaches of various high-end phone companies who were a bit off with camera tuning on new models. Blueish tinged photos ring a bell? You may have had a phone sporting a really high quality sensor that took images best described as ‘meh’. This isn’t hardware, it is tuning.
So what Qualcomm is doing is making sure the pieces all work right together. Not just right but optimally, and they spend the time and effort it takes to do it as right as possible. That said they don’t actually make or sell the end result, the modules are made by OEMs and to other OEM/ODMs to put into their new smartphones. Why? Time to market and development cost, both of paramount import to any phone OEM whose margins are rapidly declining. Qualcomm gets a value add to their Snapdragon line, they may not get money directly but if an OEM can save $1 in camera development costs it effectively makes the SoC $1 cheaper. Modules are a win/win.
The first foray into the camera module market came last year with the Snapdragon 835. It had a color camera and a monochrome one which allowed for much sharper images in bad conditions when the images were computationally combined. In the end there were three modules for the 835 generation, a Sony 16MP camera, the aforementioned color and monochrome duo, and a wide-angle camera with a second 2x optical zoom camera. Qualcomm promised more of the same and with the Spectra Module Program we now have three new variants. And a new name.
Those new modules are an always on optical security module, a passive depth module, and an active depth module. The security module is a front-facing dual camera which pairs a normal front color camera with a 2MP 10b IR camera tightly coupled to Qualcomm’s security infrastructure. The underlying software has much more than image recognition features, there are tons of ‘liveness’ checks which Qualcomm doesn’t want to get into for obvious reasons. They claim it can’t be fooled by masks, pictures, or even 3D sculptures of the right person. This will be a killer app for China and APAC if done right, secure biometrics are a must have there for online payments and doing it right is not easy.
The next two are both depth related, a two device passive and a three device active module. The passive camera variant is simple enough, take two images separated by a known distance and compute the depth via parallax differences. It is a bit crude but effective enough to get the job done. Every Snapdragon has an ISP and an immensely powerful and severely underestimated DSP to do this kind of math with. For mainstream uses, passive depth is more than good enough.
Active depth cameras are a bit more complex, they have a RGB camera coupled with an IR projector and an IR receiver, time of flight sensors are also supported but nothing is being announced today. While we won’t go into the details of this technology here, it works more than well enough to make a realtime pointcloud of a person playing the piano with visible fingers and keys. This is going to be very useful for VR and AR applications in the future, it could be a killer app to swing OEMs to Qualcomm. Like biometric based banking, depth sensing and making sense of the result is hard work, with Snapdragons OEMs get it for free. What is that worth to them?
That brings us to what is powering all of this and the main new feature on an upcoming but unnamed Qualcomm Snapdragon product is the ISP. In this case the Spectra ISP is now up to v2.0 and it is ground up all new. The idea, other than the usual better, faster, and more efficient, was to make it more granular to programs could drop in and out of the ISP pipelines at more points than before. This allows various hardware computational imaging kernels to be run with more OEM value adds available. It also allows offloading and onloading from various other hardware units, think DSP, in a more efficient way.
This obviously necessitates a completely new software architecture as well which Qualcomm has lots of SDKs ready for along with new features. There weren’t many details but for broad strokes Qualcomm listed five new features and promised more before the year is out. Multi-frame noise reduction, video capture with motion compensated temporal filtering, accelerated electronic image stabilization, machine learning accelerated computer vision, and SLAM for AR/VR.
As you can see from the above, all of these need one thing, heavy compute power. The granular nature of the new Spectra 2.0 ISP pipeline allows the data flow to go down the ISP pipe and break out to custom compute kernels, some on the ISP, some not, then back in at differing points for different use cases. The old Specta was too inflexible to do this, or at least too inflexible to do it efficiently enough. While we won’t say the problem is fully solved yet, we think Qualcomm will convince us when the kimono is fully opened.
So that is what we have from Qualcomm today, three new modules, one new name, and a peek at a new ISP architecture. We don’t have to connect the dot (cloud) manually to tell you how these things are related but the ability to do realtime fairly complex pointcloud captures with off the shelf consumer hardware is going to make for some compelling devices. On the other side (of the phone) the biometric security module is going to be a killer app for finance in Asia. Together they leave one open question, how much will the OEMs value this ‘free’ feature at? SemiAccurate thinks the answer will be quite a bit.S|A