Augmented reality, a “surreality” check from the engine room

Have you seen the amazing video clips of augmented reality, where contextual information and 3D graphics are superimposed over a scene captured with a video camera, in real-time, on a smartphone? Check this out, if you haven’t. I can assure you, it’s not Sci-Fi; it’s becoming real, though it is still a bit surreal even to those of us who work in the NovaThor U8500’s engine room.

I’m the tour guide here, and the gentlemen standing over there is my friend Peter Meier. Peter is chief technology officer of metaio and one of the world’s foremost gurus in augmented reality technology. Together we have created some very exciting technology that we would like to show you.

Do come in, let us show you around. Here we have the centerpiece of the augmented reality engine room – an integrated application processor and modem, a system-on-chip, comprising all of the functionality needed for a multimedia-centric smartphone. Let me point out the key subsystems of the chip that we learned were most critical to bring this to life:

• The high-performance multi-core CPU subsystem with enough power to drive Android OS, a bunch of background applications and the augmented reality application at the same time
• The camera subsystem supporting stereoscopic vision to sense surroundings and objects
• The sensor subsystem that keeps track of movement and other physical stimuli.
• The wireless broadband subsystem that transfers real-time data to populate the augmented reality with useful or fun information from the Internet wherever you are
• The graphics subsystem that puts it all together, building the augmented reality view in three dimensions

Over here, in this library section at the very center of the engine room, we have Peter’s baby, the metaio mobile SDK software. It’s here that the thinking, calculation and decision-making are done, in short, the creative patterns behind just about everything an augmented reality-enabled application can do. We have placed this precious piece of software right here to gain fast access to all major functions in the hardware and the firmware. That’s what all these connections are for.

This hardwiring of both software and hardware was very necessary, believe us. We have lots of CPU processing power to do all the calculations for:

• Camera compensation and video stabilization
• 3D rectification, 3D scene composition and depth extraction
• Feature tracking, feature extraction and feature matching
• Face detection, pose extraction and sensor fusion

But this is not enough. Real-time access to the video, sensor and graphics subsystems has proven to be the bottle-neck even in a state-of-the art engine room like this. We learned that no other application pushes all parts of a smartphone processor system at the same time like augmented reality. You don’t do only video, nor do you do only browsing. Every major function in the system is busy. Since augmented reality is so new, there are simply no chip-sets that have been completely optimized for it yet.

Having said that, let’s step outside the engine room for a minute and climb up the ladder to the lookout post. Over there in the ST-Ericsson labs the next generation of the hardware for engine room is being developed. They have learnt from the challenges we faced and made some pretty big improvements for the coming products. I am really looking forward to starting up that engine when it is ready!

And if that wasn’t enough, just look at these drawing we got from the engineers in that other lab. They are showing programmable graphics engines, heterogeneous clusters with processor cores, and special purpose hardware blocks all designed with augmented reality in mind; pretty futuristic-sounding stuff that apparently isn’t too far away.

With this in mind, it’s pretty awesome what our engineers and Peter’s wizards from metaio have achieved in this engine room. If it works well now, just try to imagine what is still to come. Wait; let’s start the engines and take you on a test ride. Peter, will you press that green button?

Can you see it? Yeah. Is that augmented reality, or what?