Applied Machine Learning Director Joaquin Quiñonero Candela wasted no time dropping his first AI sound bite after arriving onstage at F8, “We are taking AI out of the data centre and into the phone”. And if the tech will work on a smartphone, you can be confident that it will work in standalone VR/AR HMDs.
Candela showed a video to demonstrate that through deep learning, computers can now not only recognise every pixel in a video, but they can also identify who each pixel belongs to. He explained that this is based on Mask R-CNN - as shown by CTO Schroepfer the day before - a deep learning engine that simultaneously detects classifies and segments objects. Facebook is building a multi-modal video understanding engine that uses several AI algorithms to recognise and understand actions and words – and even translate them into other languages.
AI Killed The Video Star
Candela said, “The team has built a new neural network architecture that does 3D spatial temporal convolutions using time as a third dimension.” He disclosed that, “On Facebook, every day people watch over 100 million hours of video”, so it’s just as well that, “Our algorithms crunch over one billion frames of video every second”. These video understanding algorithms have been integrated into the Facebook AI platform.
Let’s Go For Coffee
In the last year, Facebook has built a new deep learning framework called Caffe2 that has been designed to be maximally efficient both in data centres and, crucially for standalone VR/AR hardware, on the phone. If that sounds like double the workload in coding content, Candela explained that, “Developers write exactly the same code and the framework optimises it for the target architecture.” And does it work? “On the phone, we were able to speed up neural site transfer by 100 times, making it possible to run locally and in real-time.”
Facebook has made Caffe2 open-source and is building partnerships with the likes of Amazon, Intel, Microsoft, NVIDIA, and Qualcomm. As Candela said, “This is one of the world’s largest AI deployments ever”.
