The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) Geometry-Aware Neural Rendering with Josh Tobin - #360
Mar 26, 2020
Josh Tobin, co-organizer of the Full Stack Deep Learning program and former research scientist at OpenAI, dives deep into geometry-aware neural rendering. He highlights the challenges in generating 3D scenes, the importance of domain randomization, and innovative methods bridging real-world data with simulations. The conversation also touches on the significance of encoder-decoder architectures in enhancing image rendering and how these techniques are revolutionizing AI applications in robotics.
AI Snips
Chapters
Transcript
Episode notes
Geometry-Aware Neural Rendering
- Geometry-aware neural rendering builds on DeepMind's Generative Query Networks (GQN).
- It uses epipolar geometry to constrain the search for relevant pixels, improving rendering efficiency.
GQN Architecture
- GQN uses an encoder-decoder architecture. The encoder processes viewpoints independently, sums representations, creating a scene representation.
- The decoder turns this representation into the rendered image.
Attention Mechanism
- Instead of a scene representation, an attention mechanism attends over context image representations.
- This mechanism uses epipolar geometry and a scaled dot product for efficient rendering.