Decentralized Data and AI Workloads with Hammerspace CEO David Flynn

Oct 5, 2023

On this episode, David Flynn, CEO of Hammerspace, discusses the future of data storage and management. He explores the untapped potential of Solid State and Flash technology, revealing the shortcomings of platforms like Amazon’s S3. David also shares his journey from programming on a Commodore 64 to becoming a trailblazer in the industry.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Fusion‑io Origin Story

David Flynn described founding Fusion-io and introducing high-performance SSDs to enterprise storage.
He said a single device outperformed multi-million dollar arrays and helped create the NVMe flash market.

INSIGHT

Store‑and‑Copy Is A Broken Paradigm

Flynn argues current data practices (store-and-copy) are broken and limit compute placement.
He proposes data orchestration where a global file system moves data from inside the presentation layer, making it locally accessible without external copies.

INSIGHT

S3 Is Not A Real File System

Flynn criticizes object stores like S3 as simplified, non‑POSIX interfaces that force apps to manage gets/puts.
He says AI workloads need real shared file systems so researchers don't have to rewrite applications for object APIs.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

On this episode of FYI, we’re joined by David Flynn, the mind behind Hammerspace, a privately owned company working in the realm of data storage and management. On this episode, ARK Director of Next Generation Internet Research Frank Downing, and CEO and CIO Cathie Wood dives deep into a world where data isn’t just stored, but orchestrated. David takes us through the futuristic landscape crafted by Hammerspace, leveraging the untapped potential of Solid State and Flash technology to redefine data management. From unveiling the shortcomings of platforms like Amazon’s S3 to pioneering the concept of data orchestration, David takes us on a journey through the frontier of data technology. Listeners will also hear about David’s path from programming on a Commodore 64 to becoming a trailblazer in the industry.

“Once data becomes orchestrated, and here’s the bizarre thing, data is more permanent in an orchestration platform. It’s not bound to the lifetime of any one storage system. It simply lives in the ether.” – David Flynn Key Points From This Episode:

Overview of David’s background
Leveraging luck and experience with embedded systems to exploit the potential of flash storage
Consumer electronics sparked demand, lowering costs and expanding market opportunities
Hammerspace’s approach to try to fundamentally transform data management through decentralized, dynamic data orchestration
Hammerspace’s data orchestration: facilitating direct manipulation on a global file system
Hammerspace offers a unique network attached storage (NAS) system that enhances data accessibility and boosts performance by separately handling metadata and actual data
Object storage is cheaper, simpler but still important; Hammerspace enhances utility
Companies in sectors such as artificial intelligence (AI), entertainment, science, and design benefit from a universal data interface, especially when handling large, unstructured data sets
AI and machine learning (ML) revolutionizing industries, enhancing data management and accelerating research
Transition to data orchestration technology might take around five to ten years, paralleling server virtualization’s adoption cycle but potentially hastened by AI advancements