From Bits to Tables: The Evolution of S3 Storage

71 snips

Aug 5, 2025

In this discussion, Andy Warfield, an Amazon storage enhancement expert, dives into the evolution of S3 storage. He explores the revolutionary functionalities of S3 Tables and Vectors, crucial for modern data management and analytics. Andy shares insights on how customer feedback has shaped these developments, improving performance for AI workloads. He also discusses the innovative applications of these features in industries like genomics and finance, along with the technical challenges faced in integrating advanced data types.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Understanding Vectors and S3 Vectors Role

Vectors represent data in high-dimensional space enabling similarity search and semantic retrieval.
S3 Vectors store and query arbitrary dimensional vectors, acting as building blocks linking varied applications and datasets.

INSIGHT

S3 Vectors Indexing Trade-offs

Vector indexes using graph traversal are unsuitable for high latency storage like S3.
S3 Vectors trade round trips for parallel data reads, balancing performance and cost for bursty and archival workloads.

INSIGHT

API Design Lessons from Iceberg

The S3 Tables team learned from Iceberg's evolution to unify file format and REST catalog APIs.
For S3 Vectors, starting with a REST API enables flexibility to evolve indexing without breaking clients.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the Data Engineering Podcast Andy Warfield talks about the innovative functionalities of S3 Tables and Vectors and their integration into modern data stacks. Andy shares his journey through the tech industry and his role at Amazon, where he collaborates to enhance storage capabilities, discussing the evolution of S3 from a simple storage solution to a sophisticated system supporting advanced data types like tables and vectors crucial for analytics and AI-driven applications. He explains the motivations behind introducing S3 Tables and Vectors, highlighting their role in simplifying data management and enhancing performance for complex workloads, and shares insights into the technical challenges and design considerations involved in developing these features. The conversation explores potential applications of S3 Tables and Vectors in fields like AI, genomics, and media, and discusses future directions for S3's development to further support data-driven innovation.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Tired of data migrations that drag on for months or even years? What if I told you there's a way to cut that timeline by up to 6x while guaranteeing accuracy? Datafold's Migration Agent is the only AI-powered solution that doesn't just translate your code; it validates every single data point to ensure perfect parity between your old and new systems. Whether you're moving from Oracle to Snowflake, migrating stored procedures to dbt, or handling complex multi-system migrations, they deliver production-ready code with a guaranteed timeline and fixed price. Stop burning budget on endless consulting hours. Visit dataengineeringpodcast.com/datafold to book a demo and see how they're turning months-long migration nightmares into week-long success stories.
Your host is Tobias Macey and today I'm interviewing Andy Warfield about S3 Tables and Vectors

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what your goals are with the Tables and Vector features of S3?
How did the experience of building S3 Tables inform your work on S3 Vectors?
There are numerous implementations of vector storage and search. How do you view the role of S3 in the context of that ecosystem?
The most directly analogous implementation that I'm aware of is the Lance table format. How would you compare the implementation and capabilities of Lance with what you are building with S3 Vectors?
- What opportunity do you see for being able to offer a protocol compatible implementation similar to the Iceberg compatibility that you provide with S3 Tables?
Can you describe the technical implementation of the Vectors functionality in S3?
- What are the sources of inspiration that you looked to in designing the service?
Can you describe some of the ways that S3 Vectors might be integrated into a typical AI application?
What are the most interesting, innovative, or unexpected ways that you have seen S3 Tables/Vectors used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on S3 Tables/Vectors?
When is S3 the wrong choice for Iceberg or Vector implementations?
What do you have planned for the future of S3 Tables and Vectors?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA