Kate Shaw, Senior Product Manager for Data at SnapLogic, dives into the complexities of legacy systems and their modern replacements. She highlights that legacy isn't just age—it's about risk and innovation barriers. They discuss technical debt, lost context from turnover, and the dangers of 'if it ain’t broke.' Shaw advocates for composable architectures and planning exit strategies from day one. Additionally, she touches on integrating legacy systems into AI initiatives and the importance of transparency in data governance. A must-listen for anyone navigating modernization!
01:04:16
forum Ask episode
web_stories AI Snips
view_agenda Chapters
auto_awesome Transcript
info_circle Episode notes
volunteer_activism ADVICE
Validate Lineage And Data Freshness
Continuously question data lineage and freshness, not just surface metric values.
Kate Shaw advises tracking upstream changes because connected systems can silently corrupt downstream metrics.
question_answer ANECDOTE
Unknown Jobs Can Hide Year‑End Failures
Kate describes systems nobody fully understands that only surface at year‑end failures.
She warns switching unknown jobs off can silently break critical yearly processes.
volunteer_activism ADVICE
Design For Composability And Swap‑ability
Build composable architectures so you can swap components without full rewrites.
Kate Shaw recommends robust APIs and open standards to avoid vendor lock-in and enable model or vendor swaps.
Get the Snipd Podcast app to discover more snips from this episode
Summary In this episode Kate Shaw, Senior Product Manager for Data and SLIM at SnapLogic, talks about the hidden and compounding costs of maintaining legacy systems—and practical strategies for modernization. She unpacks how “legacy” is less about age and more about when a system becomes a risk: blocking innovation, consuming excess IT time, and creating opportunity costs. Kate explores technical debt, vendor lock-in, lost context from employee turnover, and the slippery notion of “if it ain’t broke,” especially when data correctness and lineage are unclear. Shee digs into governance, observability, and data quality as foundations for trustworthy analytics and AI, and why exit strategies for system retirement should be planned from day one. The discussion covers composable architectures to avoid monoliths and big-bang migrations, how to bridge valuable systems into AI initiatives without lock-in, and why clear success criteria matter for AI projects. Kate shares lessons from the field on discovery, documentation gaps, parallel run strategies, and using integration as the connective tissue to unlock data for modern, cloud-native and AI-enabled use cases. She closes with guidance on planning migrations, defining measurable outcomes, ensuring lineage and compliance, and building for swap-ability so teams can evolve systems incrementally instead of living with a “bowl of spaghetti.”
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Kate Shaw about the true costs of maintaining legacy systems
Interview
Introduction
How did you get involved in the area of data management?
What are your crtieria for when a given system or service transitions to being "legacy"?
In order for any service to survive long enough to become "legacy" it must be serving its purpose and providing value. What are the common factors that prompt teams to deprecate or migrate systems?
What are the sources of monetary cost related to maintaining legacy systems while they remain operational?
Beyond monetary cost, economics also have a concept of "opportunity cost". What are some of the ways that manifests in data teams who are maintaining or migrating from legacy systems?
How does that loss of productivity impact the broader organization?
How does the process of migration contribute to issues around data accuracy, reliability, etc. as well as contributing to potential compromises of security and compliance?
Once a system has been replaced, it needs to be retired. What are some of the costs associated with removing a system from service?
What are the most interesting, innovative, or unexpected ways that you have seen teams address the costs of legacy systems and their retirement?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on legacy systems migration?
When is deprecation/migration the wrong choice?
How have evolutionary architecture patterns helped to mitigate the costs of system retirement?
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Data Engineering Podcast