Signals and Threads Swapping the Engine Out of a Moving Race Car with Ella Ehrlich
35 snips
Sep 12, 2022 AI Snips
Chapters
Transcript
Episode notes
Overly Flexible Types Blocked Real Evolution
- GORD originally used large gooey top-level types with string->string maps to avoid client versioning headaches.
- That flexibility prevented schema evolution of first-class message types and forced hacks like embedding rejects in order fields.
Scale By Parallelizing Per-Source Pipelines
- GORD parallelizes by upstream source pipelines; a single parser-DB-normalizer pipeline handles ~25k–50k msgs/sec and horizontal scaling adds capacity.
- This per-source ordering model trades global ordering for scalable throughput.
Switch Fanout To A Pull-Based Message Bus
- Replace push-to-many TCP client delivery with a message queue so slow consumers fall behind instead of causing buffering or kills.
- Ella's team migrated to Kafka so clients pull messages and snapshots can be preproduced once.