airhacks.fm podcast with adam bien

Apache Storm, Disruptor, JCTools and Linearizability

Apr 27, 2025
Francesco Nigro, a software engineer and JCTools contributor, shares his insights on concurrent programming and performance optimization in Java. He discusses the evolution of JCTools and its contributions to efficient data structures, including weight-free queues. The conversation highlights JCTools’ advantages over standard Java collections, the nuances of linearizability, and the differences between lock-free and wait-free algorithms. Additionally, they delve into Apache Storm’s event processing capabilities and the importance of memory management in achieving high throughput.
Ask episode
AI Snips
Chapters
Transcript
Episode notes
ANECDOTE

Storm's Performance Limitations

  • Apache Storm could only process about one or two million requests per second, far below the needed 100 million per second SLA.
  • This performance gap motivated building a custom lower-level engine leveraging libraries like JCTools for higher throughput.
INSIGHT

JCTools Delivers 400M Ops/Second

  • JCTools achieves approximately 400 million single-producer single-consumer operations per second by passing single references without copying.
  • This rate is two orders of magnitude faster than standard Java concurrent queues that create a lot of garbage.
ADVICE

Avoid Unbounded Linked-Node Queues

  • Prefer array-based queue structures to linked nodes to reduce object allocation and memory overhead.
  • Avoid unbounded linked-node queues if your application is sensitive to garbage creation and needs predictable memory behavior.
Get the Snipd Podcast app to discover more snips from this episode
Get the app