
airhacks.fm podcast with adam bien Apache Storm, Disruptor, JCTools and Linearizability
Apr 27, 2025
Francesco Nigro, a software engineer and JCTools contributor, shares his insights on concurrent programming and performance optimization in Java. He discusses the evolution of JCTools and its contributions to efficient data structures, including weight-free queues. The conversation highlights JCTools’ advantages over standard Java collections, the nuances of linearizability, and the differences between lock-free and wait-free algorithms. Additionally, they delve into Apache Storm’s event processing capabilities and the importance of memory management in achieving high throughput.
AI Snips
Chapters
Transcript
Episode notes
Storm's Performance Limitations
- Apache Storm could only process about one or two million requests per second, far below the needed 100 million per second SLA.
- This performance gap motivated building a custom lower-level engine leveraging libraries like JCTools for higher throughput.
JCTools Delivers 400M Ops/Second
- JCTools achieves approximately 400 million single-producer single-consumer operations per second by passing single references without copying.
- This rate is two orders of magnitude faster than standard Java concurrent queues that create a lot of garbage.
Avoid Unbounded Linked-Node Queues
- Prefer array-based queue structures to linked nodes to reduce object allocation and memory overhead.
- Avoid unbounded linked-node queues if your application is sensitive to garbage creation and needs predictable memory behavior.
