High volume telemetry ingestion interrupters event loop and LMAX
I have been digging deep in Azure and AWS a lot lately. And have been looking at the new world of internet of things. And eventually this combination leads you to the new Event Hubs tool from Azure and the Kinesis tool from Amazon. While researching bits for a presentation on “A Tale of Two Clouds” where I am comparing the features of the two top cloud vendors I came across a great slide deck around “Event Loop”. And nestled in that deck was this wonderful graphic that I am sure to re-use in my “Introduction to CQRS” presentation.
For those that are unaware. This graphic is depicts a developer assumedly writing code for a given application where he is struggling to fight common issues: disk I/O speed, and contention. Generally an application will have one database behind it. And generally the application will be written in such a manner that read data is shaped in the same manner as the write data. A user of the application clicks a button and some data is immediately written to the database…and then immediately turns around and queries the database. Every user interaction for that face is reading data to paint the screen or writing data as part of the interaction. The longer you go in this model the more the single database starts to lock up the disk and the more contention starts to creep up on you (where an operation from user A is blocking an operation from user B).
The guy in this picture needs to learn about distributed systems, DDD, and the CQRS pattern. While I suggest that you don’t treat these as GO-TO patterns for a new application straight away. At least understanding what they can offer you is important. Key learning's can be applied always. Even if you are just keeping your code loose enough to wire in some of those concepts down the road.