Home Advanced Event Processing Update on Scaleability 100,000 IDS Events – Hip Hip Array!

Update on Scaleability 100,000 IDS Events – Hip Hip Array!

IDS Alerts - Visualization
IDS Alerts - Visualization

One of the good things about C# game programming on a tiny three year old MacBook Air with 4MB of RAM and a boring old graphics card is that I do not have the luxury of writing bad code. Recently my graphics card ground to a halt because I was intersecting a 3D ray cast from the mouse with one of 7,500 nodes, the results of clustering 100,000 cyber events by country, IP address and intrusion ID. Using a gaming loop during each frame I had my earlier “unscalable” code looping through all 7,500 nodes during each frame, causing a huge performance problem.

After days of research, no joy. Google could not save the day. Countless forums and tech sites – no help. Nadda.

IDS Event Visualization
IDS Event Visualization

I ended up rewriting the code to change the way the gaming loop behaved. The solution was rooted in arrays, moving the processing to the CPU and off the GPU. Seems these days all my solutions end up rooted in some kind of tricky or complex array processing, trading off back end processing with visualization engine processing.

Now, 100,000 events on the MBA, no problem. Fast and smooth as silk. 100,000 events clustered into 7,500 nodes and 7,500 edges on the MBA, no problem – been there, done that. Next, I think I will go for 1,000,000 events clustered into 100,000 nodes and 100,000 edges soon on this tiny MacBook.

Should be ready for cyberspace VR programming soon 🙂


  1. Hip Hip Array! Nice and catchy title. 🙂

    Yes, very impressive that you’ve gotten all of this optimized to run on an older Macbook Air. It looks very beautiful! Awesome game programming Tim, congrats! 100,000 alerts visualized, wow!