I am truly blessed and have lived an exceptional life. After rising to the top of the heap in my career I gave it all up to semi-retire in the mountains and seaside of an Asia tropical paradise. My 1999 research into cyberspace situational awareness that established me as a leading thinker in cyberspace security and my published papers from that era have been cited thousands of times by researchers worldwide, but that was decades ago. Eighteen years ago most of us could have never imagined that cyberspace would be as dominate nor as influential over our daily lives as it is today.
After dropping out of the mainstream work force around a decade ago, living in tropical climate with an abundance of beautiful natural resources and delicious fruits and vegetables, I became a fun-loving, scuba-diving, semi-playboy, world traveling hedonist. Enjoying the single life, I guess you could say that I wasted a lot of time playing around with sexy young women and adventurous activities like dancing all night, scuba diving and motorcycling. Looking back, I felt like I had accomplished so much in life, I just wanted to enjoy myself and I often think I overdid it, especially partying in Bangkok. I did not hurt anyone; but I did seemingly waste a lot of time and money being a type of “mid-life crisis geeky” playboy in such a hedonistic city.
So, I understand why few people read my blog these days.
It’s easy to understand.
Living on the seaside with an amazing sea view towering 27 floors above the beach 12 times zone away from the USA in a tropical paradise with a beautiful younger woman, scuba diving, traveling in style and living the good life, riding around on a custom Ninja motorcycle, and eating tropical food is not how we view “hard working” or “mainstream” researchers by any stretch of the imagination. I have no formal affiliation or corporate allegiance. So, I understand why few people pay any attention to my research on cyberspace situational awareness these days. I retired from “mainstream” when I moved to Asia to scuba dive and “playboy around” 10 years ago; so it is perfectly understandable why people are more interested in social media than cutting edge research into cyberspace from a retired, geeky old man like me.
After my long “hedonistic playboy” break, I started my independent research in cyberspace situational awareness around 6 months ago. I made this move knowing that few would pay attention to my research; but none-the-less, I was motivated to get back to my research to honor those who cited and praised my work of nearly 20 years ago. I am blessed to have lived such a good life from so many perspectives. One thing is for sure, whatever “normal” is, I am not that and never will be. Writing cyberspace processing code is a lot more “lonely” than dancing all night and scuba diving.
In the past few months I have gone from a “white belt” to a “red belt” in C# and Unity 3D programming. My overworked USA PhD student, full-time job, cyberspace collaborator Rich praises me as “an amazing old man” and tells me I’m a “brown belt” in Unity 3D game engine programming already. I’m thankful to have a friendly collaborator who cheers me on as I write and debug thousands of lines of code in C# and PHP, one painstaking line at a time. Hopefully people like Rich will continue this work in years to come. I’m not getting any younger.
Today, I’m debugging processing 4.4 MB of data created by over 200,000 intrusion detection events. If you have never worked with processing big data, you cannot imagine how painstaking it can be to debug an XML or JSON file parsing error when an alert from a sensor has some malformed character in the data which causes the entire process to crash. The data is not clean and data cleansing is boring and mundane work, but critical to success. I have spent days on tracing down a single error, one character in 4,500,000 characters of data. It’s not all roses and flowers coding the data required for the eye candy in this video, LOL.
The video featured in this post represents 214,989 intrusion detection events clustered into 13,628 node and 13,627 edges. In this video we “autopilot” though cyberspace visiting 19 “priority 1” (red) events. Toward the end of the video, I briefly toggle “autopilot” to travel though the nearly 2,000 “priority 2” (yellow) events for a few seconds. I’m working on improving the video quality from my video screen capture and upload to YouTube; but frankly it’s not my highest priority. Data cleaning, object enrichment and graph processing are at the top of my priority list.
Currently I am benchmarking the difference between processing 4.5 MB of XML data version 8 MB of JSON data. The JSON file is larger than the XML file after converting the graph to a force-directed graph; but after some serious debugging and benchmarking, I noticed the size of the data file transferred across the network is less time consuming than the processing of the XML or JSON file by the visualization engine. It currently takes about 40 minutes on a stand alone Linux server to process and create a force-directed graph with around 27,000 nodes and edges. It takes a few seconds to download the file across the Internet from the USA to my condo on the opposite side of the planet; and it takes over a minute for the visualization engine to process the data and render the 27,000 object graph in this video on my 4 GB older model MacBook Air. I assume when I move to a modern gaming computer setup with a super Nvidia GPU this loading and graph rendering will improve, but I have not tested it yet.
Rich praises me as making great progress on this research project; but I feel like I have only created Cyberspace Situational Awareness Eye Candy, so far. This data represents over a week of IDS data on a single server; and I need to further enrich the event-objects to provide more situational knowledge. As mentioned in earlier posts, when we start to fuse data from myriad servers this becomes a huge scaling and big data problem on both the back-end graph processing and visualization side of the architecture.
Lucky for us, we have access to a large computing cluster thanks to Rich’s PhD work at Florida Atlantic University; so Rich is thinking about how to modify our graph processing code so it can be processed in a large computing cluster. I’m not sure yet how to build a force-directed graph across a high performance computing cluster, but I suspect it will require some serious modification to the well written C++ code we borrowed from Jason Graves. This is an important area of research and I’m pleased Rich is considering to refocus his PhD research into this area.
It’s lonely in the research lab; but luckily “someone special” is cooking sticky rice in a traditional Northern Thai style clay oven on the condo balcony. Time to BBQ some pork necks and chicken soon!
You spoke of nvidia – it has a toolkit called cuda c++ that enables you to split off code in support of parallel processing that may help speed up particular functions. It maps certain type of code, but may be something to look at. It’s free and there are books on cuda.