Technical Tuesday – 6 December 2011 – Malware Analytics at Stream Rate – An Evolving Concept by Dr. Harold Jones
Windows Portable Executables (PEs) are a workhorse for network operations – BAE Systems’ North American network sees roughly 5K unique PEs per day, the vast majority of which are associated with approved activities (e.g., patch downloads). Unfortunately, however, PEs are also the dominant mechanism by which remote intruders install malware on target machines. And because polymorphic malware is so prevalent, 10s of Ks of new malware instances are reported globally every day, even though most are likely functionally equivalent to a much smaller set of “exemplar” executables. The end result is that classical malware analytics approaches (passive and dynamic code analysis) are swamped with too many samples to effectively analyze in-depth. One obvious consequence is to delay recognition and response to new or emergent threats that constitute a heightened risk for the targeted enterprise.
A number of practitioners, including original research conducted by BAE Systems, have responded to this dilemma by proposing real-time mechanisms for harvesting informative attributes from network traffic “at stream rate” without unpacking or executing the PE. Under a program activity sponsored by NSA/R6, BAE Systems has integrated the best of these solutions into a Malware Analytics Framework capable of real-time clustering of PE streams into basic family classes, such as: Benign; Suspected Bad – Family XYZ; Suspected Bad – Family Unlabeled; Suspected Bad – Unique; and Unknown. The objective is not to make definitive classification decisions on “hard-to-analyze instances”. Rather it is to achieve a massive compression of the analysts’ workload – to automate analysis of what is clearly identifiable or clusterable, and then promote for human attention or automated code analysis only those samples or family exemplars that merit in-depth analysis.
This presentation provides a brief survey of the state of the art for “stream-rate” malware clustering – defined somewhat arbitrarily as no more than 1-5 seconds per sample for attribute/anomaly harvesting and classification. We then present an inclusive architecture incorporating the best features of alternative attribute harvesting and correlation/clustering concepts into an effective Real-time Malware Analytics Framework. The analytics are a work-in-progress; however, we will share large-scale performance results that clearly demonstrate that – with the right attribute set and correlator design – malware analytics at stream-rate should be both feasible and incredibly promising.
Presented by: Dr. Harold Jones of BAE
Dr. Jones is Chief Technical Officer for BAE Systems’ Advanced Cybersecurity Solutions business. He was responsible for the design, implementation and operation of the Leading Edge NOSC Environment (LENE), which has deployed a next-generation network defense testbed on top of BAE Systems’ 50,000 node North American network. In the LENE, he is currently leading concept development and productization of solutions for real-time network forensics and malware analysis.
Previously, Dr. Jones led the initiative to transform BAE Systems’ broad technology base in computer network operations into the mission-ready defensive and offensive cyber solutions that will be matured in the LENE. This included development of a suite of network behavior anomaly detection products under the Network Intrusion Detection and Response (NIDAR) product line. His team also developed a series of special products to support network reconnaissance missions. His research teams combined a firm grounding in computer science with a corresponding mastery of estimation theory and statistical reasoning, allowing them to build autonomic cyber systems capable of reasoning and acting in machine time.
Dr. Jones received his Ph.D. from the Massachusetts Institute of Technology.
