Real-Time Anomaly Detection

Portscan Detection

Attackers routinely scan the IP address space of a target network to seek out vulnerable hosts that they can exploit. One of the challenges is the difficulty in defining a portscan activity. How to perform portscan (i.e. the scanning rate and the coverage of the IP address) is entirely up to each scanner; therefore, the scanner can evade any detection algorithm that depends on the parameters that are under its control. However, in principle, the access pattern of port-scanning can be quite different from that of other legitimate activities. Since port scanners have little knowledge of the configuration of a target network (they would not have to scan the network otherwise), their access pattern often includes non-existent hosts or hosts that do not have the requested service running. On the contrary, there is little reason for legitimate users to initiate connection requests to inactive servers. Based on this observation, we formulate a detection problem that provides a basis on an online algorithm. For more detailed treatment of these bounds and the evaluation of the detection algorithm using real network traces, see [1].

Worm Detection and Throttling

A worm is a program containing malicious code that spreads from host to host without human intervention. One instance of such software malcode, a scanning worm, vastly probes a set of randomly chosen IP addresses to locate vulnerable servers that it wish to infect. Analogous to the previous portscan detection problem, this random scanning behavior can be used to identify an infected machine that is engaged in worm propagation. While using sequential hypothesis testing has promise for detecting scanning worms, there is a significant hurdle to overcome. In [2], we discuss problems at length and present two innovations that enable us to develop a fast detection of scanning worms

Research Agenda

  1. Understanding anomalous network activities:

    In many problem domains, we lack good models for anomalous network activities that capture unique characteristics useful for distinguishing them from benign ones. To address this, we intend to take traces from many vantage points and to look for patterns that can be incorporated into a model.

  2. Algorithms resilient to evasion:

    When designing detection algorithms in network security, one must be concerning with adversaries who can craft an attack to evade detection once the algorithm is publicized. The barrier should be high enough to resist evasion.

  3. Evaluation:

    Detection algorithms must be evaluated both analytically and through trace-driven simulation and false positive and false negative cases should be analyzed. Also, estimating the amount of states required to run the algorithm is important since real-time detection of network anomalies often requires monitoring high-bandwidth networks.

  4. Extension to distributed detection systems:

    We also plan on extending this work to distributed real-time detection problems. For instance, Internet-scale worm propagation can be better identified if detection systems are deployed over many places to cover various vantage points. Coordination among distributed detection systems should be one of the key design components.


[1] Fast Portscan Detection Using Sequential Hypothesis Testing

Jaeyeon Jung, Vern Paxson, Arthur W. Berger, and Hari Balakrishnan
In Proc. IEEE Symposium on Security and Privacy, Oakland, CA, May, 2004

[2] Fast Detection of Scanning Worm Infections

Stuart E. Schechter, Jaeyeon Jung, and Arthur W. Berger
In Proc. of The Seventh International Symposium on Recent Advances in Intrusion Detection (RAID), Sophia Antipolis, France, September 2004


NMS HomeProjectsPeoplePapersSoftware


M. I. T. Computer Science and Artificial Intelligence Laboratory · 32 Vassar Street · Cambridge, MA 02139 · USA