During the semester, we will be incrementally developing an evaluation of misuse/anomaly detection systems. This will occur in 3 parts:
- Identify properties we want to evaluate
- Contruct a testing mechanism
- Execute the tests against selected CMAD systems
In particular, we are going to do the following (this description will be enhanced during the course of the semester to further describe the project):
1) Identify properties to test
This will involve defining properties of an intrusion/misuse detection system that we view as important. Included might be:
- Ease of installation
- Clarity of documentation
- Portability across platforms
- Impact on user performance
- Intrusiveness into normal system operation
- Ability to detect anomalous behavior
- Ability to reject "false alarms"
- Usefulness of reporting
2) Develop a testing mechanism
This task will involve the
development of a tool to evalute each system we test. This will
include some subjective measurements of behavior and complexity, but
will also include some objective testing of properties. In
particular, I anticipate that we will develop a suite of modules
written in "expect" that will attempt to exploit some known (fixed,
past) security vulnerabilities. These can then be run against any
arbitrary CMAD system to determine how well it detects those attempts.
3) Test systems
The class will be composed of teams. Each
team will install a CMAD (or related) system on a local machine. They will run the
test suites against their systems, and then report the results to the
class (and in a written report). Systems suggested for inclusion in the test are:
Some subset of these systems will be tested. Others may be added to the list if they become available.
Each student will be expected to execute a final project or paper. This will be due during finals week, and will be graded in lieu of a final exam. The paper is to be a survey and analysis of some aspect of incident response or investigation. More details will be posted here as the semester prprogresses.
Gene Spafford