Technical Objectives - Visual and Acoustic Surveillance and Monitoring

Our project explores fundamental research problems related to the integration of

visual (color video and IR) and
acoustic sensory sources

for visual surveillance of urban areas for military or law enforcement purposes, and to demonstrate our research in a series of integrated surveillance scenarios. Our vision of an autonomous urban battlefield surveillance system is based on distributed visual and auditory sensors monitoring an urban site in the context of a pre-existing site model. The site model contains knowledge used by the surveillance system to focus its attention and constrain its image analysis to detect people, vehicles and their interactions. The surveillance system employs both video data processing---applied to color video imagery for daytime surveillance and to infrared imagery for nighttime surveillance---and auditory data processing to perform its detection and analysis activities. A high-level system is used to represent activities of interest, to control image and auditory data processing, and to model the situations in which human assistance is required. The surveillance system is a multi-level system, in which ``wide area'' surveillance employs sensors with large fields of view to detect potential activities of interest, invoking ``narrow angle'' surveillance subsystems that perform more detailed analyses of human and vehicle motion to recognize specific activities requiring operator intervention. Specific goals of our research program include:

Mechanisms for surveillance site model control of monitoring. Prior knowledge about the site being monitored is critical for both the efficiency and the reliability of the surveillance system. Our site models will include representations for buildings (footprints, dimensions, locations of doors, garage entrances), roadways (traffic patterns, parking areas), and structures such as lampposts which might occlude actions being observed from various vantage points.
Fast detection of human and vehicular activity. Wide area site surveillance must be able to, in real time, identify when vehicles and people enter the surveillance area, and must track their motions while in the surveillance area. New site model based algorithms for image stabilization, people detection and tracking in both color video and infrared imagery, independent moving object detection and auditory data analysis will be developed to support this requirement.
High level system for activity recognition The person and vehicle tracking algorithms will, under control of the high-level system, identify certain audio-visual ``trigger" activities that will be used to control the application of more detailed, narrow-area activity recognition algorithms to higher resolution spatio-temporal locales of the audio-video survillance sequences.
Representation and recognition of activities involving interactions of humans and vehicles. We plan to develop a constructive approach to the representation and recognition of human activity that will be applied to the ``narrow area'' surveillance tasks such as people entering or exiting buldings and vehicles, behaving ``suspiciously'' (i.e., running when others are walking), and carrying, placing, removing and exchanging objects.