Tracking Anomalies Instead of Scoring Pixels: A Look at the TAO Video Surveillance Pipeline

This story was originally published on HackerNoon at: https://hackernoon.com/tracking-anomalies-instead-of-scoring-pixels-a-look-at-the-tao-video-surveillance-pipeline.
Video is now the default way we watch public spaces. In this article, we talk about a new method for anomaly detection in video surveillance.
Check more stories related to cybersecurity at: https://hackernoon.com/c/cybersecurity. You can also check exclusive content about #surveillance, #digital-surveillance, #surveillance-system, #ai-surveillance-system, #ai-in-surveillance, #video-surveillance, #anomaly-tracking, #anomaly-detection, and more.

This story was written by: @vishwagw. Learn more about this writer by checking @vishwagw's about page, and for more stories, please visit hackernoon.com.

Anomaly detection in surveillance means catching the unusual — a fight, a weapon, a vehicle where pedestrians should be, an accident. Existing methods are either frame-centric (they tell you a frame is anomalous but not where) or object-centric (more precise, but still no clean pixel-level boundaries). Both struggle when anomalies overlap or occlude each other. TAO reframes the whole problem: instead of scoring every pixel at every moment, it treats anomaly detection as pixel-level tracking of anomalous objects across the video. It does this by pairing an object-centric detector (which draws bounding boxes around suspicious objects) with SAM2, a pretrained segmentation model that turns those boxes into precise masks — no fine-tuning on anomaly data required. The pipeline runs in four stages: bounding box extraction → anomalous box extraction → robust filtering → segmentation. The authors also introduce a dual-level benchmark that scores both object-level and pixel-level accuracy, and report state-of-the-art results on UCSD Ped2 and ShanghaiTech.