Stony Brook researchers use high-performance computing to advance AI methods that help autonomous vehicles understand and prepare for safety-critical traffic scenarios.
Autonomous vehicles (AV) must be tested against safety-critical scenarios that are hardest to study, as traffic crashes are rare events but often associated with significant consequences. These events are difficult to collect at scale in the real world, and even when crash videos are available, thoroughly understanding what happened, when risk first appeared, and how the crash could have been prevented remains challenging.
At Stony Brook University, Dr. Ruwen Qin and his PhD students are developing AI methods to support AV safety from two complementary directions. One project helps AV understand real crash events from dashcam videos. The other generates realistic and controllable crash scenarios for training and evaluating AV systems. Both rely on high-performance computing to handle the scale of modern multimodal and generative AI research.
CrashChat: Understanding Safety-Critical Events
Kaidi Liang, a PhD student in Qin's group, led the development of CrashChat, a multimodal large language model for traffic crash video analysis. CrashChat is designed to perform multiple crash-understanding tasks in a unified framework. Given a traffic video, it can recognize whether a crash occurred, identify the crash time, detect earlier pre-crash cues, describe the event, reason about likely causes, and suggest possible prevention measures.
This matters because crashes are not simple yes-or-no events. They unfold over time, often beginning with subtle visual cues before impact. A useful safety-analysis tool must connect what happened, when it happened, why it happened, and how similar events might be avoided. CrashChat trains multimodal models to combine video perception with language-based reasoning, improving its ability to interpret safety-critical driving events.
Figure 1: Overview of CrashChat (original Fig. 1 in [1])
CCFM: Generating Controllable Crash Scenarios
Ke Li, also a PhD student in Qin's group, led Collision-Constrained Flow Matching, or CCFM, a generative model for safety-critical scenario generation. Real crashes are too rare and risky to collect in large numbers, so synthetic data in a closed-loop simulation play a central role in AV testing. However, simulated crash scenarios must be realistic, controllable, and physically feasible.
CCFM addresses this challenge by combining flow matching, a modern generative modeling method, with hard physical constraints about crashes. Rather than only encouraging dangerous behavior through soft guidance, CCFM explicitly constrains generated trajectories according to collision type, contact geometry, heading, and severity. It can generate controlled rear-end, side, cut-in, and head-on safety-critical scenarios.
Figure 2: Overview of CCFM (original Fig. 1 in [2])
Together, CrashChat and CCFM support two key needs in AV safety. CrashChat helps analyze and explain real-world safety-critical events. CCFM helps create rare but important simulated crash events for training, stress-testing, and evaluating AV modules. The generated scenarios can also serve as conditions for future video generation.
The Role of High-Performance Computing
High-performance computing plays a critical role in both studies. While both projects are primarily funded by R-SEAT (Rural Safety Efficient Advanced Transportation) Center, a tier 1 University Transportation Center (UTC) of USDOT, the HPC at Stony Brook University plays a critical role in these studies. Training multimodal large language models, processing crash videos, running generative models, performing ablation studies, and evaluating closed-loop simulations all require substantial computational capacity. HPC makes it possible to conduct these experiments at the scale needed for rigorous AV safety research.
By combining crash understanding, physically constrained scenario generation, and high-performance computing, Qin's group is building tools that can help AV systems become safer, better tested, and more reliable before deployment.
Related Publications
- Liang, Kaidi, Ke Li, Xianbiao Hu, and Ruwen Qin, 2026. “CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis.” To Appear in Proceedings of the 28th International Conference on Pattern Recognition (ICPR).
- Li, Ke, Kaidi Liang, Yuxin Ding, Debojyoti Biswas, Xianbiao Hu, and Ruwen Qin. “CCFM: Collision-Constrained Flow Matching for Safety-Critical Scenario Generation.” To Appear in Proceedings of the 19th European Conference on Computer Vision (ECCV).