A Hybrid CNN–Vision Language Model Framework for Visual Fire Detection and Context-Aware Decision Support in Oil and Gas Facilities

Abstract In Nigeria's oil and gas industry, fire and explosion incidents have claimed more than 3,400 lives over the last 15 years, which are fueled by old infrastructure, chronic pipeline vandalism and lack of intelligent real time monitoring. Traditional sensor-based systems of detection are limited in their spatial coverage, have a high false-alarming rate in flare-rich conditions, and, most importantly, lack contextual information concerning the severity of the fire and the emergency response that is advisable. In the meantime, the state-of-the-art detectors, based on deep learning, boast of high localization and are blind to the context of the scenes, whereas Vision Language Models (VLMs) show high scene reasoning and cannot support real-time alerting. There is no system that combines these capabilities in terms of petroleum facility safety. In this paper, a hybrid model is introduced, which combines a Convolutional Neural Network (CNN)-based real-time fire and smoke detector with a Vlm-based contextual reasoning layer, in the context of oil and gas facilities monitoring. The stage of detection relies on the YOLO26s architecture (9.95 million parameters, 22.5 GFLOPs) which is trained on the combined and quality-controlled dataset of 30,337 annotated images obtained using the Fire and Smoke Detection Dataset (FASDD) and the Middle East Technical University (METU) fire dataset. At an inference latency of 0.5 ms per frame on an NVIDIA A100, YOLO26s on a validation set reached mean Average Precision (mAP 50) of 83.4% and balanced per-class performance, fire (83.4) and smoke (83.3). Frames meeting a confidence threshold are sent to a contextual reasoning stage, which uses the NVIDIA DAM-3B, a fully open-source, 3-billion-parameter Vision Language Model (VLM) hosted entirely on local infrastructure, to produce structured, natural-language safety reports, including fire severity classification, smoke density and combustion completeness analysis, infrastructure proximity analysis, and Health, Safety and Environment (HSE)-consistent emergency advice within 2–3 seconds of frame capture. Because both the CNN detector and the VLM reasoning layer run on self-hosted infrastructure, the end-to-end pipeline is highly cost-effective and delivers near-instantaneous inference speeds. The entire pipeline was implemented as a web application supporting real-time image and video analysis and VLM reporting. This paper provides the first end-to-end CNN-VLM fire detection and decision-support framework specialised to the petroleum industry, quantitative benchmarks in the use of YOLO26-based fire and smoke detection on fused industrial data, and a deployable architecture that closes the gap between automated fire detection and human-explainable safety intelligence, directly in response to regulatory requirements as stipulated in the Petroleum Industry Act 2021. Future work focuses on collecting Nigerian oil and gas facility fire data, collaborating with local operators to incorporate localized Nigerian facility imagery, validating model outputs against petroleum safety expert consensus, and testing the full system in real operational environments.