When searching for radiological sources in an urban area, a vehicle-borne detector system will often measure complex, varying backgrounds primarily from natural gamma-ray sources. Much work has been focused on developing spectral algorithms that account for these backgrounds in order to minimize false positive rates without sacrificing the sensitivity. However, information about the environment surrounding the detector system might also provide useful clues about the expected background, thus improving sensitivity. Recent work has focused on extensive measuring and modeling of urban areas with the goal of understanding how these complex backgrounds arise. This work presents an analysis of panoramic video images and gamma-ray background data collected in Oakland, California by the Radiological Multi-sensor Analysis Platform (RadMAP) vehicle. Features were extracted from the panoramic images by semantically labeling the images and then convolving the labeled regions with the detector response. A linear model was used to relate the image-derived responses to gamma-ray spectral features derived using Non-negative Matrix Factorization (NMF). We show that some gamma-ray background features correlate highly with image-derived features such as sky and buildings.