What you see is what you breathe? Estimating air pollution spatial variation using street level imagery

Beyond here and now: Evaluating pollution estimation across space and time from street view images with deep learning

Advances in computer vision, driven by deep learning, allows for the inference of environmental pollution and its potential sources from images. The spatial and temporal generalisability of image-based pollution models is crucial in their real-world application, but is currently understudied, particularly in low-income countries where infrastructure for measuring the complex patterns of pollution is limited and modelling may therefore provide the most utility. We employed convolutional neural networks (CNNs) for two complementary classification models, in both an end-to-end approach and as an interpretable feature extractor (object detection), to estimate spatially and temporally resolved fine particulate matter (PM2.5) and noise levels in Accra, Ghana. Data used for training the models were from a unique dataset of over 1.6 million images collected over 15 months at 145 representative locations across the city, paired with air and noise measurements. Both end-to-end CNN and object-based approaches surpassed null model benchmarks for predicting PM2.5 and noise at single locations, but performance deteriorated when applied to other locations. Model accuracy diminished when tested on images from locations unseen during training, but improved by sampling a greater number of locations during model training, even if the total quantity of data was reduced. The end-to-end models used characteristics of images associated with atmospheric visibility for predicting PM2.5, and specific objects such as vehicles and people for noise. The results demonstrate the potential and challenges of image-based, spatiotemporal air pollution and noise estimation, and that robust, environmental modelling with images requires integration with traditional sensor networks.

Characterisation of urban environment and activity across space and time using street images and deep learning in Accra

The urban environment influences human health, safety and wellbeing. Cities in Africa are growing faster than other regions but have limited data to guide urban planning and policies. Our aim was to use smart sensing and analytics to characterise the spatial patterns and temporal dynamics of features of the urban environment relevant for health, liveability, safety and sustainability. We collected a novel dataset of 2.1 million time-lapsed day and night images at 145 representative locations throughout the Metropolis of Accra, Ghana. We manually labelled a subset of 1,250 images for 20 contextually relevant objects and used transfer learning with data augmentation to retrain a convolutional neural network to detect them in the remaining images. We identified 23.5 million instances of these objects including 9.66 million instances of persons (41% of all objects), followed by cars (4.19 million, 18%), umbrellas (3.00 million, 13%), and informally operated minibuses known as tro tros (2.94 million, 13%). People, large vehicles and market-related objects were most common in the commercial core and densely populated informal neighbourhoods, while refuse and animals were most observed in the peripheries. The daily variability of objects was smallest in densely populated settlements and largest in the commercial centre. Our novel data and methodology shows that smart sensing and analytics can inform planning and policy decisions for making cities more liveable, equitable, sustainable and healthy.

Street images classification according to COVID-19 risk in Lima, Peru: a convolutional neural networks feasibility analysis

OBJECTIVES

During the COVID-19 pandemic, convolutional neural networks (CNNs) have been used in clinical medicine (eg, X-rays classification). Whether CNNs could inform the epidemiology of COVID-19 classifying street images according to COVID-19 risk is unknown, yet it could pinpoint high-risk places and relevant features of the built environment. In a feasibility study, we trained CNNs to classify the area surrounding bus stops (Lima, Peru) into moderate or extreme COVID-19 risk.

DESIGN

CNN analysis based on images from bus stops and the surrounding area. We used transfer learning and updated the output layer of five CNNs: NASNetLarge, InceptionResNetV2, Xception, ResNet152V2 and ResNet101V2. We chose the best performing CNN, which was further tuned. We used GradCam to understand the classification process.

SETTING

Bus stops from Lima, Peru. We used five images per bus stop.

PRIMARY AND SECONDARY OUTCOME MEASURES

Bus stop images were classified according to COVID-19 risk into two labels: moderate or extreme.

RESULTS

NASNetLarge outperformed the other CNNs except in the recall metric for the moderate label and in the precision metric for the extreme label; the ResNet152V2 performed better in these two metrics (85% vs 76% and 63% vs 60%, respectively). The NASNetLarge was further tuned. The best recall (75%) and F1 score (65%) for the extreme label were reached with data augmentation techniques. Areas close to buildings or with people were often classified as extreme risk.

CONCLUSIONS

This feasibility study showed that CNNs have the potential to classify street images according to levels of COVID-19 risk. In addition to applications in clinical medicine, CNNs and street images could advance the epidemiology of COVID-19 at the population level.