Geospatial Data Pipeline |

Geospatial Data Pipeline to Study the Health Effects of Environments -Limitations and Solutions-

Mon, 30 Sep 2024 00:00:00 +0000

Air Pollution Predictive Modeling

Sun, 27 Dec 2020 00:00:00 +0000

Summary

This project aims to predict air pollution concentrations at locations across South Korea where direct measurements are unavailable. Using the Universal Kriging (UK) model, we estimate the annual concentrations of key air pollutants—PM10, PM2.5, O3, and NO2. The UK model leverages observed pollutant concentrations at monitoring sites and incorporates spatial and environmental characteristics represented by geographic variables, along with spatial autocorrelation, to predict concentrations at unmonitored locations. The observational data consists of annual air pollution measurements collected from air quality monitoring stations nationwide. Additionally, the model integrates 320 geographic variables, derived from spatial data processing pipelines, which capture factors related to air pollution levels, such as road networks, elevation, population density, land use, and vegetation index (NDVI). To manage the high dimensionality of these variables, we employ Partial Least Squares (PLS) regression to reduce the dataset to 2–3 primary predictive components, which serve as inputs for the UK model. This dimensionality reduction optimizes model performance by focusing on the most relevant predictors. In many cases, our team utilizes this model to estimate air pollution exposure for cohort study participants based on their residential locations. This process involves an initial geocoding step, where participants’ address information is converted into geographic coordinates. Subsequently, approximately 320 geographic variables are calculated for each residential location. These variables are then reduced to key components via PLS, which are used as inputs in the UK model to estimate pollutant concentrations for individual residences. Through this project, we enhance the precision of individual exposure assessments by estimating air pollution concentrations directly at individuals’ residential locations rather than relying on the concentrations from nearby monitoring sites. This approach enables more accurate evaluations of personal exposure and contributes to more precise health effect assessments related to air pollution exposure.

Geospatial Data Pipeline Construction

Fri, 27 Dec 2019 00:00:00 +0000

Summary

Spatial geographic information databases play a critical role in various aspects, including the integration of location-based data, identification of risk factors, exploration of spatial patterns and correlations, and data visualization to support decision-making. However, the construction and utilization of spatial geographic information databases face numerous challenges: lack of consistency in spatiotemporal data formats, limitations of aggregate data and insufficient granularity, inaccuracies in location information, inconsistencies in coordinate systems, temporal discrepancies in infrastructure and data formats, variations in spatial resolution, and the challenge of handling large data volumes.

These issues result in significant time and cost expenditures for spatial data processing. Additionally, annual updates of spatial data necessitate a repetition of identical processing steps.

To address these challenges, it is essential to establish a comprehensive spatial geographic information data pipeline that automates the entire process from data acquisition to processing and information production. This pipeline should also include the creation of manuals and guidelines, along with a shared platform for broader accessibility.

Building such a spatial geographic information database enables researchers unfamiliar with spatial data to effectively use it. Even for experienced researchers, the pipeline can reduce repetitive data processing tasks and enhance the reproducibility of spatial data handling. Additionally, this shared analysis platform allows researchers to perform analyses without constraints on time or data capacity.

In this project, Amazon Web Services (AWS) was utilized, with ‘S3’ providing a cloud storage space for raw data and ‘Relational Database Service (RDS)’ supporting the storage and computation of relational geographic information data. This pipeline further includes the calculation of approximately 300 geographic variables using the processed data stored in the data storage. The analysis platform offers a user interface for immediate data querying, processing, and analysis through R Studio and Jupyter Lab.

In addition to data storage and processing, this pipeline includes the calculation of approximately 300 geographic variables, leveraging diverse spatial geographic information. These geographic variables enable a range of analyses, including spatial distribution analyses of environmental factors, exploration of spatial correlations with disease incidence and mortality rates, and spatial modeling of the health impacts of environmental risk factors. This expanded analytical capability allows researchers to gain insights into the spatial dynamics of environmental influences on health and other social factors.