Comparing the quality of crowdsourced data contributed by expert and non-experts

Building a community-based open harmonised reference data repository for global crop mapping

Reference data is key to produce reliable crop type and cropland maps. Although research projects, national and international programs as well as local initiatives constantly gather crop related reference data, finding, collecting, and harmonizing data from different sources is a challenging task. Furthermore, ethical, legal, and consent-related restrictions associated with data sharing represent a common dilemma faced by international research projects. We address these dilemmas by building a community-based, open, harmonised reference data repository at global extent, ready for model training or product validation. Our repository contains data from different sources such as the Group on Earth Observations Global Agricultural Monitoring Initiative (GEOGLAM) Joint Experiment for Crop Assessment and Monitoring (JECAM) sites, the Radiant MLHub, the Future Harvest (CGIAR) centers, the National Aeronautics and Space Administration Food Security and Agriculture Program (NASA Harvest), the International Institute for Applied Systems Analysis (IIASA) citizen science platforms (LACO-Wiki and Geo-Wiki), as well as from individual project contributions. Data of 2016 onwards were collected, harmonised, and annotated. The data sets spatial, temporal, and thematic quality were assessed applying rules developed in this research. Currently, the repository holds around 75 million harmonised observations with standardized metadata of which a large share is available to the public. The repository, funded by ESA through the WorldCereal project, can be used for either the calibration of image classification deep learning algorithms or the validation of Earth Observation generated products, such as global cropland extent and maize and wheat maps. We recommend continuing and institutionalizing this reference data initiative e.g. through GEOGLAM, and encouraging the community to publish land cover and crop type data following the open science and open data principles.

Human-supervised clustering of multidimensional data using crowdsourcing

Clustering is a central task in many data analysis applications. However, there is no universally accepted metric to decide the occurrence of clusters. Ultimately, we have to resort to a consensus between experts. The problem is amplified with high-dimensional datasets where classical distances become uninformative and the ability of humans to fully apprehend the distribution of the data is challenged. In this paper, we design a mobile human-computing game as a tool to query human perception for the multidimensional data clustering problem. We propose two clustering algorithms that partially or entirely rely on aggregated human answers and report the results of two experiments conducted on synthetic and real-world datasets. We show that our methods perform on par or better than the most popular automated clustering algorithms. Our results suggest that hybrid systems leveraging annotations of partial datasets collected through crowdsourcing platforms can be an efficient strategy to capture the collective wisdom for solving abstract computational problems.

A Continental Assessment of the Drivers of Tropical Deforestation With a Focus on Protected Areas

Deforestation contributes to global greenhouse gas emissions and must be reduced if the 1.5°C limit to global warming is to be realized. Protected areas represent one intervention for decreasing forest loss and aiding conservation efforts, yet there is intense human pressure on at least one-third of protected areas globally. There have been numerous studies addressing the extent and identifying drivers of deforestation at the local, regional, and global level. Yet few have focused on drivers of deforestation in protected areas in high thematic detail. Here we use a new crowdsourced data set on drivers of tropical forest loss for the period 2008–2019, which has been collected using the Geo-Wiki crowdsourcing application for visual interpretation of very high-resolution imagery by volunteers. Extending on the published data on tree cover and forest loss from the Global Forest Change initiative, we investigate the dominant drivers of deforestation in tropical protected areas situated within 30° north and south of the equator. We find the deforestation rate in protected areas to be lower than the continental average for the Latin Americas (3.4% in protected areas compared to 5.4%) and Africa (3.3% compared to 3.9%), but it exceeds that of unprotected land in Asia (8.5% compared to 8.1%). Consistent with findings from foregoing studies, we also find that pastures and other subsistence agriculture are the dominant deforestation driver in the Latin Americas, while forest management, oil palm, shifting cultivation and other subsistence agriculture dominate in Asia, and shifting cultivation and other subsistence agriculture is the main driver in Africa. However, we find contrasting results in relation to the degree of protection, which indicate that the rate of deforestation in Latin America and Africa in strictly protected areas might even exceed that of areas with no strict protection. This crucial finding highlights the need for further studies based on a bottom up crowdsourced, data collection approach, to investigate drivers of deforestation both inside and outside protected areas.

Detecting the Severity of Socio-Spatial Conflicts Involving Wild Boars in the City Using Social Media Data

The encroachment of wild boars into urban areas is a growing problem. The occurrence of wild boars in cities leads to conflict situations. Socio-spatial conflicts can escalate to a varied degree. Assessments of these conflicts can be performed by analyzing spatial data concerning the affected locations and wild boar behaviors. The collection of spatial data is a laborious and costly process that requires access to urban surveillance systems, in addition to regular analyses of intervention reports. A supporting method for assessing the risk of wild boar encroachment and socio-spatial conflict in cities was proposed in the present study. The developed approach relies on big data, namely, multimedia and descriptive data that are on social media. The proposed method was tested in the city of Olsztyn in Poland. The main aim of this study was to evaluate the applicability of data crowdsourced from a popular social networking site for determining the location and severity of conflicts. A photointerpretation method and the kernel density estimation (KDE) tool implemented in ArcGIS Desktop 10.7.1 software were applied in the study. The proposed approach fills a gap in the application of crowdsourcing data to identify types of socio-spatial conflicts involving wild boars in urban areas. Validation of the results with reports of calls to intervention services showed the high coverage of this approach and thus the usefulness of crowdsourcing data.

A Bayesian Approach to Estimate the Spatial Distribution of Crowdsourced Radiation Measurements around Fukushima

Citizen-led movements producing spatio-temporal big data are potential sources of useful information during hazards. Yet, the sampling of crowdsourced data is often opportunistic and the statistical variations in the datasets are not typically assessed. There is a scientific need to understand the characteristics and geostatistical variability of big spatial data from these diverse sources if they are to be used for decision making. Crowdsourced radiation measurements can be visualized as raw, often overlapping, points or processed for an aggregated comparison with traditional sources to confirm patterns of elevated radiation levels. However, crowdsourced data from citizen-led projects do not typically use a spatial sampling method so classical geostatistical techniques may not seamlessly be applied. Standard aggregation and interpolation methods were adapted to represent variance, sampling patterns, and the reliability of modeled trends. Finally, a Bayesian approach was used to model the spatial distribution of crowdsourced radiation measurements around Fukushima and quantify uncertainty introduced by the spatial data characteristics. Bayesian kriging of the crowdsourced data captures hotspots and the probabilistic approach could provide timely contextualized information that can improve situational awareness during hazards. This paper calls for the development of methods and metrics to clearly communicate spatial uncertainty by evaluating data characteristics, representing observational gaps and model error, and providing probabilistic outputs for decision making.

The Role of Translation in Citizen Science to Foster Social Innovation

Citizen science has become a world-wide phenomenon. Especially for citizen science projects that have a global reach, translation is crucial to overcome language and cultural barriers to reach members of the public. Translation, understood as the transfer of meaning (of a text) from one language into another language, is crucial for the transmission of information, knowledge and (social) innovations. Therefore, this paper examines the role of translation and terminology used in citizen science projects and how translation can foster (or impede) social innovation through citizen science activities. Based on a set of predefined criteria derived from the social innovation literature, this paper analyzes the factors that contribute to (social) innovation in citizen science by means of translation. A specific focus of the case study is on the aspects of agency, institutions, and social systems. The results demonstrate that translation in citizen science may support a change of social practices as ingredients of social innovations. Additional research is needed to further understand the implications of translation in citizen science and its effects on social innovation. Nevertheless, this work has been one of the first attempts to examine the relation between translation, citizen science and social innovation.

IL-13Rα2 gene expression is a biomarker of adverse outcome in patients with adrenocortical carcinoma

Adrenocortical carcinoma (ACC) is a rare but aggressive endocrine malignancy that usually results in a fatal outcome. To allow the better clinical management and reduce mortality, we searched for clinical and molecular markers that are reliable predictor of disease severity and clinical outcome in ACC patients. We determined a correlation between the overexpression of IL-13Rα2 and the clinical outcome in ACC patients using comprehensive data available in The Cancer Genome Atlas (TCGA) database. The dataset of 79 ACC subjects were divided into groups of low, medium, or high expression of IL-13Rα2 as determined by RNA-seq. These patients were also stratified by length of survival, overall survival, incidence of a new tumor event, incidence of metastasis, and production of excess hormones. We report a correlation between IL-13Rα2 expression and survival of subjects with ACC. High expression of IL-13Rα2 in ACC tumors was significantly associated with a lower patient survival rate and period of survival compared to low expression (p = 0.0084). In addition, high IL-13Rα2 expression was significantly associated with a higher incidence of new tumor events and excess hormone production compared to low or medium IL-13Rα2 expression. Within the cohort of patients that produced excess hormone, elevated IL-13Rα2 expression was significantly associated with a lower survival rate. Additionally, IL-13Rα1 had a potential relationship between transcript level and ACC survival. Our results and promising antitumor activity in preclinical models and trials indicate that IL-13Rα2 expression is an important prognostic biomarker of ACC disease outcome and a promising target for therapeutic treatment of ACC.

About the Pitfall of Erroneous Validation Data in the Estimation of Confusion Matrices

Accuracy assessment of maps relies on the collection of validation data, i.e., a set of trusted points or spatial objects collected independently from the classified map. However, collecting spatially and thematically accurate dataset is often tedious and expensive. Despite good practices, those datasets are rarely error-prone. Errors in the reference dataset propagate to the probabilities estimated in the confusion matrices. Consequently, the estimates of the quality are biased: accuracy indices are overestimated if the errors are correlated and underestimated if the errors are conditionally independent. The first findings of our study highlight the fact that this bias could invalidate statistical tests of map accuracy assessment. Furthermore, correlated errors in the reference dataset induce unfair comparison of classifiers. A maximum entropy method is thus proposed to mitigate the propagation of errors from imperfect reference datasets. The proposed method is based on a theoretical framework which considers a trivariate probability table that links the observed confusion matrix, the confusion matrix of the reference dataset and the “real” confusion matrix. The method was tested with simulated thematic and geo-reference errors. It proved to reduce the bias to the level of the sampling uncertainty. The method was very efficient with geolocation errors because conditional independence of errors can reasonably be assumed. Thematic errors are more difficult to mitigate because they require the estimation of an additional parameter related to the amount of spatial correlation. In any case, while collecting additional trusted labels is usually expensive, our result show that the benefits for accuracy assessment are much larger than collecting a larger number of questionable reference data.

Understanding Completeness and Diversity Patterns of OSM-Based Land-Use and Land-Cover Dataset in China

OpenStreetMap (OSM) data are considered essential for land-use and land-cover (LULC) mapping despite their lack of quality. Most relevant studies have employed an LULC reference dataset for quality assessment, but such a reference dataset is not freely available for most countries and regions. Thus, this study conducts an intrinsic quality assessment of the OSM-based LULC dataset (i.e., without using a reference LULC dataset) by examining the patterns of both its completeness and diversity. With China chosen as the study area, an OSM-based LULC dataset of the country was first generated and validated by using various accuracy measures. Both its completeness and diversity patterns were then mapped and analyzed in terms of each prefecture-level division of the country. The results showed the following: (1) While the overall accuracy was as high as 82.2%, most complete regions of China were not mapped well owing to a lack of diverse LULC classes. (2) In terms of socioeconomic factors and the number of contributors, higher correlations were noted for diversity patterns than completeness patterns; thus, the diversity pattern is a better reflection of socioeconomic factors and the spatial patterns of contributors. (3) Both the completeness and the diversity patterns can be combined to better understand an OSM-based LULC dataset. These results indicate that it is useful to consider diversity as a supplement for intrinsically assessing the quality of an OSM-based LULC dataset. This analytical method can also be applied to other countries and regions.

An Integrated Toolbox for the Engagement of Citizens in the Monitoring of Water Ecosystems

The monitoring of water ecosystems requires consistent and accurate sensor measurements, usually provided from traditional in-situ environmental monitoring systems. Such infrastructure, however, is expensive, hard to maintain and available only in limited areas that had been affected by extreme phenomena and require continuous monitoring. Due to climate change, the monitoring of larger areas and extended water ecosystems is imperative, raising the question of whether this monitoring can be disengaged from the in-situ monitoring systems. Due to climate change and extreme weather phenomena, more citizens are affected by environmental issues and become aware of the need to contribute to their monitoring. As a result, they are willing to offer their time to support the collection of scientific data. Collecting such data from volunteers, with no technical knowledge and while using low-cost equipment such as smart phones and portable sensors, raises the question of data quality and consistency. We present here a novel integrated toolbox that can support the organization of crowd-sourcing activities, ensure the engagement of the participants, the data collection in a consistent way, enforce extensive data quality controls and provide to local authorities and scientists access to the collected information in a uniform way, through widely accepted standards.

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.

How Response Designs and Class Proportions Affect the Accuracy of Validation Data

Reference data collected to validate land-cover maps are generally considered free of errors. In practice, however, they contain errors despite best efforts to minimize them. These errors propagate during accuracy assessment and tweak the validation results. For photo-interpreted reference data, the two most widely studied sources of error are systematic incorrect labeling and vigilance drops. How estimation errors, i.e., errors intrinsic to the response design, affect the accuracy of reference data is far less understood. In this paper, we analyzed the impact of estimation errors for two types of classification systems (binary and multiclass) as well as for two common response designs (point-based and partition-based) with a range of sub-sample sizes. Our quantitative results indicate that labeling errors due to proportion estimations should not be neglected. They further confirm that the accuracy of response designs depends on the class proportions within the sampling units, with complex landscapes being more prone to errors. As a result, response designs where the number of sub-samples is predefined and fixed are inefficient. To guarantee high accuracy standards of validation data with minimum data collection effort, we propose a new method to adapt the number of sub-samples for each sample during the validation process. In practice, sub-samples are incrementally selected and labeled until the estimated class proportions reach the desired level of confidence. As a result, less effort is spent on labeling univocal cases and the spared effort can be allocated to more ambiguous cases. This increases the reliability of reference data and of subsequent accuracy assessment. Across our study site, we demonstrated that such an approach could reduce the labeling effort by 50% to 75%, with greater gains in homogeneous landscapes. We contend that adopting this optimization approach will not only increase the efficiency of reference data collection, but will also help deliver more reliable accuracy estimates to the user community.

The CrowdWater game: A playful way to improve the accuracy of crowdsourced water level class data

Data quality control is important for any data collection program, especially in citizen science projects, where it is more likely that errors occur due to the human factor. Ideally, data quality control in citizen science projects is also crowdsourced so that it can handle large amounts of data. Here we present the CrowdWater game as a gamified method to check crowdsourced water level class data that are submitted by citizen scientists through the CrowdWater app. The app uses a virtual staff gauge approach, which means that a digital scale is added to the first picture taken at a site and this scale is used for water level class observations at different times. In the game, participants classify water levels based on the comparison of the new picture with the picture containing the virtual staff gauge. By March 2019, 153 people had played the CrowdWater game and 841 pictures were classified. The average water level for the game votes for the classified pictures was compared to the water level class submitted through the app to determine whether the game can improve the quality of the data submitted through the app. For about 70% of the classified pictures, the water level class was the same for the CrowdWater app and game. For a quarter of the classified pictures, there was disagreement between the value submitted through the app and the average game vote. Expert judgement suggests that for three quarters of these cases, the game based average value was correct. The initial results indicate that the CrowdWater game helps to identify erroneous water level class observations from the CrowdWater app and provides a useful approach for crowdsourced data quality control. This study thus demonstrates the potential of gamified approaches for data quality control in citizen science projects.

Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students

Crowdsourcing in pathology has been performed on tasks that are assumed to be manageable by nonexperts. Demand remains high for annotations of more complex elements in digital microscopic images, such as anatomical structures. Therefore, this paper investigates conditions to enable crowdsourced annotations of high-level image objects, a complex task considered to require expert knowledge. Seventy six medical students without specific domain knowledge who voluntarily participated in three experiments solved two relevant annotation tasks on histopathological images: 1) labeling of images showing tissue regions and 2) delineation of morphologically defined image objects. We focus on methods to ensure sufficient annotation quality including several tests on the required number of participants and on the correlation of participants' performance between tasks. In a set up simulating annotation of images with limited ground truth, we validated the feasibility of a confidence score using full ground truth. For this, we computed a majority vote using weighting factors based on individual assessment of contributors against scattered gold standard annotated by pathologists. In conclusion, we provide guidance for task design and quality control to enable a crowdsourced approach to obtain accurate annotations required in the era of digital pathology.

Producing knowledge by admitting ignorance: Enhancing data quality through an "I don't know" option in citizen science

The "noisy labeler problem" in crowdsourced data has attracted great attention in recent years, with important ramifications in citizen science, where non-experts must produce high-quality data. Particularly relevant to citizen science is dynamic task allocation, in which the level of agreement among labelers can be progressively updated through the information-theoretic notion of entropy. Under dynamic task allocation, we hypothesized that providing volunteers with an "I don't know" option would contribute to enhancing data quality, by introducing further, useful information about the level of agreement among volunteers. We investigated the influence of an "I don't know" option on the data quality in a citizen science project that entailed classifying the image of a highly polluted canal into "threat" or "no threat" to the environment. Our results show that an "I don't know" option can enhance accuracy, compared to the case without the option; such an improvement mostly affects the true negative rather than the true positive rate. In an information-theoretic sense, these seemingly meaningless blank votes constitute a meaningful piece of information to help enhance accuracy of data in citizen science.

The value of OpenStreetMap Historical Contributions as a Source of Sampling Data for Multi‑temporal Land Use/Cover Maps

OpenStreetMap (OSM) is a free, open-access Volunteered geographic information (VGI) platform that has been widely used over the last decade as a source for Land Use Land Cover (LULC) mapping and visualization. However, it is known that the spatial coverage and accuracy of OSM data are not evenly distributed across all regions, with urban areas being likelier to have promising contributions (in both quantity and quality) than rural areas. The present study used OSM data history to generate LULC datasets with one-year timeframes as a way to support regional and rural multi-temporal LULC mapping. We evaluated the degree to which the different OSM datasets agreed with two existing reference datasets (CORINE Land Cover and the official Portuguese Land Cover Map). We also evaluated whether our OSM dataset was of sufficiently high quality (in terms of both completeness accuracy and thematic accuracy) to be used as a sampling data source for multi-temporal LULC maps. In addition, we used the near boundary tag accuracy criterion to assesses the fitness of the OSM data for producing training samples, with promising results. For each annual dataset, the completeness ratio of the coverage area for the selected study area was low. Nevertheless, we found high thematic accuracy values (ranged from 77.3% to 91.9%). Additionally, the training samples thematic accuracy improved as they moved away from the features’ boundaries. Features with larger areas (> 10 ha), e.g., Agriculture and Forest, had a steadily positive correlation between training samples accuracy and distance to feature boundaries

Estimating the global distribution of field size using crowdsourcing

There is an increasing evidence that smallholder farms contribute substantially to food production globally, yet spatially explicit data on agricultural field sizes are currently lacking. Automated field size delineation using remote sensing or the estimation of average farm size at subnational level using census data are two approaches that have been used. However, both have limitations, for example, automatic field size delineation using remote sensing has not yet been implemented at a global scale while the spatial resolution is very coarse when using census data. This paper demonstrates a unique approach to quantifying and mapping agricultural field size globally using crowdsourcing. A campaign was run in June 2017, where participants were asked to visually interpret very high resolution satellite imagery from Google Maps and Bing using the Geo-Wiki application. During the campaign, participants collected field size data for 130 K unique locations around the globe. Using this sample, we have produced the most accurate global field size map to date and estimated the percentage of different field sizes, ranging from very small to very large, in agricultural areas at global, continental, and national levels. The results show that smallholder farms occupy up to 40% of agricultural areas globally, which means that, potentially, there are many more smallholder farms in comparison with the two different current global estimates of 12% and 24%. The global field size map and the crowdsourced data set are openly available and can be used for integrated assessment modeling, comparative studies of agricultural dynamics across different contexts, for training and validation of remote sensing field size delineation, and potential contributions to the Sustainable Development Goal of Ending hunger, achieve food security and improved nutrition and promote sustainable agriculture.

News coverage, digital activism, and geographical saliency: A case study of refugee camps and volunteered geographical information

The last several decades have witnessed a shift in the way in which news is delivered and consumed by users. With the growth and advancements in mobile technologies, the Internet, and Web 2.0 technologies users are not only consumers of news, but also producers of online content. This has resulted in a novel and highly participatory cyber-physical news awareness ecosystem that fosters digital activism, in which volunteers contribute content to online communities. While studies have examined the various components of this news awareness ecosystem, little is still known about how news media coverage (and in particular digital media) impacts digital activism. In order to address this challenge and develop a greater understanding of it, this paper focuses on a specific form of digital activism, that of the production of digital geographical content through crowdsourcing efforts. Using refugee camps from around the world as a case study, we examine the relationship between news coverage (via Google news), search trends (via Google trends) and user edit contribution patterns in OpenStreetMap, a prominent geospatial data crowdsourcing platform. In addition, we compare and contrast these patterns with user edit patterns in Wikipedia, a well-known non-geospatial crowdsourcing platform. Using Google news and Google trends to derive a measure of thematic public awareness, our findings indicate that digital activism bursts tend to take place during periods of sustained build-up of public awareness deficit or surplus. These findings are in line with two prominent mass communication theories: agenda setting and corrective action, and suggest the emergence of a novel stimulus-awareness-activism framework in today's participatory digital age. Moreover, these findings further complement existing research examining the motivational factors that drive users to contribute to online collaborative communities. This paper brings us one step closer to understanding the underlying mechanisms that drive digital activism in particular in the geospatial domain.

Using Open Street Map Data in Environmental Exposure Assessment Studies: Eastern Massachusetts, Bern Region, and South Israel as a Case Study

There is an increase in the awareness of the importance of spatial data in epidemiology and exposure assessment (EA) studies. Most studies use governmental and ordnance surveys, which are often expensive and sparsely updated, while in most developing countries, there are often no official geo-spatial data sources. OpenStreetMap (OSM) is an open source Volunteered Geographic Information (VGI) mapping project. Yet very few environmental epidemiological and EA studies have used OSM as a source for road data. Since VGI data is either noncommercial or governmental, the validity of OSM is often questioned. We investigate the robustness and validity of OSM data for use in epidemiological and EA studies. We compared OSM and Governmental Major Road Data (GRD) in three different regions: Massachusetts, USA; Bern, Switzerland; and Beer-Sheva, South Israel. The comparison was done by calculating data completeness, positional accuracy, and EA using traditional exposure methods. We found that OSM data is fairly complete and accurate in all regions. The results in all regions were robust, with Massachusetts showing the best fits (R² 0.93). Results in Bern (R² 0.78) and Beer-Sheva (R² 0.77) were only slightly lower. We conclude by suggesting that OSM data can be used reliably in environmental assessment studies.

Feature Extraction and Selection of Sentinel-1 Dual-Pol Data for Global-Scale Local Climate Zone Classification

The concept of the local climate zone (LCZ) has been recently proposed as a generic land-cover/land-use classification scheme. It divides urban regions into 17 categories based on compositions of man-made structures and natural landscapes. Although it was originally designed for temperature study, the morphological structure concealed in LCZs also reflects economic status and population distribution. To this end, global LCZ classification is of great value for worldwide studies on economy and population. Conventional classification approaches are usually successful for an individual city using optical remote sensing data. This paper, however, attempts for the first time to produce global LCZ classification maps using polarimetric synthetic aperture radar (PolSAR) data. Specifically, we first produce polarimetric features, local statistical features, texture features, and morphological features and compare them, with respect to their classification performance. Here, an ensemble classifier is investigated, which is trained and tested on already separated transcontinental cities. Considering the challenging global scope this work handles, we conclude the classification accuracy is not yet satisfactory. However, Sentinel-1 dual-Pol SAR data could contribute the classification for several LCZ classes. According to our feature studies, the combination of local statistical features and morphological features yields the best classification results with 61.8% overall accuracy (OA), which is 3% higher than the OA produced by the second best features combination. The 3% is considerably large for a global scale. Based on our feature importance analysis, features related to VH polarized data contributed the most to the eventual classification result.

A taxonomy of quality assessment methods for volunteered and crowdsourced geographic information

The growing use of crowdsourced geographic information (CGI) has prompted the employment of several methods for assessing information quality, which are aimed at addressing concerns on the lack of quality of the information provided by non-experts. In this work, we propose a taxonomy of methods for assessing the quality of CGI when no reference data are available, which is likely to be the most common situation in practice. Our taxonomy includes 11 quality assessment methods that were identified by means of a systematic literature review. These methods are described in detail, including their main characteristics and limitations. This taxonomy not only provides a systematic and comprehensive account of the existing set of methods for CGI quality assessment, but also enables researchers working on the quality of CGI in various sources (e.g., social media, crowd sensing, collaborative mapping) to learn from each other, thus opening up avenues for future work that combines and extends existing methods into new application areas and domains.

A global dataset of crowdsourced land cover and land use reference data

Global land cover is an essential climate variable and a key biophysical driver for earth system models. While remote sensing technology, particularly satellites, have played a key role in providing land cover datasets, large discrepancies have been noted among the available products. Global land use is typically more difficult to map and in many cases cannot be remotely sensed. In-situ or ground-based data and high resolution imagery are thus an important requirement for producing accurate land cover and land use datasets and this is precisely what is lacking. Here we describe the global land cover and land use reference data derived from the Geo-Wiki crowdsourcing platform via four campaigns. These global datasets provide information on human impact, land cover disagreement, wilderness and land cover and land use. Hence, they are relevant for the scientific community that requires reference data for global satellite-derived products, as well as those interested in monitoring global terrestrial ecosystems in general.

eFarm: A Tool for Better Observing Agricultural Land Systems

Currently, observations of an agricultural land system (ALS) largely depend on remotely-sensed images, focusing on its biophysical features. While social surveys capture the socioeconomic features, the information was inadequately integrated with the biophysical features of an ALS and the applications are limited due to the issues of cost and efficiency to carry out such detailed and comparable social surveys at a large spatial coverage. In this paper, we introduce a smartphone-based app, called eFarm: a crowdsourcing and human sensing tool to collect the geotagged ALS information at the land parcel level, based on the high resolution remotely-sensed images. We illustrate its main functionalities, including map visualization, data management, and data sensing. Results of the trial test suggest the system works well. We believe the tool is able to acquire the human-land integrated information which is broadly-covered and timely-updated, thus presenting great potential for improving sensing, mapping, and modeling of ALS studies.

Citizen surveillance for environmental monitoring: combining the efforts of citizen science and crowdsourcing in a quantitative data framework

Citizen science and crowdsourcing have been emerging as methods to collect data for surveillance and/or monitoring activities. They could be gathered under the overarching term citizen surveillance. The discipline, however, still struggles to be widely accepted in the scientific community, mainly because these activities are not embedded in a quantitative framework. This results in an ongoing discussion on how to analyze and make useful inference from these data. When considering the data collection process, we illustrate how citizen surveillance can be classified according to the nature of the underlying observation process measured in two dimensions—the degree of observer reporting intention and the control in observer detection effort. By classifying the observation process in these dimensions we distinguish between crowdsourcing, unstructured citizen science and structured citizen science. This classification helps the determine data processing and statistical treatment of these data for making inference. Using our framework, it is apparent that published studies are overwhelmingly associated with structured citizen science, and there are well developed statistical methods for the resulting data. In contrast, methods for making useful inference from purely crowd-sourced data remain under development, with the challenges of accounting for the unknown observation process considerable. Our quantitative framework for citizen surveillance calls for an integration of citizen science and crowdsourcing and provides a way forward to solve the statistical challenges inherent to citizen-sourced data.

Crowdsourcing: It Matters Who the Crowd Are. The Impacts of between Group Variations in Recording Land Cover

Volunteered geographical information (VGI) and citizen science have become important sources data for much scientific research. In the domain of land cover, crowdsourcing can provide a high temporal resolution data to support different analyses of landscape processes. However, the scientists may have little control over what gets recorded by the crowd, providing a potential source of error and uncertainty. This study compared analyses of crowdsourced land cover data that were contributed by different groups, based on nationality (labelled Gondor and Non-Gondor) and on domain experience (labelled Expert and Non-Expert). The analyses used a geographically weighted model to generate maps of land cover and compared the maps generated by the different groups. The results highlight the differences between the maps how specific land cover classes were under- and over-estimated. As crowdsourced data and citizen science are increasingly used to replace data collected under the designed experiment, this paper highlights the importance of considering between group variations and their impacts on the results of analyses. Critically, differences in the way that landscape features are conceptualised by different groups of contributors need to be considered when using crowdsourced data in formal scientific analyses. The discussion considers the potential for variation in crowdsourced data, the relativist nature of land cover and suggests a number of areas for future research. The key finding is that the veracity of citizen science data is not the critical issue per se. Rather, it is important to consider the impacts of differences in the semantics, affordances and functions associated with landscape features held by different groups of crowdsourced data contributors.

"Anyone Know What Species This Is?" - Twitter Conversations as Embryonic Citizen Science Communities

Social media like blogs, micro-blogs or social networks are increasingly being investigated and employed to detect and predict trends for not only social and physical phenomena, but also to capture environmental information. Here we argue that opportunistic biodiversity observations published through Twitter represent one promising and until now unexplored example of such data mining. As we elaborate, it can contribute to real-time information to traditional ecological monitoring programmes including those sourced via citizen science activities. Using Twitter data collected for a generic assessment of social media data in ecological monitoring we investigated a sample of what we denote biodiversity observations with species determination requests (N = 191). These entail images posted as messages on the micro-blog service Twitter. As we show, these frequently trigger conversations leading to taxonomic determinations of those observations. All analysed Tweets were posted with species determination requests, which generated replies for 64% of Tweets, 86% of those contained at least one suggested determination, of which 76% were assessed as correct. All posted observations included or linked to images with the overall image quality categorised as satisfactory or better for 81% of the sample and leading to taxonomic determinations at the species level in 71% of provided determinations. We claim that the original message authors and conversation participants can be viewed as implicit or embryonic citizen science communities which have to offer valuable contributions both as an opportunistic data source in ecological monitoring as well as potential active contributors to citizen science programmes.

Developing a Workflow to Identify Inconsistencies in Volunteered Geographic Information: A Phenological Case Study

Recent improvements in online information communication and mobile location-aware technologies have led to the production of large volumes of volunteered geographic information. Widespread, large-scale efforts by volunteers to collect data can inform and drive scientific advances in diverse fields, including ecology and climatology. Traditional workflows to check the quality of such volunteered information can be costly and time consuming as they heavily rely on human interventions. However, identifying factors that can influence data quality, such as inconsistency, is crucial when these data are used in modeling and decision-making frameworks. Recently developed workflows use simple statistical approaches that assume that the majority of the information is consistent. However, this assumption is not generalizable, and ignores underlying geographic and environmental contextual variability that may explain apparent inconsistencies. Here we describe an automated workflow to check inconsistency based on the availability of contextual environmental information for sampling locations. The workflow consists of three steps: (1) dimensionality reduction to facilitate further analysis and interpretation of results, (2) model-based clustering to group observations according to their contextual conditions, and (3) identification of inconsistent observations within each cluster. The workflow was applied to volunteered observations of flowering in common and cloned lilac plants (Syringa vulgaris and Syringa x chinensis) in the United States for the period 1980 to 2013. About 97% of the observations for both common and cloned lilacs were flagged as consistent, indicating that volunteers provided reliable information for this case study. Relative to the original dataset, the exclusion of inconsistent observations changed the apparent rate of change in lilac bloom dates by two days per decade, indicating the importance of inconsistency checking as a key step in data quality assessment for volunteered geographic information. Initiatives that leverage volunteered geographic information can adapt this workflow to improve the quality of their datasets and the robustness of their scientific analyses.

Can Observation Skills of Citizen Scientists Be Estimated Using Species Accumulation Curves?

Volunteers are increasingly being recruited into citizen science projects to collect observations for scientific studies. An additional goal of these projects is to engage and educate these volunteers. Thus, there are few barriers to participation resulting in volunteer observers with varying ability to complete the project's tasks. To improve the quality of a citizen science project's outcomes it would be useful to account for inter-observer variation, and to assess the rarely tested presumption that participating in a citizen science projects results in volunteers becoming better observers. Here we present a method for indexing observer variability based on the data routinely submitted by observers participating in the citizen science project eBird, a broad-scale monitoring project in which observers collect and submit lists of the bird species observed while birding. Our method for indexing observer variability uses species accumulation curves, lines that describe how the total number of species reported increase with increasing time spent in collecting observations. We find that differences in species accumulation curves among observers equates to higher rates of species accumulation, particularly for harder-to-identify species, and reveals increased species accumulation rates with continued participation. We suggest that these properties of our analysis provide a measure of observer skill, and that the potential to derive post-hoc data-derived measurements of participant ability should be more widely explored by analysts of data from citizen science projects. We see the potential for inferential results from analyses of citizen science data to be improved by accounting for observer skill.

Spatial and temporal epidemiological analysis in the Big Data era

Concurrent with global economic development in the last 50 years, the opportunities for the spread of existing diseases and emergence of new infectious pathogens, have increased substantially. The activities associated with the enormously intensified global connectivity have resulted in large amounts of data being generated, which in turn provides opportunities for generating knowledge that will allow more effective management of animal and human health risks. This so-called Big Data has, more recently, been accompanied by the Internet of Things which highlights the increasing presence of a wide range of sensors, interconnected via the Internet. Analysis of this data needs to exploit its complexity, accommodate variation in data quality and should take advantage of its spatial and temporal dimensions, where available. Apart from the development of hardware technologies and networking/communication infrastructure, it is necessary to develop appropriate data management tools that make this data accessible for analysis. This includes relational databases, geographical information systems and most recently, cloud-based data storage such as Hadoop distributed file systems. While the development in analytical methodologies has not quite caught up with the data deluge, important advances have been made in a number of areas, including spatial and temporal data analysis where the spectrum of analytical methods ranges from visualisation and exploratory analysis, to modelling. While there used to be a primary focus on statistical science in terms of methodological development for data analysis, the newly emerged discipline of data science is a reflection of the challenges presented by the need to integrate diverse data sources and exploit them using novel data- and knowledge-driven modelling methods while simultaneously recognising the value of quantitative as well as qualitative analytical approaches. Machine learning regression methods, which are more robust and can handle large datasets faster than classical regression approaches, are now also used to analyse spatial and spatio-temporal data. Multi-criteria decision analysis methods have gained greater acceptance, due in part, to the need to increasingly combine data from diverse sources including published scientific information and expert opinion in an attempt to fill important knowledge gaps. The opportunities for more effective prevention, detection and control of animal health threats arising from these developments are immense, but not without risks given the different types, and much higher frequency, of biases associated with these data.

Sources of spatial animal and human health data: Casting the net wide to deal more effectively with increasingly complex disease problems

During the last 30years it has become commonplace for epidemiological studies to collect locational attributes of disease data. Although this advancement was driven largely by the introduction of handheld global positioning systems (GPS), and more recently, smartphones and tablets with built-in GPS, the collection of georeferenced disease data has moved beyond the use of handheld GPS devices and there now exist numerous sources of crowdsourced georeferenced disease data such as that available from georeferencing of Google search queries or Twitter messages. In addition, cartography has moved beyond the realm of professionals to crowdsourced mapping projects that play a crucial role in disease control and surveillance of outbreaks such as the 2014 West Africa Ebola epidemic. This paper provides a comprehensive review of a range of innovative sources of spatial animal and human health data including data warehouses, mHealth, Google Earth, volunteered geographic information and mining of internet-based big data sources such as Google and Twitter. We discuss the advantages, limitations and applications of each, and highlight studies where they have been used effectively.

Understanding human--coyote encounters in urban ecosystems using citizen science data: what do socioeconomics tell us?

The coyote (Canis latrans) has dramatically expanded its range to include the cities and suburbs of the western US and those of the Eastern Seaboard. Highly adaptable, this newcomer's success causes conflicts with residents, necessitating research to understand the distribution of coyotes in urban landscapes. Citizen science can be a powerful approach toward this aim. However, to date, the few studies that have used publicly reported coyote sighting data have lacked an in-depth consideration of human socioeconomic variables, which we suggest are an important source of overlooked variation in data that describe the simultaneous occurrence of coyotes and humans. We explored the relative importance of socioeconomic variables compared to those describing coyote habitat in predicting human-coyote encounters in highly-urbanized Mecklenburg County, North Carolina, USA using 707 public reports of coyote sightings, high-resolution land cover, US Census data, and an autologistic multi-model inference approach. Three of the four socioeconomic variables which we hypothesized would have an important influence on encounter probability, namely building density, household income, and occupation, had effects at least as large as or larger than coyote habitat variables. Our results indicate that the consideration of readily available socioeconomic variables in the analysis of citizen science data improves the prediction of species distributions by providing insight into the effects of important factors for which data are often lacking, such as resource availability for coyotes on private property and observer experience. Managers should take advantage of citizen scientists in human-dominated landscapes to monitor coyotes in order to understand their interactions with humans.

The public health exposome: a population-based, exposure science approach to health disparities research

The lack of progress in reducing health disparities suggests that new approaches are needed if we are to achieve meaningful, equitable, and lasting reductions. Current scientific paradigms do not adequately capture the complexity of the relationships between environment, personal health and population level disparities. The public health exposome is presented as a universal exposure tracking framework for integrating complex relationships between exogenous and endogenous exposures across the lifespan from conception to death. It uses a social-ecological framework that builds on the exposome paradigm for conceptualizing how exogenous exposures "get under the skin". The public health exposome approach has led our team to develop a taxonomy and bioinformatics infrastructure to integrate health outcomes data with thousands of sources of exogenous exposure, organized in four broad domains: natural, built, social, and policy environments. With the input of a transdisciplinary team, we have borrowed and applied the methods, tools and terms from various disciplines to measure the effects of environmental exposures on personal and population health outcomes and disparities, many of which may not manifest until many years later. As is customary with a paradigm shift, this approach has far reaching implications for research methods and design, analytics, community engagement strategies, and research training.

Measuring long-term impact based on network centrality: unraveling cinematic citations

Traditional measures of success for film, such as box-office revenue and critical acclaim, lack the ability to quantify long-lasting impact and depend on factors that are largely external to the craft itself. With the growing number of films that are being created and large-scale data becoming available through crowd-sourced online platforms, an endogenous measure of success that is not reliant on manual appraisal is of increasing importance. In this article we propose such a ranking method based on a combination of centrality indices. We apply the method to a network that contains several types of citations between more than 40,000 international feature films. From this network we derive a list of milestone films, which can be considered to constitute the foundations of cinema. In a comparison to various existing lists of ‘greatest’ films, such as personal favourite lists, voting lists, lists of individual experts, and lists deduced from expert polls, the selection of milestone films is more diverse in terms of genres, actors, and main creators. Our results shed light on the potential of a systematic quantitative investigation based on cinematic influences in identifying the most inspiring creations in world cinema. In a broader perspective, we introduce a novel research question to large-scale citation analysis, one of the most intriguing topics that have been at the forefront of scientific enquiries for the past fifty years and have led to the development of various network analytic methods. In doing so, we transfer widely studied approaches from citation analysis to the the newly emerging field of quantification efforts in the arts. The specific contribution of this paper consists in modelling the multidimensional cinematic references as a growing multiplex network and in developing a methodology for the identification of central films in this network.

Comparing forests across climates and biomes: qualitative assessments, reference forests and regional intercomparisons

Communities, policy actors and conservationists benefit from understanding what institutions and land management regimes promote ecosystem services like carbon sequestration and biodiversity conservation. However, the definition of success depends on local conditions. Forests' potential carbon stock, biodiversity and rate of recovery following disturbance are known to vary with a broad suite of factors including temperature, precipitation, seasonality, species' traits and land use history. Methods like tracking over-time changes within forests, or comparison with "pristine" reference forests have been proposed as means to compare the structure and biodiversity of forests in the face of underlying differences. However, data from previous visits or reference forests may be unavailable or costly to obtain. Here, we introduce a new metric of locally weighted forest intercomparison to mitigate the above shortcomings. This method is applied to an international database of nearly 300 community forests and compared with previously published techniques. It is particularly suited to large databases where forests may be compared among one another. Further, it avoids problematic comparisons with old-growth forests which may not resemble the goal of forest management. In most cases, the different methods produce broadly congruent results, suggesting that researchers have the flexibility to compare forest conditions using whatever type of data is available. Forest structure and biodiversity are shown to be independently measurable axes of forest condition, although users' and foresters' estimations of seemingly unrelated attributes are highly correlated, perhaps reflecting an underlying sentiment about forest condition. These findings contribute new tools for large-scale analysis of ecosystem condition and natural resource policy assessment. Although applied here to forestry, these techniques have broader applications to classification and evaluation problems using crowdsourced or repurposed data for which baselines or external validations are not available.