Validating predictions of burial mounds with field data: the promise and reality of machine learning

Abstract

Purpose

This paper provides practical advice for archaeologists and heritage specialists wishing to use ML approaches to identify archaeological features in high-resolution satellite imagery (or other remotely sensed data sources). We seek to balance the disproportionately optimistic literature related to the application of ML to archaeological prospection through a discussion of limitations, challenges and other difficulties. We further seek to raise awareness among researchers of the time, effort, expertise and resources necessary to implement ML successfully, so that they can make an informed choice between ML and manual inspection approaches.

Design/methodology/approach

Automated object detection has been the holy grail of archaeological remote sensing for the last two decades. Machine learning (ML) models have proven able to detect uniform features across a consistent background, but more variegated imagery remains a challenge. We set out to detect burial mounds in satellite imagery from a diverse landscape in Central Bulgaria using a pre-trained Convolutional Neural Network (CNN) plus additional but low-touch training to improve performance. Training was accomplished using MOUND/NOT MOUND cutouts, and the model assessed arbitrary tiles of the same size from the image. Results were assessed using field data.

Findings

Validation of results against field data showed that self-reported success rates were misleadingly high, and that the model was misidentifying most features. Setting an identification threshold at 60% probability, and noting that we used an approach where the CNN assessed tiles of a fixed size, tile-based false negative rates were 95–96%, false positive rates were 87–95% of tagged tiles, while true positives were only 5–13%. Counterintuitively, the model provided with training data selected for highly visible mounds (rather than all mounds) performed worse. Development of the model, meanwhile, required approximately 135 person-hours of work.

Research limitations/implications

Our attempt to deploy a pre-trained CNN demonstrates the limitations of this approach when it is used to detect varied features of different sizes within a heterogeneous landscape that contains confounding natural and modern features, such as roads, forests and field boundaries. The model has detected incidental features rather than the mounds themselves, making external validation with field data an essential part of CNN workflows. Correcting the model would require refining the training data as well as adopting different approaches to model choice and execution, raising the computational requirements beyond the level of most cultural heritage practitioners.

Practical implications

Improving the pre-trained model’s performance would require considerable time and resources, on top of the time already invested. The degree of manual intervention required – particularly around the subsetting and annotation of training data – is so significant that it raises the question of whether it would be more efficient to identify all of the mounds manually, either through brute-force inspection by experts or by crowdsourcing the analysis to trained – or even untrained – volunteers. Researchers and heritage specialists seeking efficient methods for extracting features from remotely sensed data should weigh the costs and benefits of ML versus manual approaches carefully.

Social implications

Our literature review indicates that use of artificial intelligence (AI) and ML approaches to archaeological prospection have grown exponentially in the past decade, approaching adoption levels associated with “crossing the chasm” from innovators and early adopters to the majority of researchers. The literature itself, however, is overwhelmingly positive, reflecting some combination of publication bias and a rhetoric of unconditional success. This paper presents the failure of a good-faith attempt to utilise these approaches as a counterbalance and cautionary tale to potential adopters of the technology. Early-majority adopters may find ML difficult to implement effectively in real-life scenarios.

Originality/value

Unlike many high-profile reports from well-funded projects, our paper represents a serious but modestly resourced attempt to apply an ML approach to archaeological remote sensing, using techniques like transfer learning that are promoted as solutions to time and cost problems associated with, e.g. annotating and manipulating training data. While the majority of articles uncritically promote ML, or only discuss how challenges were overcome, our paper investigates how – despite reasonable self-reported scores – the model failed to locate the target features when compared to field data. We also present time, expertise and resourcing requirements, a rarity in ML-for-archaeology publications.

Citation

Sobotkova, A., Kristensen-McLachlan, R.D., Mallon, O. and Ross, S.A. (2024), "Validating predictions of burial mounds with field data: the promise and reality of machine learning", Journal of Documentation, Vol. 80 No. 5, pp. 1167-1189. https://doi.org/10.1108/JD-05-2022-0096

Introduction

Machine learning (ML) has taken the world by a storm, and archaeologists too have tried to harness the power of Convolutional Neural Networks (CNN) and other “deep learning” approaches to extract archaeological features from remotely sensed data. A series of recent articles have promoted the success of CNNs in cultural heritage applications (Can et al., 2021; Caspari, 2020; Caspari and Crespo, 2019; Ekim et al., 2021; Ma et al., 2021). High success rates reported for detecting and monitoring burial mounds in Siberia have highlighted CNNs as an effective approach for large-scale prospection (Caspari and Crespo, 2019). Enthusiasm arising from this study, and similar outcomes from Egypt (Woolf, 2018) must, however, be tempered by the fact that the authors targeted uniform features situated in environments with little variation in terrain or vegetation – indeed, with relatively little vegetation or other confounding factors at all. Fewer studies explore the challenges presented by more difficult environments where cultural heritage lies in diverse or thick vegetation, surrounded by obtrusive natural and artificial features (Doyle et al., 2023; Fuentes-Carbajal et al., 2023; Verschoof-van der Vaart et al., 2020).

Although few publications report the time, expertise, or costs associated with applying ML to archaeological prospection, examples from projects trying to extract symbols and text from historical maps indicate that it can be labour-intensive (Can et al., 2021; Ekim et al., 2021; Ma et al., 2021). Correct classification of linear road features with a pre-trained model required 1,250 h to digitise and annotate training datasets (Can et al., 2021, p. 62,847). Without such investment, one can expect to detect only the most consistent map features (Groom et al., 2021), despite the fact that maps offer a simplified version of reality and use standardised symbols, which are far less complex than the earth’s surface as reflected in satellite imagery.

Projects applying ML approaches to satellite imagery or LiDAR data likewise report a lack of high-quality, well-annotated training data, often due to cost and time constraints (Albrecht et al., 2019; Argyrou and Agapiou, 2022; Canedo et al., 2023; Casini et al., 2021; Gallwey et al., 2019; Kadhim and Abed, 2023; Karamitrou et al., 2022, 2023; Lambers et al., 2019; Ofli et al., 2016; Sech et al., 2023; Verschoof-van der Vaart and Landauer, 2021). Transfer learning based on pre-trained models is sometimes proposed as solution to the problem of limited training data, as well as related problems like small dataset size (Casini et al., 2021, 2022; Character et al., 2021; Gallwey et al., 2019; Sech et al., 2023; Xiong et al., 2020). Use of a pre-trained CNN potentially obviates the need to have large, high-quality, and representative datasets for ML training, and promises to bring CNN approaches within the reach of smaller-scale projects, or projects that hope to detect or monitor varied archaeological phenomena in heterogeneous landscapes. Relatively few projects applying pre-trained models, however, offer a sustained discussion of failures, limitations, or shortcomings of the approach (for exceptions see Casini et al., 2023; Sech et al., 2023).

This paper offers a cautionary tale about the challenges, limitations, and demands of ML applied to archaeological prospection. We set out to detect burial mounds in the Kazanlak Valley, Bulgaria, using IKONOS high-resolution satellite imagery. We developed a pre-trained CNN that was further trained using two datasets: (1) image cutouts of areas where mounds were discovered during pedestrian fieldwork, regardless of how visible the mounds were in the satellite imagery, and (2) image cutouts where a burial mound is clearly visible in the imagery to a human observer. Each of these CNNs was then used to predict the locations of burial mounds in the imagery from the study area. We compare the predictions from the two models and their reported success rates measured against ground-truthed data. The results show that even a sophisticated, pre-trained model that is subjected to additional training struggles when confronted by inconsistent and sometimes indistinct features in a varied landscape. Indeed, both models failed to identify burial mounds in our study area. Expert researchers, or even novice volunteers, would have been more reliable (Sobotkova et al., 2023). As such, we offer corrective insights regarding the difficulties we faced deploying a pre-trained CNN, as well as suggestions about how archaeologists might improve ML approaches they choose to adopt.

Burial mounds as heritage under threat

Burial mounds are a ubiquitous feature of the Bulgarian landscape (Oltean, 2013; Škorpil, 1925). Thousands of such mounds exist in the country; estimates range between 8,000 – 19,000 surviving today, of perhaps 50,000 originally constructed (Kitov, 1993, pp. 41–43; Shkorpil and Shkorpil, 1989, p. 20). Such mounds were built from the Early Bronze Age through the Middle Ages in the western extensions of the Asian steppes and surrounding areas. Over time, burial mound agglomerations formed entire mounded landscapes. Often they dot elevated ridges to enhance their prominence. Where terrain is flat, mounds align in linear patterns to highlight subtle morphology of the surrounding landscapes and signal their presence to the passer-by Sobotkova and Weissova (2019).

These rounded, conical piles of earth and stones vary in diameter from 10 m to 100 m and <1 m to >20 m in height (see Plates 1 and 2). Their contents vary from nothing (cenotaphs), to simple burials with or without an enclosure, to elaborate stone or brick tombs with notable architectural and artistic refinement and intrinsically valuable grave goods (Dimitrova, 2006; Tsetskhladze, 1998; Vasileva, 2005).

Despite the large number of burial mounds, they are endangered. Development in Bulgaria destroys dozens of mounds annually (Loulanski and Loulanski, 2017). Most regulated destructions result from formal rescue excavation in anticipation of housing or infrastructure construction, establishment of quarries, or similar development. In 2008, the last year for which data is available, burial mounds comprised nearly a quarter (57 of 257) of all excavations in Bulgaria (Cholakov and Chukalev, 2008, p. 91, Figure 2).

Despite administrative and legal measures to combat illicit artefact trafficking, such as a 2013 Memorandum of Understanding with the United States, looting probably damages more mounds than development (Eftimoski et al., 2017; Lazar, 2015, pp. 107–124; Mateva, 2011). The Kazanlak Valley has been especially targeted by looters as it features an unusual concentration of conspicuous burial mounds (50–100 m in diameter) that include decorated ashlar tombs containing artwork, precious metals, high-quality ceramics, and other valuable objects (Dimitrova, 2006; Kitov, 1999). Although the majority of mounds in the valley are small in size and have modest contents, the potential payoff for looters, and failures of law and order, have fuelled looting since 1989 (Bailey, 1998).

Mounded landscapes also suffer slow and continuous wear from agricultural activities (Eftimoski et al., 2017; Sobotkova and Weissova, 2020). Farmers plough and harrow arable fields annually, affecting thousands of mounds across Bulgaria. Unlike looting and (to a lesser extent) development, which capture public attention, damage from agriculture generally goes unremarked. In the course of nearly 20 years of intermittent fieldwork in Bulgaria, the authors have seen few examples of mounds that had not been damaged either by development, looting, or agriculture, despite having inventoried over 2,000 of them across two Bulgarian provinces (Ross et al., 2010, 2018; Sobotkova and Weissova, 2020).

Detecting archaeological features in satellite imagery

While burial mounds are readily identifiable on the ground due to their distinctive appearance, their visibility in satellite imagery depends on their size, surrounding terrain, and local land cover (see Figure 1). Large mounds in flat landscapes where a mound’s surface contrasts with surrounding land cover (Figure 1a, b, e) are more visible than small mounds in hilly landscapes where their surfaces blend into the surroundings (Figure 1g-i). Human observers often look for crop, shadow, or soil marks to detect the less conspicuous mounds. Furthermore, mounds in forests can be obscured by the trees, while ploughed-over mounds blend into the surrounding fields. Visibility in satellite imagery is often marginal for small or even medium-sized mounds (Sobotkova and Ross, 2010). Conversely, image features such as lozenge-shaped clusters of trees in ploughed fields can be mistaken for mounds. When participants on the Tundzha Regional Archaeology Project (TRAP) previously attempted to detect mounds via manual inspection of satellite imagery, we encountered many false negatives, classification errors, and false positives. Failures, however, tended to be predictable, and some problems with manual inspection could be overcome by training and practice. Nevertheless, a field visit is usually required to confirm identification of a burial mound. After using satellite imagery for manual mound prospection since 2008, we were curious whether ML had, by 2022, reached the point where it could detect features as accurately and efficiently as a person.

Automated approaches to remotely sensed data

Mapping cultural heritage at the scale of a country is a costly task whether it involves walking the landscape, digitising features from legacy sources like historical maps, or manually inspecting aerial or satellite images (Brophy and Cowley, 2005; Contreras, 2010; Crawford, 1929; De Laet et al., 2007; De Laet and Lambers, 2009). Manual interpretation of remotely sensed imagery is not only tedious and time-consuming, but has limitations imposed by human bias and error (Bowen et al., 2017). Today, however, more and more remotely sensed datasets are becoming available, requiring scalable and reproducible approaches to their analysis. Manual inspection is not without merit, and expert supervision is essential to – or may even outperform – automation (Bennett et al., 2014; Casini et al., 2023; Quintus et al., 2017). Human observers, however, cannot manage the growing volume of remotely sensed data, or harness all the data contained in modern, global, high-resolution, multi- or hyperspectral imagery. Expertise in Geographic Information Systems (GIS), computational skills, and data management, as well as domain expertise, are all needed for manual digitisation, a shortage of which hampers crowdsourcing and citizen science projects (Duckers, 2013; Jessop, 2007). Projects that have applied manual inspection at scale (“GlobalXplorer”, 2016; Lin et al., 2014), struggled with: (1) the recruitment and retention of reliable participants (Wald et al., 2016; West and Pateman, 2016), (2) the biases introduced by observers, and (3) the variability of naked-eye observations – problems that have only increased with the scale of the research (Cowley, 2012; Quintus et al., 2017; Sadr, 2016).

In the last decade, a number of semi- and fully automated classification approaches have emerged to complement manual inspection of imagery. They range from template-matching (Kvamme, 2013; Trier et al., 2009, 2015) to knowledge-based (D’Orazio et al., 2012, 2015; Riley, 2009) and geographic object-based image analysis (Cerrillo-Cuenca, 2017; Davis et al., 2019), to ML (Caspari et al., 2014; Guyot et al., 2018; Menze and Ur, 2012) and deep learning, a branch of ML which involves artificial neural networks such as CNNs (Berganzo-Besga et al., 2023; Trier et al., 2019; Verschoof-van der Vaart and Lambers, 2019). Ten years after Parcak’s claim that “Tools cannot match the human eye” (2009, p. 110), artificial intelligence (AI) may be catching up. With a relatively small amount of high-quality, annotated data (circa 250 features), it is argued that archaeologists assisted by technologists may now extend their vision with digital eyes (Casana, 2020).

As a result, ML and other artificial intelligence approaches to remote sensing in archaeology are becoming ever more popular. Figure 2 displays the annual publication count in each of the past 10 years (n = 70; research articles and conference papers plus one preprint) in Web of Science for “topic” including “archaeology”, and “remote sensing or satellite image”, and “machine learning or artificial intelligence” (and variations thereof), manually reviewed to eliminate any publications that did not in fact discuss archaeology or ML/AI. This search reveals that the annual count of relevant publications has increased from zero in 2014 and 2015 to 21 in 2023. These 21 publications represent about 17% of the 2023 total (n = 125) for archaeological remote sensing (generated using the same query minus “machine learning or artificial intelligence”). Publication counts from Scopus using the same query are similar (cf. Argyrou and Agapiou, 2022, who analyse publications related to AI in archaeology more broadly and find a steadier rise in adoption beginning earlier).

If publication counts are used a proxy for research, this 17% figure indicates that AI/ML is on the cusp of “crossing the chasm” separating “innovators” and “early adopters” (together 16% of the population) from the “early majority”, according to Rogers’ diffusion of innovations paradigm as modified by Moore (Moore, 1991; Rogers, 2003). Indeed, a recent review article has argued that “[i]t is now outdated to focus on whether archeology should embrace automated detection … the aim should be to answer how to embrace AI tools to overcome the challenges of data overload and rapid destruction of the archaeological site landscapes” (Argyrou and Agapiou, 2022, p. 18). The appearance of ML code libraries, pre-trained models, and other commoditised ML tools, combined with the use of large-language models as research and programming assistants (e.g. ChatGPT or GitHub Copilot), makes it likely that the technology will continue to proliferate.

As these approaches spread to a broader cohort of researchers, potential adopters need to recognise their challenges and limitations. Critical assessment has, however, been somewhat neglected in the literature. Considering the 70 papers from the Web of Science mentioned above, 44 abstracts (63%) fail to mention any negative aspects of AI/ML approaches at all. Of the 26 papers (37%) with abstracts that mention some challenge or limitation, 11 state that they were overcome by the researchers, representing unqualified successes. Only 15 papers include specific or sustained critiques of ML approaches. Of those 15, seven (10% of the corpus) present qualified successes, while four (6%) discuss attempts to deploy ML that ended in partial or complete failures (Maxwell et al., 2020; Roccetti et al., 2020; Sech et al., 2023; Verschoof-van der Vaart et al., 2020). Two others (3%) present critical assessments of ML without the authors having undertaken it themselves (Casini et al., 2021; Casana, 2020). Two (of four) review articles included critiques, one of which recognised specific challenges but generally reflected the positive tone of the literature (Argyrou and Agapiou, 2022), while the other offered a more sustained discussion of challenges and possible solutions (Kadhim and Abed, 2023). The overwhelmingly positive tone of these papers likely indicates a certain degree of “publication bias”, where positive results are more likely to be published than negative (Brown et al., 2017; Dickersin et al., 1987; Harrison et al., 2017; Ioannidis, 2005; Kühberger et al., 2014; Møller and Jennions, 2001), or at the very least a reflection of the rhetorical shift in scientific research towards less qualified or uncertain presentation of outcomes (Vinkers et al., 2015; Wheeler et al., 2021; Yao et al., 2023; Yuan and Yao, 2022). In this context, it is important to document unsuccessful attempts to apply ML techniques to archaeological remote sensing, or at least to highlight problems researchers are likely to face as they adopt the technology.

Data

Pedestrian survey

In this study we used a dataset of 773 mounds, collected by TRAP during 2009 – 2011 field survey in the Kazanlak Valley, Bulgaria (Sobotkova and Ross, 2018). This fieldwork covered some 85 sq km, inspected directly via pedestrian survey. TRAP survey identified many 0.5–20 m high (5–100 m diameter) conical features in the landscape as burial mounds (Figure 3). Each mound was recorded with a GPS point, height, diameter, and surface and surrounding land use, as well as preservation status using Likert scale and Wildesen (1982) classification (Sobotkova and Ross, 2018; Valchev and Sobotkova, 2019). While the identification of stony, submeter features (see Plate 2) as mounds was initially questioned, subsequent excavations demonstrated that these features indeed contained stone-lined graves with human remains, partly buried by colluvium or diminished by post-depositional processes (Nekhrizov et al., 2013).

Satellite imagery

The satellite imagery used in this study consists of two IKONOS scenes covering 600 sq km delivered in geoTIFF format, which were acquired through a GeoEye Foundation grant in 2009. The scenes included a panchromatic band at 1 m resolution and a multispectral image (RGBNIR) at 4 m resolution. We fused the panchromatic and multispectral bands during 2009 fieldwork, to combine the high resolution of the panchromatic with the additional information of the multispectral bands in order to facilitate visual inspection of the images. Once fused, the two scenes were mosaiced and their edges cropped to form a rectangle. Although we had 600 sq km of imagery, all data for CNN model training and validation came from the 85 sq km TRAP study area within it.

Methods

Transfer learning

Rather than training our own model from scratch, we used a pre-trained CNN, a technique known as transfer learning (Casini et al., 2022; Character et al., 2021; Gallwey et al., 2019; see also Kadhim and Abed, 2023; Pan and Yang, 2010; Sech et al., 2023; for an overview see Weiss et al., 2016; Xiong et al., 2020). Transfer learning assumes that large, complex models can be pre-trained using data from one domain, then fine-tuned for a specific task in another domain. Transfer learning has become widely accepted as a standard approach to ML problems in areas like computer vision and natural language processing (Ruder et al., 2019). In the case of transfer learning for image data, CNNs are often pre-trained on a subset of ImageNet (Deng et al., 2009), learning combinations of weights and parameters which best extract information in order to make predictions between 1,000 different classes of image (Krizhevsky et al., 2012). It is important to note that what is being transferred is not the ability to directly classify objects or make predictions about the contents of images. Instead, what pre-trained models appear to learn is the ability to perform a kind of feature extraction. The weights and parameters of the pre-trained CNN are used to extract dense, numerical vector representations of images – objects known as image embeddings. These image embeddings then act as the input to a classifier algorithm which learns to map between these abstract feature vectors and the provided labels (Akata et al., 2015).

We adopted a transfer learning approach for this project for several reasons. First, pre-trained CNNs such as VGG16 (Simonyan and Zisserman, 2015), ResNet-50 (He et al., 2016), and Inception-V3 (Szegedy et al., 2016) have been shown to perform better (than a manually-trained model) on a wide range of downstream tasks such as image classification. Second, pre-trained CNNs have been trained on much larger and more diverse datasets than those available to the average researcher, meaning that they should learn more sophisticated ways to generate image representations. Finally, the use of pre-trained models allows for faster prototyping and testing, an advantage when assessing its utility for particular field research.

After some preliminary experimentation with a range of different pre-trained models, we concluded that ResNet-50 seemed to perform best for our data. This model is one of the smaller pre-trained CNNs available, with only around 25.6m trainable parameters (for comparison, VGG16 has some 138.4m). The ResNet architecture has, however, been shown to learn high quality image embeddings more effectively due to the use of residual layers. The model was deployed using TensorFlow 2 in Python (see the GitHub repositories by Kristensen-McLachlan and Mallon, 2021, 2022).

During initial experiments, the model was found to overfit the training data, reporting close to 100% accuracy after only a few training epochs. Overfitting in such an extreme manner results in a model that does not generalise well to new data (Kadhim and Abed, 2023), meaning that such a classifier would not be useful. In order to counter this overfitting, we used proven data augmentation techniques to constrain model performance (Wang et al., 2017). Data augmentation involves making random modifications and changes to the data during training, introducing noise and variability. The training data thus becomes more difficult to learn from, resulting in models that overfit less and producing a classifier that should generalise better to unseen data.

Additional CNN training

Training data preparation

Each ML training dataset requires two sets of images: positive data containing target features (MOUND) and negative data excluding the target features (NOT MOUND). Mound points taken during fieldwork were used as centroids for the generation of 150 × 150 m square polygons (150 × 150 pixels at 1 m resolution), which were clipped from the IKONOS imagery. This process yielded 773 MOUND cutouts, each centred on a mound (Figure 3). Cutouts were made from the four-band fused images. The NIR band was merged into the Red band for the classification. The mounds were always in the centre of the cutouts. No accommodation was made for the size of the mound; 100 m diameter mounds filled 34.9% of the cutout, while 10 m diameter mounds covered only 1.4%. Likewise, no accommodation was made for surrounding land cover, mound cover, or terrain. NOT MOUND data cutouts were generated in the same manner from areas excluding the 773 ground-truthed mound points, with no manual review. Both the satellite imagery and points were projected into the same EPSG: 32,635 coordinate reference system to be mutually compatible and to provide us with metric units of measurement. The ratio of positive to negative training data was approximately 1:2 (32%–68%).

In the 2021 run of the model, we used all 773 cutouts for training regardless of what was visible in the satellite image. In the 2022 run, we selected 249 cutouts where a mound was discernible with the naked eye. All of the training images were augmented using vertical and horizontal flip and random rotation. After processing, cutouts were divided into training, validation, and test sets following a 70:20:10 ratio for automated performance validation.

Training execution

We first guided the pre-trained CNN to learn to identify burial mounds from the positive (MOUND) and negative (NOT MOUND) training data. Once this additional training was completed (including validation and testing), the model was supplied with the entirety of the two IKONOS images (inclusive of training data), preprocessed into 150 × 150 pixel tiles. The model evaluated each tile and returned a MOUND probability prediction on a 0–1 scale, with 1 being 100%.

Assessment

Performance evaluation

The performance of the model was assessed using common evaluation criteria: We calculated F1 scores, precision, recall, and accuracy. These evaluation metrics are the model’s own answers to the question “how well does the model predict MOUND/NOT MOUND in the test data, based on what it has learned from the training data?”

Model validation against field data

After the initial performance evaluation, we manually validated model performance. To do so, we verified mound predictions using points marking ground-truthed mounds in the 85 sq km section of the satellite image that had been systematically surveyed during TRAP fieldwalking.

The CNN’s predictions were first associated with the tile name and a pair of origin coordinates marking the bottom left corner of the tile. We selected prediction records whose mound-probability exceeded 0.599 and used the coordinates to generate square polygons of 150 m side. We intersected these polygons with the points marking ground-truthed mounds. Mounds that fell inside the tiles were considered successfully detected (true positives). Mounds outside the flagged tiles were missed by the model (false negatives). Tiles flagged by the model that did not contain mounds constituted false positives.

Results

First run (2021): full training dataset

In the first run of the model, it was trained using all 773 known mounds, whether or not a mound was visible in the satellite image. After image augmentation, the model reported good learning and model fit (F1 = 0.87). This F1 score indicated that the use of a pre-trained model improved performance by 0.05 compared to a previous, manually trained model (pers.comm Cormac Purcell; Kristensen-McLachlan and Mallon, 2021).

Nevertheless, only 19 out of 148 tiles (12.8%) tagged by the model with at least a 60% chance of having a mound actually contained one (see Figure 4). Some 129 of the tagged tiles (87.1%) were false positives. The 19 true-positive tiles contained 38 mounds (1–9 mounds per tile), out of 773 in the study area (4.9%), while the remaining 735 mounds went undetected. Undetected mounds were located in 381 tiles (1–20 mounds per tile) out of 400 tiles that actually contained mounds, a false negative rate of 95.3% (Sobotkova, 2022).

During a visual examination of the model predictions, we saw that the model avoided the Koprinka reservoir in the middle of the valley. It correctly detected some mounds around the reservoir as well as a few in the northeastern necropolis in the Valley. In the north and northwest, however, it missed many, including both small mounds and large royal mounds that are clearly visible during manual inspection (see Figure 1e, f). It is also puzzling that in the densely mounded landscape of the northeastern necropolis, the algorithm managed to select an area devoid of mounds. To the south of the reservoir, the algorithm incorrectly selected a lot of forest, beaches, and roads, all of which increased the false positive rate. Overall, the model seemed to select bright lines and edges (forest, roads), rather than round shapes more likely to represent mounds. The greatest surprise was that the model failed to detect the largest mounds in the valley. These round, symmetrical features stand out against surrounding agricultural fields, and are crystal clear to any human viewer (Figure 5).

Second run (2022): training data filtered for visible mounds only

The second run was trained on 249 mounds that were clearly visible during manual inspection of the satellite imagery. The rest of the method remained the same.

The second model’s performance declined to an F1 score of 0.62 (Kristensen-McLachlan and Mallon, 2022). Validation revealed that only 21 of 773 mounds (2.7%) were detected, while 752 mounds (97.3%) remained undetected. The number of tiles within the TRAP study area flagged as containing a mound (at a >60% probability) increased from 148 in the first run to 288 here. Only 15 of these 288 tiles (5.2%), however, were true positives, containing the 21 detected mounds (1–4 mounds per tile; see Figure 6). The remaining 273 of 288 tiles were false positives (94.8%). The undetected 752 mounds lay in 384 tiles (1–28 mounds per tile) out of 399 tiles that actually contained mounds, a false negative rate of 96.2% (Sobotkova, 2022).

Small mounds were detected at a lower rate than in the 2021 model, demonstrated especially in the lack of predictions in the northwestern necropolis, which contained many mounds <1 m high. In addition, the large, archetypal mounds around the village of Shipka (north-centre of the study area, Figure 5) also remained undetected (as in the first run of the model). False positive tiles show that edges of forest, beach, and sections of roads are again classified as mounds. Furthermore, the model flagged parts of the reservoir as a mound with >60% probability, despite the homogeneous water surface, bringing the question “what is the CNN actually detecting?” to the fore. Overall, despite additional curation of the training data, the second run was even less successful than the first; the false positives and false negatives increased and fewer mounds were detected.

Discussion

Limitations and challenges of pre-trained CNNs

A number of factors contribute to the poor success rate of our CNN model when attempting to detect burial mounds in satellite imagery. The challenge centres around the variability of the appearance of the mounds themselves, combined with the heterogeneous landscape surrounding them and the noise of non-mound features.

In terms of CNN mechanics, when detecting mounds the model split the 600 sq km geoTIFF into non-overlapping 150 × 150 pixel tiles rather than analyse the mosaiced image via a moving window of variable size. As a result, mounds could sit on a tile boundary and elude detection. Many of the large undetected mounds, for example, fell on the edge of a tile. Second, an intersection of our CNN process and the nature of the training data produces low sensitivity to different mound sizes. The training data with the smallest mounds leaves a lot of non-target pixels in the tile (ca. 98%), while introducing other repeating and prominent features that confuse the classifier, such as the sharp edges of different land-use zones (e.g. stands of trees versus annual agriculture or pasture) and reflective linear features like roads and shorelines. Models have been observed to predict class membership based on incidental background features in the periphery of the tiles that happen to accompany the target phenomenon in the centre. Peter Haas pointed to this problem in “The Real Reason to be Afraid of Artificial Intelligence” when he explored how a deep learning model differentiated wolves from dogs in photographs. Wolves were recognised by the snow in the background rather than any intrinsic characteristic (TEDx, 2017). This effect arose from bias in the training data, where wolves were often on snowy background, while dogs appeared in a variety of other backgrounds. In our case, when the first model using the entire training dataset underperformed our expectations, we sought to address the problem by retraining the model using more prominent mounds that hopefully stood out better from background noise. This correction, however, failed to improve performance – indeed, it deteriorated further.

Overall, training the model with a set of highly variable features with even more varied and complex backgrounds may have misled it regarding the target of detection. This confusion likely arose from the variety of land cover on the surface of mounds and surrounding the mounds, which may or may not provide a contrast (or may provide different kinds of contrast).

Building a better model

Focus the positive training data

To counter the tendency of the model to detect extraneous background features, we could better focus the training data. Confronted with small and varied features in noisy environments, Verschoof-van der Vaart et al. (2020, pp. 7–8), for example, invested some effort adjusting model parameters to account for target feature size. In our case, to steer the CNN away from “reading the background”, we would likely need to resize each tile according to the recorded diameter of the mound. We could, for example, specify a particular ratio (determined through experimentation) between mound and tile size, thereby controlling the target to background pixel ratio. That way we would end up with stamps of variable size containing minimal background noise. While this could be time-consuming for features of variable shapes, our dataset contains consistently circular mounds whose diameter was measured on-site. The only complication is that the measured diameter often varies from the diameter visible in the image due to indistinct mound boundaries.

Segment the training data

To address the problem of land-cover variability, we could create different training dataset for each class of surrounding land cover, each class of over-mound cover, or perhaps for various combinations of surrounding and over-mound cover. Segmentation of training data into runs of bare mounds in ploughed fields, scrub-covered mounds in ploughed fields, bare mounds in meadows, scrub-covered mounds in meadows, etc., would offer more consistency and might improve success rates. With all these possibilities, however, we reduce the amount of training data for each run. We also risk returning to the templating approaches of Kvamme (2013) and Trier et al. (2009), which would defeat the purpose of deep neural networks, which are supposed to learn their way towards recognising the target features more autonomously.

Emphasise negative training data

It may be counterintuitive, but care in creating the NOT MOUND tiles is as – if not more – important than the MOUND tiles. One lesson we learnt is that CNN cannot “ignore” without extensive training on negative examples (Gao et al., 2019; Tang et al., 2017). We can tell a person to ignore roads and forests, and she or he will instantly recognise and ignore them. For a CNN to action such a request we need to expose the model to them using negative training data. Especially when target features are small or occur in noisy backgrounds, care must be taken to provide a CNN with more negative examples (tiles without archaeological features) than positive. Our experience suggests that even a 1:2 (positive:negative) ratio was not sufficient. For comparison, Verschoof-van der Vaart et al. (2020) suggest a ratio of 1:1.6 in training data.

Employ alternative discovery strategies

During discovery, iterating the CNN over the image using overlapping tiles of different sizes would give each mound a higher likelihood of falling completely inside a tile. Using a moving window would also avoid missing mounds at tile edges (Tang et al., 2017). Employing a Regional CNN, which can detect multiple, even overlapping features in an image could also improve results, especially if combined with iterating over variations of tiles (Maxwell et al., 2020; Verschoof-van der Vaart et al., 2020). Overlapping tiles or moving windows, however, would impose a substantial computational overhead as they would exponentially increase the imagery needing analysis. While upscaling the analysis is possible in a virtual environment, it would not be runnable on a typical personal computer, making such analysis infeasible for many cultural heritage specialists with modest resources.

Is it worth it?

Our outcomes compare poorly to the other applications of ML to archaeological prospection cited above (Caspari and Crespo, 2019; Ekim et al., 2021). Based on our experience and the available literature, solutions exist that can address the problems we encountered. But is the extra effort worthwhile? Although time-on-task data regarding ML approaches in archaeology is limited (Sobotkova et al., 2023), such activities can absorb a lot of resources. Can et al. (2021), for example, report spending 1,250 h manually creating a training dataset of road features and a further seven days on testing and training a CNN model to detect roads in Austro-Hungarian imperial maps. While their CNN outperformed other models, the magnitude of manual data preparation, expertise required, and the computational infrastructure needed for training is within reach of only a few well-funded projects.

Developing our CNN model required approximately 135 person-hours from conceptualisation and experiments to validation and documentation. Our team included two digital archaeologists, a machine-learning specialist with experience applying ML approaches to cultural heritage data, and a junior developer who wrote much of the code used to implement these models. The approach used minimally processed datasets from previous fieldwork. Our approach to training was driven by expectations that the pre-trained CNN could tolerate a great deal of variation in training images, such as of mound size within a tile or the presence of confounding features, and that training data would not require extensive curation. We also brought with us from manual inspection of satellite imagery the preconception that burial mounds were relatively easy to recognise in imagery, at least compared to latent, “flat” sites. Our outcomes, however, indicate that even an advanced pre-trained model (ours is the same as that used by Can and Kabadayi, and perhaps also Crespo and Caspari) is likely to fail when asked to detect somewhat variable features against very heterogeneous backgrounds with only a low-touch approach to training. Improving the performance of our model is technically possible, mostly through more nuanced training approaches (see above). For our project, however, the return on additional time spent developing the ML approach diminishes rapidly, since we are already approaching a threshold where training student volunteers to identify mounds would be more efficient.

Nor is success guaranteed. To take one well-explained example, Verschoof-van der Vaart et al. (2020) implemented a sophisticated approach to LiDAR imagery using a region-based CNN. This approach is designed to classify multiple, adjacent, or overlapping features in a single image – potentially addressing some of the problems presented here. They also took care to provide adequate negative training data. Nevertheless, to reduce false positives, the project had to introduce a manual location-based ranking step, where experts identified, ranked, and mapped landscape characteristics (largely based on post-depositional processes) that might affect the survival or visibility of archaeological remains (Verschoof-van der Vaart et al., 2020, pp. 9-10). Despite these manual interventions, and noting that performance evaluation involved realistic but artificial test data, the model still produced many false positives from natural or anthropogenic geometric shapes (not unlike our experience). In the end, Verschoof-van der Vaart et al. report that their models never reached the performance of crowdsourcing using volunteers (again, mostly due to its false positives), and the authors observe that the cost of ground truthing would be high.

Maxwell et al. (2020) achieve better results applying an R-CNN to LiDAR data for more prominent features (fill-valley faces produced by coal mining in the Appalachian Mountains), but the approach did not transfer to photogrammetric datasets. Even many successful ML applications require significant human intervention either in the form of crowdsourcing or specialist work (see, e.g. Casini et al., 2023; Doyle et al., 2023; Verschoof-van der Vaart and Lambers, 2019). As a result of challenges like these, Casana (2020), goes so far as to reject ML approaches entirely, instead favouring “brute force” manual inspection of satellite imagery by experts. Two of the authors on this paper themselves have had excellent large-scale (ca. 10,000 features) results from crowdsourcing (Sobotkova et al., 2023). We reiterate the call in that paper for researchers to publish the challenges, costs, and resourcing requirements alongside results – including for serious but failed attempts – so that researchers, especially potential adopters of ML, can make informed decisions about what approach best fits their circumstances.

Conclusion

We set out to test the efficacy of a Machine Learning approach to detecting burial mounds situated in varied terrain and vegetation in the Kazanlak Valley of Bulgaria. We had at our disposal high-resolution, multispectral satellite imagery and a dataset of some 773 mounds registered during pedestrian survey. We employed a pre-trained CNN model with a low-touch approach to additional training, running the model twice using different collections of target features for training. We first used an indiscriminate collection of burial mounds, regardless of their visibility in the satellite imagery. We next used only those mounds that we deemed easily discernible in the satellite imagery. Unlike many projects, we had enough training data, but we did not devote a lot of time to prepare the training cutouts, e.g. by resizing them according to mound size or excluding background features. We believed that the volume of training data would offset other shortcomings. After this training, we applied the models to our imagery and validated results using our field data.

Although the first model reported a good fit (F1 = 0.87), validation with field data showed the model was confusing our target features with other phenomena. Tiles where the model predicted a mound with >60% probability contained a mound only 12.8% of the time – yielding a high rate of false positives (87.1%). At the same time, the model found only 38 of 773 known mounds (4.9%). Despite increased effort in the selection of training data, the second run of the model reported a lower F1 score (0.62) and at >60% probability produced even more false positives (94.8%) and false negatives (96.2%). This run found only 21 mounds (2.7%).

The high number of both false negatives and false positives demonstrated that our use of a low-touch approach – a pre-trained model plus sufficient but minimally curated training data – did not work. To overcome these challenges we would need to invest much more time, effort, and computing resources into additional steps such as: resize each training cutout based on the size of the mound to eliminate clipping and reduce noise from extraneous background features; segment the training data into different mound covers versus surrounding land covers; increase the ratio of negative training data; and utilise overlapping tiles, a moving window, and/or a Regional CNN.

A computationally manageable, low-touch model capable of discovering archaeological features in widely available satellite imagery with modest training data would be a valuable tool for cultural heritage managers who need to register such features and monitor them for change. Such managers often lack access to cloud computing infrastructure, or to technologists with the necessary geospatial analysis, data-wrangling, or machine-learning skills. It is not clear, however, that an ML approach remains cost effective if these additional steps are taken. Our work consumed about 135 person-hours and required significant domain and technical expertise, although it required only modest computing resources. The proposed modifications would certainly increase the time required, and potentially the expertise and computational resources. Meanwhile, our first intervention to improve the results added 20 h to the total time, but led to a poorer outcome. In our case, given the size of our study area and the number of features it likely contains, manual approaches like visual inspection by experts or crowdsourcing to volunteers would probably be more efficient.

Each project must make a decision regarding the trade-offs of different approaches to archaeological prospection using remotely sensed data, but the relevant literature underreports failures, challenges, and limitations of ML when used for this application. Very few projects, furthermore, report the time and resources invested in their approach, whether manual or automated. As ML approaches become more popular, researchers need more negative examples, discussion of problems, and resourcing information to make informed decisions about how to approach feature extraction from large remote-sensing datasets.

Figures

Plate 1

Medium-sized burial mound in the Kaznlak Valley (TRAP inventory #2082)

Plate 2

Submeter mound in the Kazanlak Valley (TRAP inventory #3341)

Figure 1

150 by 150 m cutouts from the satellite imagery with mounds in the center used for ML training; (a–c) show medium-sized well-visible mounds, (d–f) large mounds covered in scrub and surrounded by varied vegetation, (g–i) show small mounds, poorly visible in satellite imagery

Figure 2

Annual publication count for AI or ML applications related to archaeological remote sensing

Figure 3

Map of 773 mounds documented during pedestrian survey in the Kazanlak Valley; circle radius represents height from 0.5 to 20 m

Figure 4

Results of 2021 CNN run, blue squares represent tiles with 60%+ probability of containing mounds, circles mark true positives (actual mounds detected)

Figure 5

Poor detection of large mounds at the royal necropolis of Shipka in the Kazanlak Valley; blue dots mark the actual mounds, blue square represents tile with 60%+ probability of containing a mound

Figure 6

Results of 2022 CNN run; blue squares show tiles with 60%+ probability of containing mounds, circles mark true positives (actual mounds detected)

Code: Data processing and analysis was performed using R and Python and the scripts are available in public repositories: (1) Training data preparation and CNN prediction validation can be found in https://github.com/adivea/cnn-testing (2) 2021 CNN classifier training and mound prediction is implemented in https://github.com/centre-for-humanities-computing/burial-mounds (3) 2022 CNN classifier training and mound prediction is implemented in https://github.com/centre-for-humanities-computing/MoundDetection