Improving the evidence base for delivery of public goods from public money in agri-environment schemes

Abstract

There is growing interest around the world in more effectively linking public payments to the provision of public goods from agriculture. However, published evidence syntheses suggest mixed, weak or uncertain evidence for many agri-environment scheme options. To inform any future “public money for public goods” based policy, further synthesis work is needed to assess the evidence-base for the full range of interventions currently funded under agri-environment schemes. Further empirical research and trials should then focus on interventions for which there is mixed or limited evidence. Furthermore, to ensure the data collected is comparable and can be synthesised effectively, it is necessary to reach agreement on essential variables and methods that can be prioritised by those conducting research and monitoring. Future policy could then prioritise public money for the public goods that can most reliably be delivered, offering better value for taxpayers and improving the provision of ecosystem services from agricultural landscapes.

Citation

Reed, M.S., Chapman, P.J., Ziv, G., Stewart, G., Kendall, H., Taylor, A. and Kopansky, D. (2023), "Improving the evidence base for delivery of public goods from public money in agri-environment schemes", Emerald Open Research, Vol. 1 No. 10. https://doi.org/10.1108/EOR-10-2023-0006

Introduction

Productivist agricultural policies have been blamed around the world for declines in farmland, wildlife and water pollution, as well as an overall decline in the provision of ecosystem services from agricultural landscapes (e.g. Donald et al., 2006; Reif & Vermouzek, 2019; UNCCD, 2017). Arguing the real culprit lies in the way agriculture is subsidised, there is growing interest in more effectively linking public payments to the provision of public goods from agricultural land. Successive reforms of the European Union’s (EU) Common Agricultural Policy (CAP) have pushed for “greening” measures that integrate environmental measures into Pillar I (Hart & Radley, 2016; Matthews, 2013), although CAP budget allocations suggest that the policy retains a primarily productivist approach that may have limited environmental benefits (Wilson & Hart, 2001). This has led some to describe the greening of the CAP as “greenwashing” (Alons, 2017; Erjavec & Erjavec, 2015). Now, as the UK leaves the EU, each UK country is pursuing an approach that prioritises environmental measures more than was possible under CAP, with England putting the principle of “public money for public goods” at the heart of the Agriculture Act 2020 (Bateman & Balmford, 2018; Coe & Finlay, 2019). The definition of public goods in the Act is broad, encompassing the management land, water and livestock to protect the environment, support public access, protect cultural and natural heritage, mitigate or adapt to climate change, protect against environmental hazards, protect or enhance animal welfare, and protect or enhance plant or soil health Explanatory Notes to the Bill p7–10 (Bill 7-EN).

However, there are concerns that many of the environmental measures promoted in existing agri-environmental schemes may not deliver the outcomes that are expected of them. This may be because schemes are typically based on payments for land management inputs or actions, which are assumed to deliver desired outputs of ecosystem services, based on research linking inputs to outputs (Burton & Schwarz, 2013; Hanley et al., 2012). Uptake may be limited because the standardised payment rates do not reflect spatial or temporal variations in biophysical conditions, management costs or the value of ecosystem services (Armsworth et al., 2012; ECA, 2011). The environmental, including biodiversity impacts, often depend on wider spatial scales across multiple land management units, whereas payments are linked to individual farm units (Engel et al., 2008; Goldman et al., 2007; Groth, 2005; Klimek et al., 2008; Moxey et al., 1999; Wunscher et al., 2008). Finally, the choices farmers are given in agri-environmental schemes are based on limited evidence, which is biased towards protecting biodiversity and water quality. In contrast, present-day goals, for example in the UK 25 Year Environmental Plan (Defra, 2018), are much broader and among others include climate mitigation and cultural heritage, which are also important ecosystems services.

In this short thought piece, we discuss:

• Evidence from literature reviews for the effectiveness of land management interventions currently funded under agri-environmental schemes across Europe;

• How agri-environment schemes might be redesigned in light of mixed, weak and missing evidence; and

• How a new approach to the collection and reporting of evidence might enable future schemes to be based on robust evidence synthesis, and in turn increase the likelihood that they deliver public goods.

Evidence for public goods from agri-environment schemes

Current shortcomings

Systematic reviews suggest there may be insufficient evidence to determine the effects of many agri-environmental scheme options. In part, this is because much of the data collected is not comparable (see next section). In addition, certain outcomes, indicators, interventions, farming systems and countries are less studied than others, with this leaving important evidence gaps. Some of the best evidence is available for the effects of agri-environment schemes on farmland biodiversity. For example, a systematic map of evidence for agri-environment scheme options in temperate Europe identified 743 studies showing the effects of interventions to increase farmland biodiversity. It included organic, conventional and integrated systems and a wide range of interventions, from reduction in tillage and protection of field margins, to different types of grazing systems and hedgerows (Randall & James, 2012). The review found there to be a lack of evidence across all interventions about effects on amphibians and reptiles. A more recent rapid evidence synthesis by Tresise et al. (2021) also found limited evidence on mammalian and avian responses to hedgerow planting and management and found limited evidence that biodiversity increased with hedgerow age (although Biffi et al. (2022) have since published evidence of increased soil carbon storage with hedgerow age). Furthermore, there was a bias towards evidence from Western European countries, in particular the UK. There was also limited published evidence about the effects of certain scheme options, such as under sown cereals and cereal based whole crop silage on biodiversity, and there were too few published studies to draw conclusive findings about the effects of scheme options in different farming systems (e.g. organic versus conventional). Similarly, Benedetti (2017) conducted a systematic review that identified 90 studies from Western European countries and noted that just 1% considered effects of interventions on amphibians, arachnids, bryophytes, lichens and reptiles.

Other issues relate to the quality of the studies conducted. For example, Kleijn & Sutherland (2003) analysed 62 evaluations of agri-environment schemes in Western Europe and concluded that in the majority of these studies, the research design was inadequate to reliably assess scheme outcomes. Almost a third of the studies they reviewed did not contain statistical analyses, and experimental designs were typically “weak and biased towards giving a favourable result”. A prime example of this is that they did not use before-after-control-impact (BACI) designs. Even within well designed studies (including BACI designs and other approaches to ensure unbiased sampling that accounts for potential confounding factors), there are a number of sources of uncertainty. These include soil depth sampled, choice of species/combination of species sampled and the different ways in which options are implemented in very different biophysical contexts. For example, there is strong evidence that agroforestry increases soil organic carbon storage, but most studies have been carried out in the tropics, leading to uncertainty about the effectiveness of agroforestry for delivering similar public goods in many European contexts (Chapman et al., 2018).

Another challenge is presented by the significant time lags that exist between the implementation of interventions and the delivery of many public goods. Despite decades of research and interventions aimed at reducing nitrates in ground and surface waters, they still remain high across Europe (Grizzetti et al., 2011; Wiering et al., 2020). Similarly, greenhouse gas emissions from peatlands often experience a temporary methane spike after restoration, despite providing long-term climate mitigation benefits (e.g. Green et al., 2018). As a result, short-term studies may provide misleading findings.

Intervention effectiveness

Despite the challenges presented by these evidence gaps, research design and time lags, there is evidence for the effectiveness of some interventions on biodiversity and other ecosystem services. This literature remains dominated by biodiversity studies, despite the range of public goods that schemes typically seek to deliver, as well as the strategic importance of climate change mitigation as a policy objective. Sutherland et al. (2020) in their edited book and online database, “What Works in Conservation” reviewed evidence for conservation benefits of agricultural interventions, including 119 interventions in their farmland conservation synopsis (Dicks et al., 2017). Of these, only twelve were clearly beneficial and a further 18 considered likely to be beneficial, with the rest unknown effectiveness (limited evidence), no evidence found or likely to be ineffective or harmful. Kleijn & Sutherland (2003) showed that only 4 out of 19 studies reported positive effects of agri-environmental schemes on bird species richness or abundance. They found that two studies gave negative results and 11 studies showed results in both directions. In addition, 11 out of 20 arthropod studies showed an increase in species richness or abundance, 3 showed mixed results and none showed a decrease. With plant studies, they found 6 out of 14 showed increases in species richness or abundance and two showed decreases. Roberts & Pullin (2007) analysed 3,070 studies of the effects of four UK agri-environment schemes (with a specific focus on stubble and wild-bird cover options) on farmland bird densities. Based on summer and winter data respectively, only six out of 17 species and eight out of 15 species studied had higher densities in fields where scheme options had been implemented compared to conventional cropping, concluding that there was no statistically significant difference between densities of other species studied between fields with and without scheme options.

Looking beyond farmland birds, Kleijn et al. (2006) showed that agri-environment schemes in five European countries performed poorly for uncommon and rare taxa, but provided poor to moderate benefits for more abundant and widespread taxa, including vascular plants, birds, bees, grasshoppers, crickets and spiders. Batáry et al. (2015) merged data from three meta‐analyses and found that schemes implemented after the EU's agri‐environmental programs were revised in 2007 were no more effective in enhancing species richness than those implemented before revision. However, their analysis suggested that scheme options targeting non-productive areas (e.g. field margins and hedgerows) were more effective than options targeting productive areas (e.g. arable crops or grasslands).

Considering outcomes other than biodiversity, Elliot et al. (2022) reviewed evidence for the soil carbon sequestration benefits of a range of interventions, including grazing management, no and low till, cover crops, hedgerow planting, residue incorporation, leys in crop rotation, agroforestry, arable land use change, field margins, paludiculture, enhanced rock weathering, biochar and Bioenergy with carbon capture and storage. They found robust evidence for only hedgerow planting, residue incorporation, arable land use change, paludiculture and biochar, but noted potentially significant socio-economic barriers to the adoption of paludiculture and issues with leakage for arable land use change. Haddaway et al. (2018) conducted a systematic evidence map of research on the effects of vegetated strips on ecosystem services. They identified a number of evidence gaps, including evidence for climate regulation benefits, cultural ecosystem services and long-term impacts, as well as the relationship between pest populations and crop yield, which is a barrier to adoption by many farmers. Carrick et al. (2019) found only seven papers (out of 156 papers reviewed) met the inclusion criteria for their review on the effects of tree planting on reducing flood risk. Although they found that tree planting reduced channel discharge slightly, the effect was variable and potentially confounded, and in the absence of more direct evidence from primary research, it was not possible to say whether or not there was a flood risk reduction.

There is a lack of multi-year empirical data to assess the efficacy of many scheme options designed to improve water quality. Furthermore, model simulations tend to assume interventions remain effective over time with no decrease in effectiveness with age (Liu et al., 2017). They concluded that public goods arising from interventions designed to improve water quality “have not been as rapid or large as expected” with some interventions actually becoming sources of pollution under certain conditions. For example, grass buffer strips may initially retain particulate phosphorus, but can then release it slowly over time (Dodd & Sharpley, 2016; Sharpley et al., 2015). Similarly, Stevens & Quinton (2009) showed that no single cropping system intervention was able to reduce all water pollutants, with some interventions reducing one target pollutant while increasing inputs of other pollutants. They emphasised the need to target interventions to individual pollutants and sites. This echoed the findings of Kay et al. (2009) who emphasised the site-dependency of many water quality interventions depending on soil type, hydrology and pollutant chemistry. They also highlighted the need to evaluate effectiveness on a site-specific basis.

More broadly, a review of evidence for the public goods arising from Ireland's Rural Environment Protection Scheme (Finn & O Huallachain, 2012) concluded that there was insufficient evidence to judge the effectiveness of the scheme at a national scale. This was due to the spatial scale of studies, lack of baseline data and emphasis on biodiversity compared to other ecosystem outcomes.

Policy implications

A more evidence-based approach to delivering public goods from public money

Early agri-environment schemes tended to focus on enhancing biodiversity with a more recent (and less well-studied) focus on scheme options to enhance water quality and other ecosystem services. As we see a move towards the delivery of a wider suite of public goods, it is more important than ever before to understand the scientific evidence needed to assess whether these investments are likely to deliver the intended benefits. However, the evidence syntheses discussed in the previous section suggest that the evidence for public goods arising from many agri-environment scheme options is mixed, weak or uncertain.

This lack of evidence should not be mistaken as evidence that the investments or the interventions they supported do not work; rather it demonstrates that further empirical research and evidence synthesis would be required to provide more robust evidence that the interventions provide public goods. Public funding for agriculture is predicated on the assumption that funded land management inputs or actions lead to outputs that are public goods. However, published evidence suggests that this assumption may be optimistic, and in the absence of stronger evidence, there is a possibility that some publicly funded interventions may have limited benefits and even unwanted effects.

For this reason, there is an urgent need to understand the nature and strength of evidence across the full range of publicly funded agri-environmental scheme options. In addition to peer-reviewed literature, there is a large amount of data available in grey literature produced by Government agencies, which could be included in future synthesis work. This would of course be subject to quality control inclusion criteria. There is then a need to commission targeted new empirical research and on-farm trials, based on evidence gaps identified in the literature reviews, to evaluate the public goods arising from interventions for which there is weak or mixed evidence. Given the inherent heterogeneity of agroecological systems, systematic collection of contextual data (e.g. soil type, altitude, slope and land use cover, management and history) is essential to enable the interpretation of conflicting results from different studies. For example, cover crops may increase soil carbon, but only on some soil types and for specific cover crops species or mix of species (Kay et al., 2009).

England’s post-Brexit Agriculture Bill includes a series of tests and trials to shape the design of the English Land Management Scheme (ELMS; Defra & RPA, 2021), which will replace the EU Rural Development Programme for England. A database is being designed to enable integration and analysis of data between trials, but unless similar outcomes are measured and reported in comparable ways opportunities for evidence synthesis may be limited. This will be explored further in the next section.

In England, as in the devolved administrations of the UK, there is a unique opportunity to re-evaluate existing options and as a result of this prioritise funding towards interventions that are more likely to deliver public goods. There are a number of ways this might be achieved:

• Given the focus of post-Brexit agricultural policy across the UK on public money for public goods, it is logical that priority should be given to interventions for which there is an evidential link to the provision of public goods. Given the time taken for empirical research, field trials and subsequent evidence synthesis, the inclusion of scheme options would need to be flexible so new options could be added when sufficiently robust new data enables conclusions to be drawn via evidence synthesis. If evidence synthesis is not possible due to insufficient evidence, or if synthesis continues to show limited or mixed evidence, then interventions would remain on hold pending sufficient further evidence to enable synthesis work to be done to evaluate the evidence-base as a whole.

• Options with a limited or mixed evidence-base could be included in schemes, but with additional funding for long-term data collection by land managers to build a more robust future evidence-base. Although this may be limited to indicators and proxies in some cases, there is evidence that citizen science approaches can provide robust insights for policy and practice. This is especially the case when complemented by more localised hypothesis-driven research (Dickinson et al., 2010) and Payment By Results schemes (e.g. Allen et al., 2014). Natural England’s recent trial (Chaplin et al., 2019) showed that farmers are able to collect relatively sophisticated monitoring data. It is important to note that interventions with mixed or limited evidence may emerge when the success of the intervention depends on external factors, and hence understanding when and where each intervention may work better is of particular importance. This can avoid neglecting interventions with mixed support, as those may be important for a given situation.

• Funding could be prioritised towards scheme options with the best evidence for delivering public goods via an ‘evidence-based premium’ for the options we know are most likely to deliver multiple public goods. This would enable farmers to continue to choose from a wider range of scheme options relevant to their farm characteristics and management objectives. However, given the weak or mixed evidence for these options, they would receive a lower payment rate in order to reduce uptake and reduce taxpayer exposure to risks of non-delivery of public goods. It is possible that this could be one consideration in the setting of different payment tiers as is currently proposed for ELMS.

• Alternatively, risk of non-delivery could be reduced across the scheme by giving farmers flexibility to adapt or discontinue interventions that are not delivering outcomes. For example, interventions designed to enhance water quality were sometimes reducing a target pollutant while increasing inputs of other pollutants, and if not maintained over time these interventions could lose efficacy or in some cases become a source of pollution themselves. The heterogeneity of conditions from field-to-field can also lead to highly variable results across a land holding. However, if farmers are collecting monitoring data, then this could be used to adapt intervention delivery, if sufficient flexibility were designed into the scheme. Alternatively, codes of good practices could be made part of future schemes to drive more effective implementation of interventions. The recent Defra Code of Good Agricultural Practice (COGAP) for reducing ammonia emissions provides simple, evidence-based ways to reduce NH ₃ emissions from agriculture.

In parallel with this, it is important that more funding is made available for evidence syntheses to be carried out by the research community, which would cover a targeted range of interventions/options that farmers are likely to take up, as well as assess the ability of these interventions/options to deliver multiple public goods. In the same way that our synthesis work identified evidence gaps, we would expect this to provide a more comprehensive assessment of key research needs, which could be targeted in future research funding calls. However, just generating more evidence will not be sufficient. How that evidence is generated and reported is also important if the data from future research is to be included in evidence synthesis and in turn provide robust evidence for policy and practice.

A new approach to generating evidence for agricultural policy

Even where high quality evidence was available, a lack of standardised approaches for designing studies, collecting and reporting data, and making these reproducible and open, often makes it difficult to synthesise data statistically. Study design and the choice of variables, data collection methods, analysis and reporting are frequently inconsistent across projects. For example, different projects may assess the effectiveness of an intervention in terms of its effects on soil health or biodiversity. Even if a number of projects focus on the same type of outcome, such as soil health outcomes, they may choose very different outcome measures or variables, for example both earthworm abundance and soil organic matter content are widely used as soil health indicators.

Even if the same variables are chosen, they may be measured in different ways. Soil organic carbon can be quantified via loss-on-ignition or the Walkley-Black acid digestion method. Even if the same variables are measured in the same way, they may be reported differently. Studies often fail to report sample sizes, variance measures, and details of experimental treatment or intervention, as well as other contextual information, such as location, soil type, management history, slope or altitude, which can be important for evaluating the comparability of findings from different studies. Even if all these challenges can be overcome, unclear reporting of data (e.g. only in graph form), selective reporting and publication bias can hamper attempts to draw unbiased conclusions for policy and practice from published research.

This makes it difficult or impossible to reconcile apparently conflicting evidence about important processes or practices and create robust syntheses for national and international decision-makers. There is growing recognition that to generate a robust understanding of the effects of land use interventions, there needs to be a more consistent focus on measuring and reporting essential or “core” outcomes that can be compared and synthesised between studies to provide more robust evidence for policy and practice (Gurevitch et al., 2018; Reed et al., in press). Through taking this approach, greenhouse gases would be prioritised over other gaseous emissions from agriculture for measurement. Furthermore, reporting on the climate mitigation benefits of any given intervention and standardised contextual variables would be reported alongside this data. For example, this would include location, farming system, soil type and altitude, as well as other variables to determine comparability of data and aid interpretation of results.

In response to this challenge, a number of programmes have now attempted to standardisation of data collection and reporting (Reed et al., in press). For example, the Global Biodiversity Information Facility (GBIF) is an international network and data infrastructure providing open access to biodiversity data (Flemons et al., 2007). Several United Nations agencies collaborated to set up the Global Terrestrial Observing System (GTOS) and proposed datasets and approaches for essential terrestrial climate variables for climate change assessment (Sessa & Dolman, 2008). There are a number of global data collection initiatives, such as the Amazon Forest Inventory Network (RAINFOR) (Malhi et al., 2004), ForestPlots (ForestPlots.net et al., 2021), the FLUXNET Network (Baldocchi et al., 2001), the Food and Agriculture Organization of the United Nations (FAO) Global Forest Resources Assessment (Keenan et al., 2015), the TRY Plant Trait Database (Fraser, 2020), a database of northern peatland soil properties and Holocene carbon and nitrogen accumulation (Loisel et al., 2014), the newly established PeatDataHub (Young et al., 2016) and the Eyes on the Bog citizen science initiative (Lindsay et al., 2019).

However, environmental science lacks a unified process that can be used by researchers in different disciplines to propose, agree, and prioritise what should be measured and how it should be measured. To tackle this problem, N8 AgriFood researchers from the Yorkshire Integrated Catchment Solutions Programme (iCASP) project and the Resilient Dairy Landscapes project (see Grant information) worked with the Global Peatlands Initiative led by the United Nations Environment Programme to identify and standardise core soil health and other core common outcomes that are likely to be affected by agri-environment scheme options for extensive agriculture on UK and tropical peatlands. While this is only one soil type, it is hoped that if the approach works for peatlands, it could be extended to other soil types and systems.

These problems have been addressed in the medical domain through the EQUATOR initiative (www.equator-network.org), which has established reporting guidelines that are widely adhered to by research funders and journal editors. This has substantially increased the amount of research that can be synthesised for evidence-based policy and practice, helping to reduce research waste, which has been estimated at 80% in the medical domain (Glasziou & Chalmers, 2018). The Core Outcome Measures in Effectiveness Trials (COMET) initiative now provides methodological guidance on developing core common outcome sets, which has informed our work.

To apply these methods the environmental domain, we designed a workshop that included academics, the third sector, and policy and practice stakeholders to assist in the identification of the full range of climate, biodiversity and hydrology outcomes that can be measured in peatland systems. A voting procedure was used to identify the essential core outcomes that should be measured wherever possible. Researchers and practitioners will then be able to choose if they wish to measure outcomes from the core list, which in turn increases the likelihood that their data can be combined with other studies in future evidence syntheses. Alternatively, they may choose to measure different outcomes if they are more relevant to their study, in the knowledge that these will be less likely to be synthesisable in the future. Standardisation need not stifle innovation (Clarke, 2007) as researchers frequently measure additional study specific outcomes, as well as either mandatory or desirable core outcomes (Reed et al., in press).

One of the implications of this approach is that the outcomes will need to be measured in a number of agreed ways and it will depend on the type of project. This will also be tackled as part of the process of developing core outcomes. A small number of outcome measures are being chosen as a starting point and all available methods for measuring these outcomes will be reviewed. The next step is to identify criteria to evaluate methods and protocols for collecting outcome data. These might, for example, include accuracy, ease of use and cost. We will then identifying relevant methods and protocols for assessing each prioritised outcome measure, and systematically evaluating each identified method and protocol against each criterion. The result will be a menu of reliable methods for each core outcome, which will be published in the peer-reviewed literature and made available via the Global Peatlands Initiative. They will range from very accurate but potentially highly technical and expensive methods, through to reliable but easier to measure methods that could be used by practitioners or in citizen science initiatives. This work has the potential to significantly reduce research waste in these research fields and enable more research to be synthesised to generate robust evidence for policy and practice.

Data availability

No data is associated with this article.

Publisher’s note

This article was originally published on the Emerald Open Research platform hosted by F1000, under the “Sustainable Food Systems and N8 AgriFood” for All gateway.

The original DOI of the article was 10.35241/emeraldopenres.13833.2

This is Version 2 of the article. Version 1 is available as supplementary material.

Author roles

Reed MS: Conceptualization, Formal Analysis, Funding Acquisition, Methodology, Project Administration, Writing - Original Draft Preparation, Writing - Review & Editing; Chapman PJ: Conceptualization, Writing - Original Draft Preparation, Writing - Review & Editing; Ziv G: Writing - Original Draft Preparation, Writing - Review & Editing; Stewart G: Writing - Original Draft Preparation; Kendall H: Writing - Review & Editing; Taylor A: Writing - Review & Editing; Kopansky D: Writing - Review & Editing

Amendments from Version 1

The definition used in the Agriculture Act (it is no longer a Bill) is now summarised in the paper. The literature has been updated to include a number of key new syntheses, including a signpost to Conservation Evidence, and the section has been re-organised as suggested. ough we focus less on open science protocols, these are relevant and are now mentioned, alongside the other links in the chain, and we have reviewed other similar initiatives to provide a fuller account of related work in this area. Ideas around funding new data collection for future synthesis work from the reviewers have been integrated into the paper. We have now cited the ELMS database and provided an overview of similar initiatives with relevant references.

Grant information:

The Resilient Dairy Landscapes project is funded by the Global Food Security's ‘Resilience of the UK Food System in a Global Context’ programme along with the Biotechnology and Biological Sciences Research Council (BBSRC), the Economic and Social Research Council (ESRC), the Natural Environment Research Council (NERC) and the Scottish Government. The Yorkshire Integrated Catchment Solutions Programme (iCASP) is funded by NERC. Research on core common outcomes is funded by the Peatland Tipping Points project which is part of NERC's Valuing Nature Programme, as well as the Global Peatland Initiative's Global Peatland Research Funding Working Group. The latter is funded by the Global Challenges Research Fund, ESRC Impact Accelerator and Research England Higher Education Innovation Fund. The Global Peatlands Initiative is part of the International Climate Initiative (IKI). The Federal Ministry for the Environment, Nature Conservation and Nuclear Safety (BMU) supports this initiative on the basis of a decision adopted by the German Bundestag.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests

Competing interests: Prof Reed is Research Lead for IUCN UK Peatland Programme

Reviewer response for version 2

Ignasi Bartomeus, Estación Biológica de Doñana (EBD-CSIC), Sevilla, Spain

Competing interests: No competing interests were disclosed.

This review was published on 24 August 2022.

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Recommendation: approve

The authors added a series of amendments that complement and clarify the existing evidence. I have no further comments on this manuscript, which I hope will stimulate the uptake of evidence-based solutions by policymakers.

Is the topic of the opinion article discussed accurately in the context of the current literature?

Yes

Is the argument information presented in such a way that it can be understood by a non-academic audience?

Yes

Could any solutions being offered be effectively implemented in practice?

Yes

Are arguments sufficiently supported by evidence from the published literature?

Yes

Are all factual statements correct and adequately supported by citations?

Yes

Is real-world evidence provided to support any conclusions made?

Yes

Does the piece present solutions to actual real world challenges?

Yes

Are the conclusions drawn balanced and justified on the basis of the presented arguments?

Yes

Reviewer Expertise:

Biodiversity

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Reviewer response for version 1

Ignasi Bartomeus, Estación Biológica de Doñana (EBD-CSIC), Sevilla, Spain; Elena Velado-Alonso, Integrative Ecology Department, Doñana Biological Station (EBD-CSIC), Sevilla, Spain

Competing interests: No competing interests were disclosed.

This review was published on 24 January 2022.

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Recommendation: approve-with-reservations

This opinion paper argues that agri-environmental schemes need to be based on solid evidence. We concur with the author's opinion, but nonetheless, some clarifications are warranted to provide a balanced view.

First, a clear definition of what is considered as a “public good” under the new UK Agriculture Bill would help broader audiences to better understand which is referred under this term and which are the similarities with other commonly used keywords such as “Ecosystem Services” and differences to the economic term.

Second, while the authors make a short summary of the existing evidence, most papers mentioned are published before 2015. A brief comment on how the last 5 years of research may have increased the existing evidence may be interesting. In this regard, efforts by https://www.conservationevidence.com/ may be relevant. Besides, we suggest a reorganization of this section (“Evidence for public goods from agri-environment schemes”) into a) current shortcomings – clarifying major identified limitations: lack of evidence, biases, inadequate experimental design, effects of spatial scales and temporal lags– and how to overcome them, and b) intervention effectiveness – including achieved success by implemented interventions.

The piece makes a strong emphasis on synthesis, and we appreciate a call for common protocols that allow data integration, but the full chain of events is more complex than that and includes experimental design, data gathering, data publishing, reproducibility, and discoverability, which are non-trivial. Many words have been written on how to make data FAIR (https://www.go-fair.org/fair-principles/) and the authors can capitalize on this work to suggest not only common protocols, but ways of ensuring data is reproducible, preserved in long term, discoverable and interoperable. A key feature here would be embracing already existing metadata standards (e.g. EML).

The authors also point to the importance of funding synthesis, but along these lines, we would also like to stress the importance of funding the collection of the appropriate data (e.g. continuous monitoring on farms). Also note that interventions with mixed or limited evidence may emerge when the success of the intervention depends on external factors, and hence understanding when and where each intervention may work better is of paramount importance. This is in line with the author's message, but it can be reinforced to avoid neglecting interventions with mixed support, as those may be important for a given situation. In this sense, the uptake of interventions could be integrated into the discussion in relation to extension and transfer.

You mention the ELMS database, but It would be great to have a reference for it. In addition, there are already initiatives to create standardized indicators (e.g. Herzog F, Balázs K, Dennis P, Friedel J, Geijzendorffer I, Jeanneret P, Kainz M, Pointereau P. Biodiversity indicators for European farming systems: a guidebook. ART-Schriftenreihe. 2012(17) ¹.) which may be relevant to avoid duplicating efforts.

Is the topic of the opinion article discussed accurately in the context of the current literature?

Yes

Is the argument information presented in such a way that it can be understood by a non-academic audience?

Yes

Could any solutions being offered be effectively implemented in practice?

Yes

Are arguments sufficiently supported by evidence from the published literature?

Yes

Are all factual statements correct and adequately supported by citations?

Yes

Is real-world evidence provided to support any conclusions made?

Yes

Does the piece present solutions to actual real world challenges?

Yes

Are the conclusions drawn balanced and justified on the basis of the presented arguments?

Yes

Reviewer Expertise:

Biodiversity and ecosystem services.

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

References

1. Biodiversity indicators for European farming systems: A guidebook. Agroscope. 2012.

Reviewer response for version 1

Piran White, Department of Environment and Geography, University of York, York, United Kingdom

Competing interests: No competing interests were disclosed.

This review was published on 1 April 2021.

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Recommendation: approve

This is a nicely-written and argued article about the future of agri-environment policy and the need for a better evidence base, including suggestions as to how this could be developed more strategically. I have no major comments on how the article has been constructed, although I have a few points on the conclusions drawn and potential unintended consequences that could arise from the proposed future approach.

I agree with the principal argument of this article that future agri-environment policy needs to be more evidence-based in terms of outcomes. However, if a result of this is that “Future policy could then prioritise public money for the public goods that can most reliably be delivered”, this potentially means that some public goods may not be delivered either because their delivery is inherently unreliable and/or because there is uncertainty around their delivery. Since this is a potential consequence of the type of targeting the authors recommend, I think there should be some more discussion around this, in relation to the ‘hard-to-deliver’ public goods.

Equally, because there is more evidence for beneficial effects for common species, this approach to prioritisation may mean that less common or rare species are overlooked. The authors state that if there is insufficient evidence available to support interventions, then “interventions would remain on hold pending sufficient further evidence to enable synthesis work to be done to evaluate the evidence-base as a whole.” The resulting delays could be critical for some endangered species. It may be more pertinent to consider different standards of evidence, since for rarer species or harder-to-measure outcomes and greater uncertainties around these, strong evidence is inherently harder to produce.

The authors suggest a lower payment rate for interventions with less certainty. My concern with this is that farmers may go preferentially for the higher payment rate interventions and interventions with lower payment rates would have a much lower take-up rate.

I agree on the need for common outcomes. Inconsistency of outcomes measures and methodologies has been a major limitation on the evidence base on many environmental interventions. Natural England is also doing work on indicators to be used to environmental outcomes. We have supported them with this work, and an online tool showing the use of different indicators can be found at https://natcapindicators.simomics.com/#/.

Is the topic of the opinion article discussed accurately in the context of the current literature?

Yes

Are arguments sufficiently supported by evidence from the published literature?

Yes

Are all factual statements correct and adequately supported by citations?

Yes

Are the conclusions drawn balanced and justified on the basis of the presented arguments?

Yes

Reviewer Expertise:

Biodiversity, ecosystem services, environmental management

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.