Carol Lefebvre, Julie Glanville, Simon Briscoe, Anne Littlewood, Chris Marshall, Maria-Inti Metzendorf, Anna Noel-Storr, Tamara Rader, Farhad Shokraneh, James Thomas, L. Susan Wieland; on behalf of the Cochrane Information Retrieval Methods Group

Cite this chapter as: Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6. London: Cochrane, 2019.

This PDF chapter is made available for personal use of Cochrane members only, and is not for general distribution. All content remains the copyright of Cochrane.

4.1 Introduction

Cochrane Reviews take a systematic and comprehensive approach to identifying studies that meet the eligibility criteria for the review. This chapter outlines some general issues in searching for studies; describes the main sources of potential studies; and discusses how to plan the search process, design and carry out search strategies, manage references found during the search process, correctly document the search process and select studies from the search results.

This chapter aims to provide review authors with background information on all aspects of searching for studies so that they can better understand the search process. All authors of systematic reviews should, however, identify an experienced medical/healthcare librarian or information specialist to provide support for the search process. The chapter also aims to provide advice and guidance for medical/healthcare librarians and information specialists (within and beyond Cochrane) involved in the search process to identify studies for inclusion in systematic reviews.

This chapter focuses on searching for randomized trials. Many of the search principles discussed, however, will also apply to other study designs. Considerations for searching for non-randomized studies are discussed in Chapter 24 (see also Chapter 19 when these are specifically for adverse effects). Other discussion of searching for specific types of evidence appears in chapters dedicated to these types of evidence, such as Chapter 17 on complex and public health interventions, Chapter 20 on economics evidence and Chapter 21 on qualitative research.

An online Technical Supplement to this chapter provides more detail on searching methods and is available from Cochrane Training.

4.2 General issues

4.2.1 Role of the information specialist/librarian

Medical/healthcare librarians and information specialists have an integral role in the production of Cochrane Reviews. There is increasing evidence to support the involvement of an information specialist in the review to improve the quality of various aspects of the search process (Rethlefsen et al 2015, Meert et al 2016, Metzendorf 2016).

Most Cochrane Review Groups (CRGs) employ an information specialist to support authors. The range of services, however, offered by CRGs and/or their information specialists varies according to the resources available. Cochrane Review authors should, therefore, contact their Cochrane Information Specialist at the earliest stage to find out what advice and support is available to them. Authors conducting their own searches should seek advice from their Cochrane Information Specialist not only on which sources to search, but also with respect to the exact strategies to be run (see Section 4.4). If the CRG does not provide this service or employ an information specialist, we recommend that review authors seek guidance from a medical/healthcare librarian or information specialist, preferably one with experience in supporting systematic reviews.

Cochrane Information Specialists are responsible for providing assistance to authors with searching for studies for inclusion in their reviews, and for keeping up to date with Cochrane methodological developments in information retrieval (Littlewood et al 2017). A key element of the role is the maintenance of a Specialized Register for their Review Group, containing reports of trials relating to the group’s scope. Within the limits of licensing restrictions, the content of these group registers is shared with users worldwide via the Cochrane Central Register of Controlled Trials (CENTRAL), part of the Cochrane Library (see Section 4.3.3).

Most CRGs offer support to authors in study identification from the early planning stage to the final write-up of the review, and the support available may include some or all of the following:

• advising authors on which databases and other sources to search;
• designing, or providing guidance on designing, search strategies for the main bibliographic databases and/or trials registers;
• running searches in databases and/or registers available to the review team;
• saving and collating search results, and sharing them with authors in appropriate formats;
• advising authors on how to run searches in other sources and how to download results;
• drafting, or assisting authors in drafting, the search methods sections of a Cochrane Protocol and Review and/or Update;
• ensuring that Cochrane Protocols, Reviews and Updates meet the requirements set out in the Methodological Expectations of Cochrane Intervention Reviews (MECIR) relating to searching activities for reviews;
• organizing translations, or at least data extraction, of papers where required to enable authors to assess papers for inclusion/exclusion in their reviews;
• obtaining copies of trial reports for review teams when required (within copyright legislation);
• providing advice and support to author teams on the use of reference management tools, and other software used in review production, including review production tools such as RevMan, Covidence and EPPI-Reviewer; and
• checking and formatting the references to included and/or excluded studies in line with the Cochrane Style Manual.

The Cochrane Information Specialists’ Handbook (Chapter 6, Author support) contains further information about how Cochrane Information Specialists can support authors (Littlewood et al 2017).

4.2.2 Minimizing bias

Systematic reviews require a thorough, objective and reproducible search of a range of sources to identify as many eligible studies as possible (within resource limits). This is a major factor distinguishing systematic reviews from traditional narrative reviews, which helps to minimize bias and achieve more reliable estimates of effects and uncertainties. A search of MEDLINE alone is not considered adequate. Research evidence indicates that not all known published randomized trials are available in MEDLINE and that even if relevant records are in MEDLINE, it can be difficult to retrieve them (see Section 4.3.2).

Going beyond MEDLINE is important not only for ensuring that as many relevant studies as possible are identified, but also to minimize selection bias for those that are found. Relying exclusively on a MEDLINE search may retrieve a set of reports unrepresentative of all reports that would have been identified through a wider or more extensive search of several sources.

Time and budget restraints require the review team to balance the thoroughness of the search with efficiency in the use of time and funds. The best way of achieving this balance is to be aware of, and try to minimize, the biases such as publication bias and language bias that can result from restricting searches in different ways (see Chapters 8 and 13 for further guidance on assessing these biases). Unlike for tasks such as study selection or data extraction, it is not considered necessary (or even desirable) for two people to conduct independent searches in parallel. It is strongly recommended, however, that all search strategies should be peer reviewed by a suitably qualified and experienced medical/healthcare librarian or information specialist (see Section 4.4.8).

4.2.3 Studies versus reports of studies

Systematic reviews have studies as the primary units of interest and analysis. A single study may have more than one report about it, and each of these reports may contribute useful information for the review (see Section 4.6.1). For most of the sources listed in Section 4.3, the search process will retrieve individual reports of studies, so that multiple reports of the same study will need to be identified and associated with each other manually by the review authors. There is, however, an increasing number of study-based sources, which link multiple records of the same study together, such as the Cochrane Register of Studies and the Specialized Registers of a number of CRGs and Fields (see online Technical Supplement), and some other trials registers and regulatory and industry sources. Processes and software to select and group publications by study are discussed in Section 4.6.

4.2.4 Copyright and licensing

It is Cochrane policy that all review authors and others involved in Cochrane should adhere to copyright legislation and the terms of database licensing agreements. With respect to searching for studies, this refers in particular to adhering to the terms and conditions of use when searching databases and other sources and downloading records, as well as adhering to copyright legislation when obtaining copies of publications. Review authors should seek guidance on this from their medical/healthcare librarian or information specialist, as copyright legislation varies across jurisdictions and licensing agreements vary across organizations.

4.3 Sources to search

4.3.1 Bibliographic databases

4.3.1.1 Introduction to bibliographic databases

The search for studies in a Cochrane Review should be as extensive as possible in order to reduce the risk of reporting bias and to identify as much relevant evidence as possible (see MECIR Box 4.3.a). Searches of health-related bibliographic databases are generally the most efficient way to identify an initial set of relevant reports of studies (EUnetHTA 2017). Database selection should be guided by the review topic (Suarez-Almazor et al 2000, Stevinson and Lawlor 2004, Lorenzetti et al 2014). When topics are specialized, cross-disciplinary, or involve emerging technologies (Rice et al 2016), additional databases may need to be identified and searched (Wallace et al 1997, Stevinson and Lawlor 2004).

MECIR Box 4.3.a Relevant expectations for conduct of intervention reviews

The three bibliographic databases generally considered to be the most important sources to search for reports of trials are CENTRAL, MEDLINE (Halladay et al 2015, Sampson et al 2016) and Embase (Woods and Trewheellar 1998, Sampson et al 2003, Bai et al 2007). These databases are described in more detail in Sections 4.3.1.2 and 4.3.1.3 and in the online Technical Supplement. For Cochrane Reviews, CENTRAL, MEDLINE and Embase (if access to Embase is available to the review team) should be searched (see MECIR Box 4.3.a). These searches may be undertaken specifically for the review, or indirectly by searching the CRG’s Specialized Register.

Some bibliographic databases, such as MEDLINE and Embase, include abstracts for the majority of recent records. A key advantage of such databases is that they can be searched electronically both for words in the title or abstract and by using the standardized indexing terms, or controlled vocabulary, assigned to each record (see Section 4.3.1.2). Cochrane has developed a database of reports of randomized trials called the Cochrane Central Register of Controlled Trials (CENTRAL), which is published within the Cochrane Library. Since its inception, CENTRAL has been considered to be the best single source of reports of trials that might be eligible for inclusion in Cochrane Reviews (Egger and Smith 1998).

Bibliographic databases are available to individuals for a fee (by subscription or on a ‘pay-as-you-go’ basis) or free at the point of use. They may be available through national provisions, site-wide licences at institutions such as universities or hospitals, through professional organizations as part of their membership packages or free-of-charge on the internet. Some international initiatives provide free or low-cost online access to databases (and full-text journals) over the internet. The Health InterNetwork Access to Research Initiative (HINARI) programme, set up by the World Health Organization (WHO) together with major publishers, provides access to a wide range of databases including the Cochrane Library for healthcare professionals in local, not-for-profit institutions in more than 115 countries, areas and territories. The International Network for the Availability of Scientific Publications (INASP) also provides access to a wide range of databases (and journals) including the Cochrane Library. Electronic Information for Libraries (EIFL) is a similar initiative based on library consortia to support affordable licensing of journals and other sources in more than 60 low-income and transition countries in central, eastern and south-east Europe, the former Soviet Union, Africa, the Middle East and South-east Asia.

The online Technical Supplement provides more detailed information about how to search these sources and other databases. It also provides a list of general healthcare databases by region and healthcare databases by subject area. Further evidence-based information about sources to search can be found on the SuRe Info portal, which is updated twice per year.

4.3.1.2 MEDLINE and Embase

Cochrane Reviews of interventions should include a search of MEDLINE (see MECIR Box 4.3.a). MEDLINE (as of August 2018) contains over 25 million references to journal articles in biomedicine and health from 1946 onwards. More than 5200 journals in about 40 languages are indexed for MEDLINE (US National Library of Medicine 2019).

PubMed provides access to a free version of MEDLINE that also includes up-to-date citations not yet indexed for MEDLINE (US National Library of Medicine 2018). Additionally, PubMed includes records from journals that are not indexed for MEDLINE and records considered ‘out-of-scope’ from journals that are partially indexed for MEDLINE (US National Library of Medicine no date).

MEDLINE is also available on subscription from a number of other database vendors, such as EBSCO, Ovid, ProQuest and STN. Access is usually ‘free at-the-point-of-use’ to members of the institutions paying the subscriptions (e.g. hospitals and universities). Ovid MEDLINE (segment name ‘medall’) covers all of the available content and metadata in PubMed with a delay of one day (except during the annual reload, at the end of each year, when Ovid-MEDLINE will not match the PubMed baseline). Aside from the MEDLINE records, Ovid includes all content types available in PubMed including; Ahead of Print, Supplied by publisher, PubMed-not MEDLINE, In-process citations and citations for books available on the NCBI Bookshelf.

When searching MEDLINE via service providers or interfaces other than Ovid or PubMed, we recommend verification of the exact coverage of the database in relation to PubMed, where no explicit information on this is readily available.

Cochrane Reviews of interventions should include a search of Embase (if access to Embase is available to the review team) (see MECIR Box 4.3.a). Embase (as of June 2018) contains over 30 million records from more than 8000 currently published journals. Embase now includes all MEDLINE records, thus, technically, allowing both databases to be searched simultaneously. Further details on the implications of this for searching are available in the online Technical Supplement. There are more than 6 million records in Embase, from more than 2900 journals that are not indexed in MEDLINE (Elsevier 2016a). Embase includes articles from about 90 countries. Embase Classic provides access to almost 2 million records digitized from the Excerpta Medica print journals (the original print indexes from which Embase was created) from 1947 to 1973 (Elsevier 2016b).

Embase is only available by subscription, either directly via Elsevier (as Embase.com) or from other database vendors, such as Ovid, ProQuest or STN. It is mandatory for Cochrane intervention reviews to include a search of Embase if access is available to the review team (see MECIR Box 4.3.a). Note that Embase is searched regularly by Cochrane for reports of trials. These records are included in CENTRAL (see online Technical Supplement).

The online Technical Supplement provides guidance on how to search MEDLINE and Embase for reports of trials. The actual degree of reference overlap between MEDLINE and Embase varies widely according to the topic, but studies comparing searches of the two databases have generally concluded that a comprehensive search requires that both databases be searched (Lefebvre et al 2008) (see MECIR Box 4.3.a).

Conversely, two recent studies examined different samples of Cochrane Reviews and identified the databases from which the included studies of these reviews originated (Halladay et al 2015, Hartling et al 2016). Halladay showed that the majority of included studies could be identified via PubMed (range 75% to 92%) and Hartling showed that the majority of included studies could be identified by using a combination of two databases, but the two databases were different in each case. Both studies, one across all healthcare areas (Halladay et al 2015) and the other on child health (Hartling et al 2016), report a minimal extent to which the inclusion of studies not indexed in PubMed altered the meta-analyses. Hence, the current recommendation of searching multiple databases needs to be evaluated further, so as to confirm under which circumstances more comprehensive searches of multiple databases is warranted.

4.3.1.3 The Cochrane Central Register of Controlled Trials (CENTRAL)

Since its inception, the Cochrane Central Register of Controlled Trials (CENTRAL) has been recognized as the most comprehensive source of reports of randomized trials (Egger and Smith 1998). CENTRAL is published as part of the Cochrane Library and is updated monthly. As of June 2018, CENTRAL contains over 1,275,000 records of reports of trials/trials registry records potentially eligible for inclusion in Cochrane Reviews, by far the majority of which are randomized trials.

Many of the records in CENTRAL have been identified through systematic searches of MEDLINE and Embase (see online Technical Supplement). CENTRAL, however, also includes citations to reports of randomized trials that are not indexed in MEDLINE, Embase or other bibliographic databases; citations published in many languages; and citations that are available only in conference proceedings or other sources that are difficult to access. It also includes records from trials registers and trials results registers.

These additional records are, for the most part, identified by Cochrane Information Specialists, many of whom conduct comprehensive searches to populate CRG Specialized Registers, collecting records of trials eligible for Cochrane Reviews in their field. These Specialized Registers are included in CENTRAL. Where a Specialized Register is available, for which sufficiently comprehensive searching has been conducted, a search of the Specialized Register may be conducted instead of separately searching CENTRAL, MEDLINE and Embase for a specific review. In these cases, the search will be more precise, but an equivalent number of included studies will be identified with lower numbers of records to screen. There will, however, be a time-lag between records appearing in databases such as MEDLINE or Embase and their inclusion in a Specialized Register.

CENTRAL is available through the Cochrane Library. Many review authors have access free-of-charge at the point-of-use through national provisions and other similar arrangements, or as part of a paid subscription to the Cochrane Library. All Cochrane Information Specialists have access to CENTRAL.

The online Technical Supplement provides information on what is in CENTRAL from MEDLINE, Embase and other sources, as well as guidance on searching CENTRAL.

4.3.1.4 Other bibliographic databases

Many countries and regions produce bibliographic databases that focus on the literature produced in those regions and which often include journals and other literature not indexed elsewhere. There are also subject-specific bibliographic databases, such as AMED (alternative therapies), CINAHL (nursing and allied health) and PsycINFO (psychology and psychiatry). It is highly desirable that searches be conducted of appropriate national, regional and subject specific bibliographic databases (see MECIR Box 4.3.b). Further details are provided in the online Technical Supplement.

Citation indexes are bibliographic databases that record instances where a particular reference is cited, in addition to the standard bibliographic content. Citation indexes can be used to identify studies that are similar to a study report of interest, as it is probable that other reports citing or cited by a study will contain similar or related content.

MECIR Box 4.3.b Relevant expectations for conduct of intervention reviews

4.3.2 Ongoing studies and unpublished data sources

Initiatives to provide access to ongoing studies and unpublished data constitute a fast-moving field (Isojarvi et al 2018). Review authors should therefore consult their medical/healthcare librarian or information specialist for current advice.

It is important to identify ongoing studies, so that when a review is updated these can be assessed for possible inclusion. Awareness of the existence of a possibly relevant ongoing study and its expected completion date might affect not only decisions with respect to when to update a specific review, but also when to aim to complete a review. Information about possibly relevant ongoing studies should be included in the review in the ‘Characteristics of ongoing studies’ table.

Even when studies are completed, some are never published. An association between ‘statistically significant’ results and publication has been documented across a number of studies, as summarized in Chapter 13. Finding out about unpublished studies, and including their results in a systematic review when eligible and appropriate (Cook et al 1993), is important for minimizing bias. Several studies and other articles addressing issues around identifying unpublished studies have been published (Easterbrook et al 1991, Weber et al 1998, Manheimer and Anderson 2002, MacLean et al 2003, Lee et al 2008, Chan 2012, Bero 2013, Schroll et al 2013, Chapman et al 2014, Kreis et al 2014, Scherer et al 2015, Hwang et al 2016, Lampert et al 2016).

There is no easy and reliable single way to obtain information about studies that have been completed but never published. There have, however, been several important initiatives resulting in better access to studies and their results from sources other than the main bibliographic databases and journals. These include trials registers and trials results registers (see Section 4.3.3), regulatory agency sources and clinical study reports (CSRs); (the very detailed reports prepared by industry for regulatory approval) (see Section 4.3.4). A recent study (Halfpenny et al 2016) assessed the value and usability for systematic reviews and network meta-analyses of data from trials registers, CSRs and regulatory authorities, and concluded that data from these sources have the potential to influence systematic review results. Two earlier studies showed that a considerably higher proportion of CSRs prepared for regulatory approval of drugs provided complete information on study methods and results than did trials register records or journal publications (Wieseler et al 2012) and that conventional, publicly available sources (European Public Assessment Reports, journal publications, and trials register records) provide insufficient information on new drugs, especially on patient relevant outcomes in approved subpopulations (Kohler et al 2015).

A Cochrane Methodology Review examined studies assessing methods for obtaining unpublished data and concluded that those carrying out systematic reviews should continue to contact authors for missing data and that email contact was more successful than other methods (Young and Hopewell 2011). An annotated bibliography of published studies addressing searching for unpublished studies and obtaining access to unpublished data is also available (Arber et al 2013). One particular study focused on the contribution of unpublished studies, including dissertations, and studies in languages other than English, to the results of meta-analyses in reviews relevant to children (Hartling et al 2017). They found that, in their sample, unpublished studies and studies in languages other than English rarely had any impact on the results and conclusions of the review. They did, however, concede that inclusion of these study types may have an impact in situations where there are few relevant studies, or where there are ‘questionable vested interests’ in the published literature.

Correspondence can be an important source of information about unpublished studies. It is highly desirable for authors of Cochrane Reviews of interventions to contact relevant individuals and organizations for information about unpublished or ongoing studies (see MECIR Box 4.3.c). Letters of request for information can be used to identify completed but unpublished studies. One way of doing this is to send a comprehensive list of relevant articles along with the eligibility criteria for the review to the first author of reports of included studies, asking if they know of any additional studies (ongoing or completed; published or unpublished) that might be relevant. This approach may be especially useful in areas where there are few trials or a limited number of active research groups. It may also be desirable to send the same letter to other experts and pharmaceutical companies or others with an interest in the area. Some review teams set up websites for systematic review projects, listing the studies identified to date and inviting submission of information on studies not already listed.

MECIR Box 4.3.c Relevant expectations for conduct of intervention reviews

Asking researchers for information about completed but never published studies has not always been found to be fruitful (Hetherington et al 1989, Horton 1997) though some researchers have reported that this is an important method for retrieving studies for systematic reviews (Royle and Milne 2003, Greenhalgh and Peacock 2005, Reveiz et al 2006). The RIAT (Restoring Invisible and Abandoned Trials) initiative (Doshi et al 2013) aims to address these problems by offering a methodology that allows others to re-publish mis-reported and to publish unreported trials. Anyone who can access the trial data and document trial abandonment can use this methodology. The RIAT Support Centre offers free-of-charge support and competitive funding to researchers interested in this approach. It has been suggested that legislation such as Freedom of Information Acts in various countries might be used to gain access to information about unpublished trials (Bennett and Jull 2003, MacLean et al 2003).

4.3.3 Trials registers and trials results registers

A recent study suggested that trials registers are an important source for identifying additional randomized trials (Baudard et al 2017). Cochrane Reviews of interventions should search relevant trials registers and repositories of results (see MECIR Box 4.3.d). Although there are many other trials registers, ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform (ICTRP) portal (Pansieri et al 2017) are considered to be the most important for searching to identify studies for a systematic review. Research has shown that even though ClinicalTrials.gov is included in the WHO ICTRP Search Portal, not all ClinicalTrials.gov records can be successfully retrieved via searches of the ICTRP Search Portal (Glanville et al 2014, Knelangen et al 2018). Therefore, it is not sufficient to search the ICTRP alone. Guidance for searching these and other trials registers is provided in the online Technical Supplement.

In addition to Cochrane, other organizations such as the Agency for Healthcare Research and Quality (AHRQ) (Agency for Healthcare Research and Quality 2014) and the US Institute of Medicine (Institute of Medicine 2011) also advocate searching trials registers.

There has been an increasing acceptance by investigators of the importance of registering trials at inception and providing access to their trials results. Despite perceptions and even assertions to the contrary, however, there is no global, universal legal requirement to register clinical trials at inception or at any other stage in the process, although some countries are beginning to introduce such legislation (Viergever and Li 2015).

Efforts have been made by a number of organizations, including organizations representing the pharmaceutical industry and individual pharmaceutical companies, to begin to provide central access to ongoing trials and in some cases trial results on completion, either on a national or international basis. A recent audit of pharmaceutical companies’ policies on access to trial data, results and methods, however, showed that the commitments made by companies to transparency of trials were highly variable (Goldacre et al 2017). Increasingly, as already noted, trials registers such as ClinicalTrials.gov also contain the results of completed trials, not just simply listing the details of the trial.

MECIR Box 4.3.d Relevant expectations for conduct of intervention reviews

4.3.4 Regulatory agency sources and clinical study reports

Potentially relevant regulatory agency sources include the EU Clinical Trials Register, Drugs@FDA and OpenTrialsFDA. Details of these are provided in the online Technical Supplement. Clinical study reports (CSRs) are the reports of clinical trials providing detailed information on the methods and results of clinical trials submitted in support of marketing authorization applications. In late 2010, the European Medicines Agency (EMA) began releasing CSRs (on request) under their Policy 0043. In October 2016, they began to release CSRs under their Policy 0070. The policy applies only to documents received since 1 January 2015. The terms of use for access are based on the purposes to which the clinical data will be put.

A recent study by Jefferson and colleagues (Jefferson et al 2018) that looked at use of regulatory documents in Cochrane Reviews, found that understanding within the Cochrane community was limited and guidance and support would be required if review authors were to engage with regulatory documents as a source of evidence. Specifically, guidance on how to use data from regulatory sources is needed. For more information about using CSRs, see the online Technical Supplement. Further guidance on collecting data from CSRs is provided in Chapter 5.

4.3.5 Other sources

The online Technical Supplement describes several other important sources of reports of studies. The term ‘grey literature’ is often used to refer to reports outside of traditional commercial publishing. Review authors should generally search sources such as dissertations and conference abstracts (see MECIR Box 4.3.e).

Review authors may also consider searching the internet, handsearching of journals and searching full texts of journals where available (see online Technical Supplement for details). They should examine previous reviews on the same topic and check reference lists of included studies and relevant systematic reviews (see MECIR Box 4.3.e).

MECIR Box 4.3.e Relevant expectations for conduct of intervention reviews

4.4 Designing search strategies

4.4.1 Introduction to search strategies

This section highlights some of the issues to consider when designing search strategies. Designing search strategies can be complex and the section does not fully address the many complexities in this area. Review teams will benefit from the skills and expertise of a medical/healthcare librarian or information specialist. Many of the issues highlighted relate to both the subject aspects of the search (e.g. the PICO elements) and to the study method (e.g. randomized trials). For a search to be robust, both aspects require attention to be sure that relevant records are not missed.

Issues to consider in planning a search include:

• the nature or type of the intervention(s) being assessed;
• the complexity of the review question and the need to consider additional conceptual frameworks (see Chapters 3 and 17);
• the time period when any evaluations of the interventions may have taken place (as specified in the review protocol) (see Section 4.4.5);
• any geographic considerations, such as the need to search the African Index Medicus for studies relating to African populations or the Chinese literature for studies in Chinese herbal medicine (see online Technical Supplement);
• whether the review is limited to randomized trials or other study designs are eligible (see Chapter 24);
• whether a validated methodological search filter (for specific study designs) is available (see Section 4.4.7);
• whether unpublished data are to be sought specifically; and
• whether the review has specific eligibility criteria around study design to address adverse effects (see Chapter 19), economic issues (see Chapter 20) or qualitative research questions (see Chapter 21), in which case searches to address these criteria should be undertaken (see MECIR Box 4.4.a).

Further evidence-based information about designing search strategies can be found on the SuRe Info portal, which is updated twice per year.

MECIR Box 4.4.a Relevant expectations for conduct of intervention reviews

4.4.2 Structure of a search strategy

The starting point for developing a search strategy is to consider the main concepts being examined in a review. This is often referred to as PICO – that is Patient (or Participant or Population or Problem), Intervention, Comparison and Outcomes (Richardson et al 1995): see also Chapters 2 and 3 for guidance on developing and refining PICO definitions that will be operationalized in the search strategy. Examples are provided in the appendices to the Cochrane Information Specialists’ Handbook (Littlewood et al 2017). For a Cochrane Review, the review objective should provide the PICO concepts, and the eligibility criteria for studies to be included will further assist in the selection of appropriate subject headings and text words for the search strategy.

The structure of search strategies in bibliographic databases should be informed by the main concepts of the review (see Chapter 3), using appropriate elements from PICO and study design (see MECIR Box 4.4.b). It is usually unnecessary, however, and may even be undesirable, to search on every aspect of the review’s clinical question. Although a research question may specify particular comparators or outcomes, these concepts may not be well described in the title or abstract of an article and are often not well indexed with controlled vocabulary terms. Therefore, in general databases, such as MEDLINE, a search strategy will typically have three sets of terms: (i) terms to search for the health condition of interest, i.e. the population; (ii) terms to search for the intervention(s) evaluated; and (iii) terms to search for the types of study design to be included. Typically, a broad set of search terms will be gathered for each concept, and combined with the OR Boolean operator to achieve sensitivity within concepts. The results for each concept are then combined using the AND Boolean operator, to ensure each concept is represented in the final search results.

It is important to consider the structure of the search strategy on a question-by-question basis. In some cases it is possible and reasonable to search for the comparator, for example if the comparator is explicitly placebo; in other cases the outcomes may be particularly well defined and consistently reported in abstracts. The advice on whether or not to search for outcomes for adverse effects differs from the advice given earlier (see Chapter 19).

MECIR Box 4.4.b Relevant expectations for conduct of intervention reviews

Some search strategies may not easily divide into the structure suggested, particularly for reviews addressing complex or unknown interventions, or diagnostic tests (Huang et al 2006, Irvin and Hayden 2006, Petticrew and Roberts 2006, de Vet et al 2008, Booth 2016). Cochrane Reviews of public health interventions and of qualitative data may adopt very different search approaches to those described here (Lorenc et al 2014, Booth 2016) (see Chapter 17 on complex and public health interventions, and Chapter 21 on qualitative research). Some options to explore for such situations include:

• use a single concept such as searching for the intervention alone (Khan et al 2001);
• break a concept into two or more subconcepts;
• use a multi-stranded or multi-faceted approach that uses a series of searches, with different combinations of concepts, to capture a complex research question (Lefebvre et al 2013);
• use a variety of different search approaches to compensate for when a specific concept is difficult to define (Shemilt et al 2014); or
• use citation searching on key papers in addition to a database search (Haddaway et al 2015, Hinde and Spackman 2015) (see online Technical Supplement).

4.4.3 Sensitivity versus precision

Searches for systematic reviews aim to be as extensive as possible in order to ensure that as many of the relevant studies as possible are included in the review. It is, however, necessary to strike a balance between striving for comprehensiveness and maintaining relevance when developing a search strategy.

The properties of searches are often quantified using ‘sensitivity’ (also called ‘recall’) and ‘precision’ (see Table 4.4.a). Sensitivity is defined as the number of relevant reports identified divided by the total number of relevant reports in the resource. Precision is defined as the number of relevant reports identified divided by the total number of reports identified. Increasing the comprehensiveness (or sensitivity) of a search will reduce its precision and will usually retrieve more non-relevant reports.

Searches for Cochrane Reviews should seek to maximize sensitivity whilst striving for reasonable precision (see MECIR Box 4.4.b). Article abstracts identified through a database search can usually be screened very quickly to ascertain potential relevance. At a conservatively estimated reading rate of one or two abstracts per minute, the results of a database search can be screened at the rate of 60–120 per hour (or approximately 500–1000 over an 8-hour period), so the high yield and low precision associated with systematic review searching may not be as daunting as it might at first appear in comparison with the total time to be invested in the review.

Table 4.4.a Sensitivity and precision of a search

4.4.4 Controlled vocabulary and text words

MEDLINE and Embase (and many other databases) can be searched using a combination of two retrieval approaches. One is based on text words, that is terms occurring in the title, abstract or other relevant fields available in the database. The other is based on standardized subject terms assigned to the references by indexers (specialists who appraise the articles and describe their topics by assigning terms from a specific thesaurus or controlled vocabulary). Searches for Cochrane Reviews should use an appropriate combination of these two approaches (see MECIR Box 4.4.c). Approaches for identifying text words and controlled vocabulary to combine appropriately within a search strategy, including text mining approaches, are presented in the online Technical Supplement.

 

MECIR Box 4.4.c Relevant expectations for conduct of intervention reviews

4.4.5 Language, date and document format restrictions

Searches should capture as many studies as possible that meet the eligibility criteria, ensuring that relevant time periods and sources are covered and not restricted by language or publication status (see MECIR Box 4.3.a). Review authors should justify the use of any restrictions in the search strategy on publication date and publication format (see MECIR Box 4.4.d). For example, excluding letters is not recommended because letters may contain important additional information relating to an earlier trial report or new information about a trial not reported elsewhere (Iansavichene et al 2008). In addition, articles indexed as ‘Comments’ should not be routinely excluded without further examination as these may contain early warnings of suspected fraud (see Section 4.4.6).

MECIR Box 4.4.d Relevant expectations for conduct of intervention reviews

Evidence indicates that excluding non-English studies does not change the conclusions of most systematic reviews (Morrison et al 2012, Jiao et al 2013, Hartling et al 2017), although exceptions have been observed for complementary and alternative medicine (Moher et al 2003, Pham et al 2005, Wu et al 2013). There is, however, also research related to language bias that supports the inclusion of non-English studies in systematic reviews (Egger et al 1997). For further discussion of these issues see Chapter 13.

Inclusion of non-English studies may also increase the precision of the result and the generalizability and applicability of the findings. There may be differences in therapeutic response to pharmaceutical agents according to ethnicity, either because of phenotype and pathogenesis of disease due to environmental factors or because of population pharmacogenomics and pharmacogenetics (Brusselle and Blasi 2015). The inclusion of non-English studies also makes it possible to perform sensitivity analyses to find out if there is any geographical bias in reporting the positive findings (Vickers et al 1998, Kaptchuk 1999). It also could be an indicator of quality of systematic reviews (Wang et al 2015).

Limiting searching to databases containing predominantly English-language records, even if no language restrictions are applied, may result in missed relevant studies (Pilkington et al 2005). Review authors should, therefore, attempt to identify and assess for eligibility all possibly relevant reports of trials irrespective of language of publication. If a Cochrane Review team requires help with translation of and/or data extraction from non-English language reports of studies, they should seek assistance to do so (this is a common task for which volunteer assistance can be sought via Cochrane’s TaskExchange platform, accessible to both Cochrane and non-Cochrane review teams). Where it is not possible to extract the relevant information and data from non-English language reports, the review team should file the study in ‘Studies Awaiting Classification’ rather than ‘Excluded Studies’, to inform readers of the review of the availability of other possibly relevant reports and reflect this information in the PRISMA flow diagram as ‘Studies Awaiting Classification’.

4.4.6 Identifying fraudulent studies, other retracted publications, errata and comments

When considering the eligibility of studies for inclusion in a Cochrane Review, it is important to be aware that some studies may have been found to contain errors or to be fraudulent or may, for other reasons, have been corrected or retracted since publication. Review authors should examine any relevant retraction statements and errata for information (MECIR Box 4.4.e). This applies both to ‘new’ studies identified for inclusion in a review and to studies that are already included in a review when the review is updated. For review updates, it is important to search MEDLINE for the latest version of the citations to the records for the (previously) included studies, in case they have since been corrected or retracted.

Errata are published to correct unintended errors (accepted as errors by the author(s)). Retraction notices are published (usually by the journal editor) where data have been found to be fraudulent, for example in the case of plagiarism. Comments are published under a range of circumstances including when errors are suggested by others and also for early concerns regarding fraud.

Including data from studies that are fraudulent or studies that include errors can have an impact on the overall estimates in systematic reviews. Details of how to identify fraudulent studies, other retracted publications, errata and comments are described in the online Technical Supplement.

MECIR Box 4.4.e Relevant expectations for conduct of intervention reviews

4.4.7 Search filters

Search filters are search strategies that are designed to retrieve specific types of records, such as those of a particular methodological design. When searching for randomized trials in humans, a validated filter should be used to identify studies with the appropriate design (see MECIR Box 4.4.f). Filters to identify randomized trials have been developed specifically for MEDLINE and Embase: see the online Technical Supplement for details. CENTRAL, however, aims to contain only reports with study designs possibly relevant for inclusion in Cochrane Reviews, so searches of CENTRAL should not use a trials ‘filter’ or be limited to human studies.

The InterTASC Information Specialists’ Subgroup Search Filter Resource offers a collection of search filters, focusing predominantly on methodological search filters and providing critical appraisals of some of these filters. The site includes, amongst others, filters for identifying systematic reviews, randomized and non-randomized studies and qualitative research in a range of databases and across a range of service providers (Glanville et al 2008). For further discussion around the design and use of search filters, see the online Technical Supplement.

MECIR Box 4.4.f: Relevant expectations for conduct of intervention reviews

4.4.8 Peer review of search strategies

It is strongly recommended that search strategies should be peer reviewed. Peer review of search strategies is increasingly recognized as a necessary step in designing and executing high-quality search strategies to identify studies for possible inclusion in systematic reviews. Studies have shown that errors occur in the search strategies underpinning systematic reviews (Sampson and McGowan 2006) and that search strategies are not always conducted or reported to a high standard (Mullins et al 2014, Layton 2017). The PRESS Evidence-Based Checklist can be used to assess which elements are important in peer review of electronic search strategies (McGowan et al 2016a, McGowan et al 2016b). The checklist covers not only the technical accuracy of the strategy (line numbers, spellings, etc), but also that the search strategy covers all relevant aspects of the protocol and has interpreted the research question appropriately. Research has shown that peer review using a specially designed checklist can improve the quality of searches (Relevo and Paynter 2012, Spry et al 2013). The names, credentials and institutions of the peer reviewers of the search strategies should be noted in the review (with their permission) in the Acknowledgements section.

4.4.9 Alerts

Alerts, also called literature surveillance services, ‘push’ services or SDIs (selective dissemination of information), are an excellent method of staying up to date with the medical literature currently being published, as a supplement to designing and running specific searches for specific reviews. In practice, alerts are based on a previously developed search strategy, which is saved in a personal account on the database platform (e.g. ‘My EBSCOhost – search alerts’ on EBSCO, ‘My searches & alerts’ on Ovid and ‘MyNCBI – saved searches’ on PubMed). These saved strategies filter the content as the database is being updated with new information. The account owner is notified (usually via email) when new publications meeting their specified search parameters are added to the database. In the case of PubMed, the alert can be set up to be delivered weekly or monthly, or in real-time and can comprise email or RSS feeds.

For review authors, alerts are a useful tool to help monitor what is being published in their review topic after the original search has been conducted. By following the alert, authors can become aware of a new study that meets the review’s eligibility criteria, and decide either to include it in the review immediately or mention it as a ‘study awaiting assessment’ for inclusion during the next review update (see online Chapter IV). Authors should consider setting up alerts so that the review can be as current as possible at the time of publication.

Another way of attempting to stay current with the literature as it emerges is by using alerts based on journal tables of contents (TOCs). These usually cannot be specifically tailored to the information needs in the same way as search strategies developed to cover a specific topic. They can, however, be a good way of trying to keep up to date on a more general level by monitoring what is currently being published in journals of interest. Many journals, even those that are available by subscription only, offer TOC alert services free of charge. In addition, a number of publishers and organizations offer TOC services (see online Technical Supplement). Use of TOCs is not proposed as a single alternative to the various other methods of study identification necessary for undertaking systematic reviews, rather as a supplementary method. (See also Chapter 22, Section 22.2 for a discussion of new technologies to support evidence surveillance in the context of ‘living’ systematic reviews.)

4.4.10 Timing of searches

The published review should be as up to date as possible. Searches for all the relevant databases should be rerun prior to publication, if the initial search date is more than 12 months (preferably six months) from the intended publication date (see MECIR Box 4.4.g). The results should also be screened to identify potentially eligible studies. Ideally, the studies should be incorporated fully in the review. If not, then the potentially eligible studies will need to be reported as references under ‘Studies awaiting classification’ (or under ‘Ongoing studies’ if they are not yet completed).

MECIR Box 4.4.g Relevant expectations for conduct of intervention reviews

4.4.11 When to stop searching

Developing a search is often an iterative and exploratory process. It involves exploring trade-offs between search terms and assessing their overall impact on the sensitivity and precision of the search. It is often difficult to decide in a scientific or objective way when a search is complete and search strategy development can stop. The ability to decide when to stop typically develops through experience of developing many strategies. Suggestions for stopping rules have been made around the retrieval of new records, for example to stop if adding in a series of new terms to a database search strategy yields no new relevant records, or if precision falls below a particular cut-off (Chilcott et al 2003). Stopping might also be appropriate when the removal of terms or concepts results in missing relevant records. Another consideration is the amount of evidence that has already accrued: in topics where evidence is scarce, authors might need to be more cautious about deciding when to stop searching. Although many methods have been described to assist with deciding when to stop developing the search, there has been little formal evaluation of the approaches (Booth 2010, Wood and Arber 2019).

At a basic level, investigation is needed as to whether a strategy is performing adequately. One simple test is to check whether the search is finding the publications that have been recommended as key publications or that have been included in other similar reviews (EUnetHTA 2017). It is not enough, however, for the strategy to find only those records, otherwise this might be a sign that the strategy is biased towards known studies and other relevant records might be being missed. In addition, citation searches and reference checking are useful checks of strategy performance. If those additional methods are finding documents that the searches have already retrieved, but that the team did not necessarily know about in advance, then this is one sign that the strategy might be performing adequately. Also, the PRESS Evidence-Based Checklist (McGowan et al 2016b) should be used to assess whether the search strategy is adequate (see Section 4.4.8). If some of the PRESS dimensions seem to be missing without adequate explanation or arouse concerns, then the search may not yet be complete.

Statistical techniques can be used to assess performance, such as capture-recapture (Spoor et al 1996) (also known as capture-mark-recapture; (Kastner et al 2009), or the relative recall technique (Sampson et al 2006, Sampson and McGowan 2011). Kastner suggests the capture-mark-recapture technique merits further investigation since it could be used to estimate the number of studies in a literature prospectively and to determine where to stop searches once suitable cut-off levels have been identified. Kastner’s approach involves searching databases, conducting record selection, calculating capture-mark-recapture and then making decisions about whether further searches are necessary. This would entail potentially an iterative search and selection process. Capture-recapture needs results from at least two searches to estimate the number of missed studies. Further investigation of published prospective techniques seems warranted to learn more about the potential benefits.

Relative recall (Sampson et al 2006, Sampson and McGowan 2011) requires a range of searches to have been conducted so that the relevant studies have been built up by a set of sensitive searches. The performance of the individual searches can then be assessed in each individual database by determining how many of the studies that were deemed eligible for the evidence synthesis and were indexed within a database, can be found by the database search used to populate the synthesis. If a search in a database did not perform well and missed many studies, then that search strategy is likely to have been suboptimal. If the search strategy found most of the studies that were available to be found in the database then it was likely to have been a sensitive strategy. Assessments of precision could also be made, but these mostly inform future search approaches since they cannot affect the searches and record assessment already undertaken. Relative recall may be most useful at the end of the search process since it relies on the achievement of several searches to make judgements about the overall performance of strategies.

In evidence synthesis involving qualitative data, searching is often more organic and intertwined with the analysis such that the searching stops when new information ceases to be identified (Booth 2016). The reasons for stopping need to be documented and it is suggested that explanations or justifications for stopping may centre around saturation (Booth 2016). Further information on searches for qualitative evidence can be found in Chapter 21.

4.5 Documenting and reporting the search process

Review authors should document the search process in enough detail to ensure that it can be reported correctly in the review (see MECIR Box 4.5.a). The searches of all the databases should be reproducible to the extent that this is possible. By documenting the search process, we refer to internal record-keeping, which is distinct from reporting the search process in the review (discussed in online Chapter III).

MECIR Box 4.5.a Relevant expectations for conduct of intervention reviews

Medical/healthcare librarians and information specialists who have been involved in designing and running the search strategies for a review are increasingly being asked to draft, or at least comment on, the search strategy sections of the review as part of the sign-off process prior to the review being published.

There is currently no clear consensus regarding optimum reporting of systematic review search methods, although suboptimal reporting of commonly recommended items has been observed (Sampson et al 2008, Roundtree et al 2009, Niederstadt and Droste 2010). Research has also shown a lack of compliance with guidance in the Handbook with respect to search strategy description in published Cochrane Reviews (Sampson and McGowan 2006, Yoshii et al 2009, Franco et al 2018). The PRISMA-Search (PRISMA-S) Extension, an extension to the PRISMA Statement, addressing the reporting of search strategies in systematic reviews, should go some way to addressing this, as should the major revision of PRISMA itself, which is due to report in 2019.

It is recommended that review authors seek guidance from their medical/healthcare librarian or information specialist at the earliest opportunity with respect to documenting the search process. For publication in the Cochrane Library, the bibliographic database search strategies should be copied and pasted into an appendix exactly as run and in full, together with the search set numbers and the total number of records retrieved by each search strategy. The search strategies should not be re-typed, because this can introduce errors. Creating a report of the search process can be accomplished through methodical documentation of the steps taken by the searcher. This need not be onerous if suitable record keeping is performed during the process of the search, but it can be nearly impossible to recreate post hoc. Many database interfaces have facilities for search strategies to be saved online or to be emailed; an offline copy in text format should also be saved. For some databases, taking and saving a screenshot of the search may be the most practical approach (Rader et al 2014).

Documenting the searching of sources other than databases, including the search terms used, is also required if searches are to be reproducible (Atkinson et al 2015, Chow 2015, Witkowski and Aldhouse 2015). Details about contacting experts or manufacturers, searching reference lists, scanning websites, and decisions about search iterations can be kept internally for future updates or external requests and can be reproduced as an appendix in the final document. Since the purpose of search documentation is to support transparency, internal assessment, and reference for any future update, it is important to plan how to record searching of sources other than databases since some activities (contacting experts, reference list searching, and forward citation searching) will occur later on in the review process after the database results have been screened (Rader et al 2014). The searcher should record any correspondence on key decisions and report a summary of this correspondence alongside the search strategy. The narrative describes the major decisions that shaped the strategy and can give a peer reviewer an insight into the rationale for the search approach (Craven and Levay 2011).

It is particularly important to save locally or file print copies of any information found on the internet, such as information about ongoing and/or unpublished trials, as this information may no longer be accessible at the time the review is written. Local copies should be stored in a structured way to allow retrieval when needed. There are also web-based tools which archive webpage content for future reference, such as WebCite (Eysenbach and Trudel 2006). The results of web searches will not be reproducible to the same extent as bibliographic database searches because web content and search engine algorithms frequently change, and search results can differ between users due to a general move towards localization and personalization. It is still important, however, to document the search process to ensure that the methods used can be transparently reported (Briscoe 2018). In cases where a search engine retrieves more results than it is practical to screen in full (it is rarely practical to search thousands of web results, as the precision of web searches is likely to be relatively low), the number of results that are documented and reported should be the number that were screened rather than the total number (Dellavalle et al 2003, Bramer 2016).

Decisions should be documented for all records identified by the search. Details of the flow of studies from the number(s) of references identified in the search to the number of studies included in the review will need to be reported in the final review, ideally using a flow diagram such as that proposed by PRISMA (see online Chapter III); these can be generated using software including Covidence, DistillerSR (https://www.evidencepartners.com), EPPI-Reviewer, the METAGEAR package for R, the PRISMA Flow Diagram Generator (http://prisma.thetacollaborative.ca), and RevMan. A table of ‘Characteristics of excluded studies’ will also need to be presented (see Section 4.6.5). Numbers of records are sufficient for exclusions based on initial screening of titles and abstracts. Broad categorizations are sufficient for records classed as potentially eligible during an initial screen. Authors will need to decide for each review when to map records to studies (if multiple records refer to one study). The flow diagram records initially the total number of records retrieved from various sources, then the total number of studies to which these records relate. Review authors need to match the various records to the various studies in order to complete the flow diagram correctly. Lists of included and excluded studies must be based on studies rather than records (see also Section 4.6.1).

4.6 Selecting studies

4.6.1 Studies (not reports) as the unit of interest

A Cochrane Review is a review of studies that meet pre-specified eligibility criteria. Since each study may have been reported in several articles, abstracts or other reports, an extensive search for studies for the review may identify many reports for each potentially relevant study. Two distinct processes are therefore required to determine which studies can be included in the review. One is to link together multiple reports of the same study; and the other is to use the information available in the various reports to determine which studies are eligible for inclusion. Although sometimes there is a single report for each study, it should never be assumed that this is the case.

As well as the studies that inform the systematic review, other studies will also be identified and these should be recorded or tagged as they are encountered, so that they can be listed in the relevant tables in the review:

• records of ongoing trials for which results (either published or unpublished) are not (yet) available; and
• records of studies which seem to be eligible but for which data are incomplete or the publication related to the record could not be obtained.

4.6.2 Identifying multiple reports from the same study

Duplicate publication can introduce substantial biases if studies are inadvertently included more than once in a meta-analysis (Tramèr et al 1997). Duplicate publication can take various forms, ranging from identical manuscripts to reports describing different outcomes of the study or results at different time points (von Elm et al 2004). The number of participants may differ in the different publications. It can be difficult to detect duplicate publication and some ‘detective work’ by the review authors may be required.

Some of the most useful criteria for comparing reports are:

• trial identification numbers (e.g. ClinicalTrials.gov Identifier (NCT number); ISRCTN; Universal Trial Number (UTN) (assigned by the ICTRP); other identifiers such as those from the sponsor);
• author names (most duplicate reports have one or more authors in common, although this is not always the case);
• location and setting (particularly if institutions, such as hospitals, are named);
• specific details of the interventions (e.g. dose, frequency);
• numbers of participants and baseline data; and
• date and duration of the study (which can also clarify whether different sample sizes are due to different periods of recruitment).

Where uncertainties remain after considering these and other factors, it may be necessary to correspond with the authors of the reports.

Multiple reports of the same study should be collated, so that each study, rather than each report, is the unit of interest in the review (see MECIR Box 4.6.a). Review authors will need to choose and justify which report (the primary report) to use as a source for study results, particularly if two reports include conflicting results. They should not discard other (secondary) reports, since they may contain additional outcome measures and valuable information about the design and conduct of the study.

MECIR Box 4.6.a Relevant expectations for conduct of intervention reviews

4.6.3 A typical process for selecting studies

A typical process for selecting studies for inclusion in a review is as follows (the process should be detailed in the protocol for the review):

1. Merge search results from different sources using reference management software, and remove duplicate records of the same report (i.e. records reporting the same journal title, volume and pages).

2. Examine titles and abstracts to remove obviously irrelevant reports (authors should generally be over-inclusive at this stage).

3. Retrieve the full text of the potentially relevant reports.

4. Link together multiple reports of the same study (see Section 4.6.2).

5. Examine full-text reports for compliance of studies with eligibility criteria.

6. Correspond with investigators, where appropriate, to clarify study eligibility (it may be appropriate to request further information, such as missing methods information or results, at the same time). If studies remain incomplete/unobtainable they should be tagged/recorded as incomplete, and should be listed in the table of ‘Studies awaiting assessment’ in the review.

7. Make final decisions on study inclusion and proceed to data collection.

8. Tag or record any ongoing trials which have not yet been reported so that they can be added to the ongoing studies table.

Note that studies should not be omitted from a review solely on the basis of measured outcome data not being reported (see MECIR Box 4.6.b and Chapter 13).

MECIR Box 4.6.b Relevant expectations for conduct of intervention reviews

4.6.4 Implementation of the selection process

Decisions about which studies to include in a review are among the most influential decisions that are made in the review process and they involve judgement.

Use (at least) two people working independently to determine whether each study meets the eligibility criteria.

Ideally, screening of titles and abstracts to remove irrelevant reports should be done in duplicate by two people working independently (although it is acceptable that this initial screening of titles and abstracts is undertaken by only one person). It is essential, however, that two people working independently are used to make a final determination as to whether each study meets the eligibility criteria based on the full text of the study report(s) (see MECIR Box 4.6.c).

MECIR Box 4.6.c Relevant expectations for conduct of intervention reviews

It has been shown that using at least two authors may reduce the possibility that relevant reports will be discarded (Edwards et al 2002) although other case reports have suggested single screening approaches may be adequate (Doust et al 2005, Shemilt et al 2016). Opportunities for screening efficiencies seem likely to become available through promising developments in single human screening in combination with machine learning approaches (O'Mara-Eves et al 2015).

Experts in a particular area frequently have pre-formed opinions that can bias their assessment of both the relevance and validity of articles (Cooper and Ribble 1989, Oxman and Guyatt 1993). Thus, while it is important that at least one author is knowledgeable in the area under review, it may be an advantage to have a second author who is not a content expert.

Disagreements about whether a study should be included can generally be resolved by discussion. Often the cause of disagreement is a simple oversight on the part of one of the review authors. When the disagreement is due to a difference in interpretation, this may require arbitration by another person. Occasionally, it will not be possible to resolve disagreements about whether to include a study without additional information. In these cases, authors may choose to categorize the study in their review as one that is awaiting assessment until the additional information is obtained from the study authors.

A single failed eligibility criterion is sufficient for a study to be excluded from a review. In practice, therefore, eligibility criteria for each study should be assessed in order of importance, so that the first ‘no’ response can be used as the primary reason for exclusion of the study, and the remaining criteria need not be assessed. The eligibility criteria order may be different in different reviews and does not always need to be the same.

For most reviews it will be worthwhile to pilot test the eligibility criteria on a sample of reports (say six to eight articles, including ones that are thought to be definitely eligible, definitely not eligible and doubtful). The pilot test can be used to refine and clarify the eligibility criteria, train the people who will be applying them and ensure that the criteria can be applied consistently by more than one person.

For Cochrane Reviews the selection process must be documented in sufficient detail to be able to complete a flow diagram and a table of ‘Characteristics of excluded studies’ (see MECIR Box 4.6.d). During the selection process it is crucial to keep track of the number of references and subsequently the number of studies so that a flow diagram can be constructed. The decision and reasons for exclusion can be tracked using reference software, a simple document or spreadsheet, or using specialist systematic review software (see Section 4.6.6.1).

MECIR Box 4.6.d Relevant expectations for conduct of intervention reviews

4.6.5 Selecting ‘excluded studies’

A Cochrane Review includes a list of excluded studies called ‘Characteristics of excluded studies’, detailing the specific reason for exclusion for any studies that a reader might plausibly expect to see among the included studies. This covers all studies that may, on the surface, appear to meet the eligibility criteria but which, on further inspection, do not. It also covers those that do not meet all of the criteria but are well known and likely to be thought relevant by some readers. By listing such studies as excluded and giving the primary reason for exclusion, the review authors can show that consideration has been given to these studies. The list of excluded studies should be as brief as possible. It should not list all of the reports that were identified by an extensive search. It should not list studies that obviously do not fulfil the eligibility criteria for the review, such as ‘Types of studies’, ‘Types of participants’, and ‘Types of interventions’. In particular, it should not list studies that are obviously not randomized if the review includes only randomized trials. Based on a (recent) sample of approximately 60% of the intervention reviews in The Cochrane Library which included randomized trials (only), the average number of studies listed in the ‘excluded studies’ table is 30.

4.6.6 Software support for selecting studies

An extensive search for eligible studies in a systematic review can often identify thousands of records that need to be manually screened. Selecting studies from within these records can be a particularly time-consuming, laborious and logistically challenging aspect of conducting a systematic review. These and other challenges have led to the development of various software tools and packages that offer support for the selection process.

Broadly, software to support selecting studies can be classified as:

• systems that support the study selection process, typically involving multiple reviewers (see Section 4.6.6.1); and
• tools and techniques based on text mining and/or machine learning, which aim to semi- or fully-automate the selection process (see Section 4.6.6.2).

Software to support the selection process, along with other stages of a systematic review, including text mining tools, can be identified using the Systematic Review Toolbox. The SR Toolbox is a community driven, web-based catalogue of tools that provide support for systematic reviews (Marshall and Brereton 2015).

4.6.6.1 Software for managing the selection process

Managing the selection process can be challenging, particularly in a large-scale systematic review that involves multiple reviewers. Basic productivity tools can help (such as word processors, spreadsheets and reference management software), and several purpose-built systems are also available that offer support for the study selection process.

Examples of tools that support selecting studies include:

• Abstrackr – a free web-based screening tool that can prioritize the screening of records using machine learning techniques.
• Covidence – a web-based software platform for conducting systematic reviews, which includes support for collaborative title and abstract screening, full-text review, risk-of-bias assessment and data extraction. Full access to this system normally requires a paid subscription but is free for authors of Cochrane Reviews. A free trial for non-Cochrane Review authors is also available.
• DistillerSR – a web-based software application for undertaking bibliographic record screening and data extraction. It has a number of management features to track progress, assess interrater reliability and export data for further analysis. Reduced pricing for Cochrane and Campbell reviews is available.
• EPPI-Reviewer – web-based software designed to support all stages of the systematic review process, including reference management, screening, risk of bias assessment, data extraction and synthesis. The system is free to use for Cochrane and Campbell reviews, otherwise it requires a paid subscription. A free trial is available.
• Rayyan – a web-based application for collaborative citation screening and full-text selection. The system is currently available free of charge (June 2018).

Compatibility with other software tools used in the review process (such as RevMan) may be a consideration when selecting a tool to support study selection. Covidence and EPPI-Reviewer are Cochrane-preferred tools, and are likely to have the strongest integration with RevMan.

4.6.6.2 Automating the selection process

Research into automating the study selection process through machine learning and text mining has received considerable attention over recent years, resulting in the development of various tools and techniques for reviewers to consider. The use of automated tools has the potential to reduce the workload involved with selecting studies significantly (Thomas et al 2017). For example, research suggests that adopting automation can reduce the need for manual screening by at least 30% and possibly more than 90%, although sometimes at the cost of up to a 5% reduction in sensitivity (O'Mara-Eves et al 2015).

Machine learning models (or ‘classifiers’) can be built where sufficient data are available. Of particular practical use to Cochrane Review authors is a classifier (the ‘RCT Classifier’) that can identify reports of randomized trials based on titles and abstracts. The classifier is highly accurate because it is built on a large dataset of hundreds of thousands of records screened by Cochrane Crowd, Cochrane’s citizen science platform, where contributors help to identify and describe health research (Marshall et al 2018). Guidance on using the RCT Classifier in Cochrane Reviews, for example to exclude studies already flagged as not being randomized trials, or to access Cochrane Crowd to assist with screening, is available from Cochrane Information Specialists handbook (https://training.cochrane.org/resource/cochrane-information-specialists-handbook).

In addition to learning from large datasets such as those generated by Cochrane Crowd, it is also possible for machine learning models to learn how to apply eligibility criteria for individual reviews. This approach uses a process called ‘active learning’ and it is able to semi-automate study selection by continuously promoting records most likely to be relevant to the top of the results list (O'Mara-Eves et al 2015). It is difficult for authors to determine in advance when it is safe to stop screening and allow some records to be eliminated automatically without manual assessment. The automatic elimination of records using this approach has not been recommended for use in Cochrane Reviews at the time of writing. This active learning process can still be useful, however, since by prioritizing records for screening in order of relevance, it enables authors to identify the studies that are most likely to be included much earlier in the screening process than would otherwise be possible. A number of software tools support ‘active learning’ including:

• Abstrackr (http://abstrackr.cebm.brown.edu/);
• Colandr (https://www.colandrapp.com/);
• EPPI-Reviewer (http://eppi.ioe.ac.uk/);
• Rayyan (http://rayyan.qcri.org/);
• RobotAnalyst (http://nactem.ac.uk/robotanalyst/); and
• Swift-review (http://swift.sciome.com/swift-review/).

Finally, tools are available that use natural language processing to highlight sentences and key phrases automatically (e.g. PICO elements, trial characteristics, details of randomization) to support the reviewer whilst screening (Tsafnat et al 2014).

4.7 Chapter information

Authors: Carol Lefebvre, Julie Glanville, Simon Briscoe, Anne Littlewood, Chris Marshall, Maria-Inti Metzendorf, Anna Noel-Storr, Tamara Rader, Farhad Shokraneh, James Thomas, L. Susan Wieland; on behalf of the Cochrane Information Retrieval Methods Group

Acknowledgements: This chapter has been developed from sections of previous editions of the Cochrane Handbook co-authored since 1995 by Kay Dickersin, Julie Glanville, Kristen Larson, Carol Lefebvre and Eric Manheimer. Many of the sources listed in this chapter and the accompanying technical supplement have been brought to our attention by a variety of people over the years and we should like to acknowledge this. We should like to acknowledge: Ruth Foxlee, (formerly) Information Specialist, Cochrane Editorial Unit; Miranda Cumpston, (formerly) Head of Learning & Support, Cochrane Central Executive; Colleen Finley, Product Manager, John Wiley and Sons, for checking sections relating to searching the Cochrane Library; the (UK) National Institute for Health and Care Excellence and the German Institute for Quality and Efficiency in Health Care (IQWiG) for support in identifying some of the references; the (US) Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program Scientific Resource Center Article Alert service; Tianjing Li, Co-Convenor, Comparing Multiple Interventions Methods Group, for text and references that formed the basis of the re-drafting of parts of Section 4.6 Selecting studies; Lesley Gillespie, Cochrane author and former Editor and Trials Search Co-ordinator of the Cochrane Bone, Joint and Muscle Trauma Group, for copy-editing an early draft; The Cochrane Information Specialist Executive, the Cochrane Information Specialists’ Support Team, Cochrane Information Specialists and members of the Cochrane Information Retrieval Methods Group for comments on drafts.

Funding: FS is a full-time research fellow at University of Nottingham and an information specialist at Cochrane Schizophrenia Group which is supported by Cochrane infrastructure funding from National Institute of Health Research (NIHR). JT is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

4.8 References

Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews: AHRQ publication no. 10(14)-EHC063-EF. 2014 https://effectivehealthcare.ahrq.gov/topics/cer-methods-guide/overview

Arber M, Cikalo M, Glanville J, Lefebvre C, Varley D, Wood H. Annotated bibliography of published studies addressing searching for unpublished studies and obtaining access to unpublished data. 2013. https://methods.cochrane.org/sites/methods.cochrane.org.irmg/files/public/uploads/Annotatedbibliographtifyingunpublishedstudies.pdf.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Research Synthesis Methods 2015;6: 87-95.

Bai Y, Gao J, Zou D, Li Z. Is MEDLINE alone enough for a meta-analysis? Alimentary Pharmacology and Therapeutics 2007;26: 125-126; author reply 126.

Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ 2017;356: j448.

Bennett DA, Jull A. FDA: untapped source of unpublished trials. Lancet 2003;361: 1402-1403.

Bero L. Searching for unpublished trials using trials registers and trials web sites and obtaining unpublished trial data and corresponding trial protocols from regulatory agencies. 2013. http://web.archive.org/web/20150108071243/http://methods.cochrane.org:80/projects-developments/searching-unpublished-trials-using-trials-registers-and-trials-web-sites-and-o

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. International Journal of Technology Assessment in Health Care 2010;26: 431-435.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews 2016;5: 74.

Bramer WM. Variation in number of hits for complex searches in Google Scholar. Journal of the Medical Library Association 2016;104: 143-145.

Briscoe S. A review of the reporting of web searching to identify studies for Cochrane systematic reviews. Research Synthesis Methods 2018;9: 89-99.

Brusselle GG, Blasi F. Risk of a biased assessment of the evidence when limiting literature searches to the English language: macrolides in asthma as an illustrative example. Pulmonary Pharmacology and Therapeutics 2015;31: 109-110.

Chan AW. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ 2012;344: d8013.

Chapman SJ, Shelton B, Mahmood H, Fitzgerald JE, Harrison EM, Bhangu A. Discontinuation and non-publication of surgical randomised controlled trials: observational study. BMJ 2014;349: g6870.

Chilcott J, Brennan A, Booth A, Karnon J, Tappenden P. The role of modelling in prioritising and planning clinical trials. Health Technology Assessment 2003;7: iii, 1-125.

Chow TK. Electronic search strategies should be repeatable. European Journal of Pain 2015;19: 1562-1563.

Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993;269: 2749-2753.

Cooper H, Ribble RG. Influences on the outcome of literature searches for integrative research reviews. Knowledge: Creation, Diffusion, Utilization 1989;10: 179-201.

Craven J, Levay P. Recording database searches for systematic reviews - What is the value of adding a narrative to peer-review checklists? A case study of NICE interventional procedures guidance. Evidence Based Library and Information Practice 2011;6: 72-87.

de Vet H, Eisinga A, Riphagen I, Aertgeerts B, Pewsner D. Chapter 7: Searching for studies. In: Deeks J, Bossuyt P, Gatsonis C, editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 04 (updated September 2008): The Cochrane Collaboration; 2008. https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/Chapter07-Searching-%28September-2008%29.pdf.

Dellavalle RP, Hester EJ, Heilig LF, Drake AL, Kuntzman JW, Graber M, Schilling LM. Information science. Going, going, gone: lost Internet references. Science 2003;302: 787-788.

Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ 2013;346: f2865.

Doust JA, Pietrzak E, Sanders S, Glasziou PP. Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. Journal of Clinical Epidemiology 2005;58: 444-449.

Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337: 867-872.

Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Statistics in Medicine 2002;21: 1635-1640.

Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997;350: 326-329.

Egger M, Smith GD. Bias in location and selection of studies. BMJ 1998;316: 61-66.

Elsevier. Embase content 2016a. https://www.elsevier.com/solutions/embase-biomedical-research/embase-coverage-and-content.

Elsevier. Embase classic fact sheet 2016b. https://www.elsevier.com/__data/assets/pdf_file/0005/58982/R_D-Solutions_Embase_Fact-Sheet_Classic-DIGITAL.pdf.

EUnetHTA. Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness (Version 1.2). Germany: European network for Health Technology Assessment; 2017 https://www.eunethta.eu/wp-content/uploads/2018/01/Guideline_Information_Retrieval_V1-2_2017.pdf

Eysenbach G, Trudel M. Going, going, still there: using the WebCite service to permanently archive cited web pages. Journal of Medical Internet Research 2006;7: e60.

Franco JVA, Garrote VL, Escobar Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Research Synthesis Methods 2018;9: 408-416.

Glanville J, Lefebvre C, Wright Ke. ISSG search filter resource York (UK): The InterTASC Information Specialists' Sub-Group 2008. https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home.

Glanville JM, Duffy S, McCool R, Varley D. Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? Journal of the Medical Library Association 2014;102: 177-183.

Goldacre B, Lane S, Mahtani KR, Heneghan C, Onakpoya I, Bushfield I, Smeeth L. Pharmaceutical companies' policies on access to trial data, results, and methods: audit study. BMJ 2017;358: j3334.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005;331: 1064-1065.

Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PloS One 2015;10: e0138237.

Halfpenny NJ, Quigley JM, Thompson JC, Scott DA. Value and usability of unpublished data sources for systematic reviews and network meta-analyses. Evidence-Based Medicine 2016;21: 208-213.

Halladay CW, Trikalinos TA, Schmid IT, Schmid CH, Dahabreh IJ. Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions. Journal of Clinical Epidemiology 2015;68: 1076-1084.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Medical Research Methodology 2016;16: 127.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Medical Research Methodology 2017;17: 64.

Hetherington J, Dickersin K, Chalmers I, Meinert CL. Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 1989;84: 374-380.

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. Pharmacoeconomics 2015;33: 5-11.

Horton R. Medical editors trial amnesty. Lancet 1997;350: 756.

Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annual Symposium Proceedings/AMIA Symposium 2006: 359-363.

Hwang TJ, Carpenter D, Lauffenburger JC, Wang B, Franklin JM, Kesselheim AS. Failure of investigational drugs in late-stage clinical development and publication of trial results. JAMA Internal Medicine 2016;176: 1826-1833.

Iansavichene AE, Sampson M, McGowan J, Ajiferuke ISY. Should systematic reviewers search for randomized, controlled trials published as letters? Annals of Internal Medicine 2008;148: 714-715.

Institute of Medicine. Finding what works in health care: Standards for systematic reviews. Washington, DC: National Academies Press; 2011 http://books.nap.edu/openbook.php?record_id=13059

Irvin E, Hayden J. Developing and testing an optimal search strategy for identifying studies of prognosis [Poster]. 14th Cochrane Colloquium; 2006 October 23-26; Dublin, Ireland 2006.

Isojarvi J, Wood H, Lefebvre C, Glanville J. Challenges of identifying unpublished data from clinical trials: getting the best out of clinical trials registers and other novel sources. Research Synthesis Methods 2018: 561-578.

Jefferson T, Doshi P, Boutron I, Golder S, Heneghan C, Hodkinson A, Jones M, Lefebvre C, Stewart LA. When to include clinical study reports and regulatory documents in systematic reviews. BMJ Evidence-Based Medicine 2018;23: 210-217.

Jiao S, Tsutani K, Haga N. Review of Cochrane reviews on acupuncture: how Chinese resources contribute to Cochrane reviews. Journal of Alternative and Complementary Medicine 2013;19: 613-621.

Kaptchuk T. Certain countries produce only positive trial results. Focus on Alternative and Complementary Therapies 1999;4: 86-87.

Kastner M, Straus SE, McKibbon KA, Goldsmith CH. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. Journal of Clinical Epidemiology 2009;62: 149-157.

Khan KS, ter Riet G, Glanville J, Sowden AJ, Kleijnen, J. (editors). Undertaking systematic reviews of research on effectiveness: CRD's guidance for those carrying out or commissioning reviews (CRD Report Number 4 2nd edition). York (UK): NHS Centre for Reviews and Dissemination, University of York. 2001.

Knelangen M, Hausner E, Metzendorf MI, Sturtz S, Waffenschmidt S. Trial registry searches for randomized controlled trials of new drugs required registry-specific adaptation to achieve adequate sensitivity. Journal of Clinical Epidemiology 2018;94: 69-75.

Kohler M, Haag S, Biester K, Brockhaus AC, McGauran N, Grouven U, Kolsch H, Seay U, Horn H, Moritz G, Staeck K, Wieseler B. Information on new drugs at market entry: retrospective analysis of health technology assessment reports versus regulatory reports, journal publications, and registry reports. BMJ 2015;350: h796.

Kreis J, Panteli D, Busse R. How health technology assessment agencies address the issue of unpublished data. International Journal of Technology Assessment in Health Care 2014;30: 34-43.

Lampert A, Hoffmann GF, Ries M. Ten years after the International Committee of Medical Journal Editors' clinical trial registration initiative, one quarter of phase 3 pediatric epilepsy clinical trials still remain unpublished: a cross sectional analysis. PloS One 2016;11: e0144973.

Layton D. A critical review of search strategies used in recent systematic reviews published in selected prosthodontic and implant-related journals: Are systematic reviews actually systematic? International Journal of Prosthodontics 2017;30: 13-21.

Lee K, Bacchetti P, Sim I. Publication of clinical trials supporting successful new drug applications: a literature analysis. PLoS Medicine 2008;5: e191.

Lefebvre C, Eisinga A, McDonald S, Paul N. Enhancing access to reports of randomized trials published world-wide--the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library. Emerging Themes in Epidemiology 2008;5: 13.

Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future? Systematic Reviews 2013;2: 78.

Littlewood A, Bridges C, for the Cochrane Information Specialist Support Team. Cochrane Information Specialists' Handbook Oslo: The Cochrane Collaboration; 2017. http://training.cochrane.org/resource/cochrane-information-specialists-handbook.

Lorenc T, Tyner EF, Petticrew M, Duffy S, Martineau FP, Phillips G, Lock K. Cultures of evidence across policy sectors: systematic review of qualitative evidence. European Journal of Public Health 2014;24: 1041-1047.

Lorenzetti DL, Topfer LA, Dennett L, Clement F. Value of databases other than MEDLINE for rapid health technology assessments. International Journal of Technology Assessment in Health Care 2014;30: 173-178.

MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG, Southern California Evidence-Based Practice Center. How useful are unpublished data from the Food and Drug Administration in meta-analysis? Journal of Clinical Epidemiology 2003;56: 44-51.

Manheimer E, Anderson D. Survey of public information about ongoing clinical trials funded by industry: evaluation of completeness and accessibility. BMJ 2002;325: 528-531.

Marshall C, Brereton P. Systematic review toolbox: a catalogue of tools to support systematic reviews. Proceedings of the 2015 International Conference on Evaluation and Assessment in Software Engineering (EASE) 2015: Article no. 23.

Marshall I, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying randomized controlled trials: an evaluation and practitioner's guide. Research Synthesis Methods 2018;9: 602-614.

McGowan J, Sampson M, Salzwedel D, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline explanation and elaboration Ottawa: CADTH; 2016a https://www.cadth.ca/sites/default/files/pdf/CP0015_PRESS_Update_Report_2016.pdf

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. Journal of Clinical Epidemiology 2016b;75: 40-46.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association 2016;104: 267-277.

Metzendorf MI. Why medical information specialists should routinely form part of teams producing high quality systematic reviews – a Cochrane perspective. Journal of the European Association for Health Information and Libraries 2016;12: 6-9.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technology Assessment 2003;7: 1-90.

Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, Mierzwinski-Urban M, Clifford T, Hutton B, Rabb D. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. International Journal of Technology Assessment in Health Care 2012;28: 138-144.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods 2014;5: 116-130.

Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. International Journal of Technology Assessment in Health Care 2010;26: 450-457.

O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews 2015;4: 5.

Oxman AD, Guyatt GH. The science of reviewing research. Annals of the New York Academy of Sciences 1993;703: 125-133; discussion 133-134.

Pansieri C, Pandolfini C, Bonati M. Clinical trial registries: more international, converging efforts are needed. Trials 2017;18: 86.

Petticrew M, Roberts H, editors. Systematic Reviews in the Social Sciences. Oxford (UK): Blackwell; 2006.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. Journal of Clinical Epidemiology 2005;58: 769-776.

Pilkington K, Boshnakova A, Clarke M, Richardson J. "No language restrictions" in database searches: what does this really mean? Journal of Alternative and Complementary Medicine 2005;11: 205-207.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Research Synthesis Methods 2014;5: 98-115.

Relevo R, Paynter R. Peer Review of Search Strategies. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012 https://www.ncbi.nlm.nih.gov/books/NBK98353/

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of Clinical Epidemiology 2015;68: 617-626.

Reveiz L, Cardona AF, Ospina EG, de Agular S. An e-mail survey identified unpublished studies for systematic reviews. Journal of Clinical Epidemiology 2006;59: 755-758.

Rice DB, Kloda LA, Levis B, Thombs BD. Are MEDLINE searches sufficient for systematic reviews and meta-analyses of the diagnostic accuracy of depression screening tools? A review of meta-analyses. Journal of Psychosomatic Research 2016;87: 7-13.

Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well built clinical question: a key to evidence based decisions. ACP Journal Club 1995;123: A12-13.

Roundtree AK, Kallen MA, Lopez-Olivo MA, Kimmel B, Skidmore B, Ortiz Z, Cox V, Suarez-Almazor ME. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review. Journal of Clinical Epidemiology 2009;62: 128-137.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. International Journal of Technology Assessment in Health Care 2003;19: 591-603.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P. Should meta-analysts search Embase in addition to Medline? Journal of Clinical Epidemiology 2003;56: 943-955.

Sampson M, Zhang L, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, Klassen TP, Moher D. An alternative to the hand searching gold standard: Validating methodological search filters using relative recall. BMC Medical Research Methodology 2006;6: 33.

Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology 2006;59: 1057-1063.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. Journal of Clinical Epidemiology 2008;61: 748-754.

Sampson M, McGowan J. Inquisitio validus Index Medicus: A simple method of validating MEDLINE systematic review searches. Research Synthesis Methods 2011;2: 103-109.

Sampson M, de Bruijn B, Urquhart C, Shojania K. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. Journal of Clinical Epidemiology 2016;78: 108-115.

Scherer RW, Ugarte-Gil C, Schmucker C, Meerpohl JJ. Author’s reasons for unpublished research presented at biomedical conferences: a systematic review. Journal of Clinical Epidemiology 2015;68: 803-810.

Schroll JB, Bero L, Gøtzsche PC. Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ 2013;346: f2231.

Shemilt I, Simon A, Hollands GJ, Marteau TM, Ogilvie D, O'Mara-Eves A, Kelly MP, Thomas J. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Research Synthesis Methods 2014;5: 31-49.

Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews 2016;5: 140.

Spoor P, Airey M, Bennett C, Greensill J, Williams R. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ 1996;313: 342-343.

Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? 21st Cochrane Colloquium; 2013; Quebec City, Canada. https://abstracts.cochrane.org/2013-qu%C3%A9bec-city/peer-review-literature-search-strategies-does-it-make-difference.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine 2004;12: 228-232.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Controlled Clinical Trials 2000;21: 476-487.

Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, Glasziou P, Shemilt I, Synnot A, Turner T, Elliott J, Living Systematic Review N. Living systematic reviews: 2. Combining human and machine effort. Journal of Clinical Epidemiology 2017;91: 31-37.

Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997;315: 635-640.

Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Systematic Reviews 2014;3: 74.

US National Library of Medicine. PubMed. 2018. https://www.nlm.nih.gov/bsd/pubmed.html.

US National Library of Medicine. MEDLINE®: Description of the Database. 2019. https://www.nlm.nih.gov/bsd/medline.html.

US National Library of Medicine. MEDLINE, PubMed, and PMC (PubMed Central): How are they different? no date. https://www.nlm.nih.gov/bsd/difference.html.

Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Controlled Clinical Trials 1998;19: 159-166.

Viergever RF, Li K. Trends in global clinical trial registration: an analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013. BMJ Open 2015;5: e008932.

von Elm E, Poglia G, Walder B, Tramèr MR. Different patterns of duplicate publication: an analysis of articles used in systematic reviews. JAMA 2004;291: 974-980.

Wallace S, Daly C, Campbell M, Cody J, Grant A, Vale L, Donaldson C, Khan I, Lawrence P, MacLeod A. After MEDLINE? Dividend from other potential sources of randomised controlled trials. Second International Conference Scientific Basis of Health Services & Fifth Annual Cochrane Colloquium; 1997; Amsterdam, The Netherlands.

Wang Z, Brito JP, Tsapas A, Griebeler ML, Alahdab F, Murad MH. Systematic reviews with language restrictions and no author contact have lower overall credibility: a methodology study. Clinical Epidemiology 2015;7: 243-247.

Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA 1998;280: 257-259.

Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T. Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications. BMJ 2012;344: d8141.

Witkowski MA, Aldhouse N. Transparency and reproducibility of supplementary search methods in NICE single technology appraisal manufacturer submissions. Value in Health 2015;18: A721-722.

Wood H, Arber M. Search strategy development [webpage]. Summarized Research in Information Retrieval for HTA (SuRe Info) 2019. http://www.htai.org/vortal/?q=node/790.

Woods KD, Trewheellar K. Medline and Embase complement each other in literature searches. BMJ 1998;316: 1166.

Wu XY, Tang JL, Mao C, Yuan JQ, Qin Y, Chung VC. Systematic reviews and meta-analyses of traditional Chinese medicine must search Chinese databases to reduce language bias. Evidence-Based Complementary and Alternative Medicine 2013: Article ID 812179.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association 2009;97: 21-29.

Young T, Hopewell S. Methods for obtaining unpublished data. Cochrane Database of Systematic Reviews 2011;11: MR000027.