Skip to Main Content

Systematic Reviews: Grey Literature

This Guide will help to start and proceed through the main stages of systematic review

About

 

The Fourth International Conference on Grey Literature in 1999 arrived at the following definition:

"That which is produced on all levels of government, academics, business,

and industry in print and electronic formats, but which is not controlled by

commercial publishers". Examples of grey literature include conference abstracts,

presentations, proceedings; regulatory data; unpublished trial data; government publications; reports

(such as white papers, working papers, internal documentation); dissertations/theses; patents; and policies & procedures.

Inclusion of grey literature into a systematic review is recommended in order to help minimize publication bias. The inclusion of grey literature in systematic reviews is widely recognized as important and international organizations have incorporated this information in their guidelines and manuals for working on reviews and meta-analyses.

  • AHRQ: Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions
    This is a chapter from AHRQ's "Methods Guide for Effectiveness and Comparative Effectiveness Reviews."
  • Cochrane Handbook: Unpublished and Onging Studies
    From the Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]
  • Cochrane Handbook: Including unpublished studies in systematic reviews
    From the Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]
  • The Joanna Briggs Institute Reviewer's Manual
    Aromataris E, Munn Z (Editors). Joanna Briggs Institute Reviewer's Manual. The Joanna Briggs Institute, 2017. Available from https://reviewersmanual.joannabriggs.org/
  • PRISMA for systematic review protocols (PRISMA-P)
    Item 9: "Describe all intended information sources (such as electronic databases, contact with study authors, trial registers or other grey literature sources) with planned dates of coverage."

Searching

Searching the grey literature can be a daunting task. You should search those resources that make the most sense for your research question. At a minimum, consider searching Abstracts and Conferences. If your question involves drugs and interventions, check trial registries and pharma data. Also, check out the papers and reports of relevant stakeholder organizations.

The links below may shed some additional light on the process.

 

Evaluating

Use the same criteria to evaluate the literature as those used to evaluate any kind of information.

Consider:

Source of the report  Locate the source of the report. Check if this is a reliable author, organization or source.
Transparency of methods  It should be clear where data and other types of information came from, how it is analyzed and how the final report was  compiled.
Currentness:   Find the data a report was issued. If the report (or other material) is older, try to find a more recent version or an update

Tip: The AACODS Checklist (pdf), designed for evaluation and critical appraisal of Grey Literature.

Documenting

Most Grey Literature is found using hand-searching or in databases that do not offer systematic search. However, for systematic reviews, reproducibility of the search is key. Make sure you document your searches well.

  • If possible, document used search terms
  • Make an overview of all resources used
  • Document search results for each resource
  • Record the source of the reference in EndNote or another reference manager
  • Document contact details or organisations that provided the reference.

Sources

Bibliographic Databases with Conference Indexing and Abstracting:

Networked Digital Library of Theses and Dissertations  
ProQuest Dissertations & Theses

 

The approval process for new drugs and devices involves submission of data that might not be published elsewhere. These documents can help identify publication bias even when complete methodological details of unpublished trials are unavailable. This information is not available prior to a drug’s approval and may be heavily redacted; however, when available, reviewers can compare results of published and unpublished trials.
 
 
Collections of newspapers, journals, dissertations, archives, government publications, and other traditional and digital resources​.
 

Official Information Sources of the Prime Minister of the Republic of Kazakhstan (in Russian and Kazakh)

Open Government of the Republic of Kazakhstan (in Russian and Kazakh)

HSRProj : Health Services Research Projects in Progress
NIH Reporter
 

 

Online trial registries may include results of completed but unpublished clinical trials. These resources can be helpful in identifying otherwise unreachable trials and provide additional details of published trials. Many individual drug companies also have posted trial registries on their websites, although their quality varies. Even without results, knowledge that the trial exists can be helpful for reviewers, because they can contact the principal investigator for more information. 
 
ClinicalStudyDataRequest.com (CSDR) - Access to clinical trial data, including patient data from 5 pharmaceutical companies for pre-approved researchers. 
Open Trials (Beta)
 
 

Agency for Healthcare Research & Quality (AHRQ)  
Health Technology Assessment (HTA) Database Overview

Centre for Reviews and Dissemination.
National Technical Information Service (NTIS)

NYAM: Grey Lit Report
Home, dedicated search   
Catalog Search: Reports collected from a long list of organizations
Note: no longer being added to as of January 2017

Repositories are collections of research produced by an institution's researchers. Institutions can be governmental, organizational, and academic, such as Universities. Often contain electronic theses.

Nazarbayev University Repository

Nazarbayev University Repository is an institutional electronic archive for long term storage, accumulation and provision of long-term and reliable open access to scientific research results and intellectual products of the academic community of Nazarbayev University associated with them.

OAIster               
Records for items in digital libraries, institutional repositories.
 
European grey lit (reports, dissertations, conference papers, etc) large archival section, 2004 or earlier, started adding new material in 2014 from partner institutions. 
Library Homepage Facebook Youtube Instagram Twitter Telegram E-mail