Skip to Main Content

Systematic and Other Reviews: Testing and Evaluating Search Design

This guide contains information about systematic reviews and links to resources to help you conduct one.

Test the Search

 

An important and often overlooked part of search design is testing and peer review. This is sometimes called evaluation.

Test 1: Check Results

Look over the first few pages of the results. Do they match your search terms? Do you see papers that you would include in your review?

Some of the results will be irrelevant, and that's normal. Investigate these papers further, looking at the title and abstract, and try to figure out why they came up in your search.

Three common problems might come up in the test. Troubleshoot them below:

Test 2: Pearl Article Test

Often a reviewer already knows a number of articles that perfectly match their selection criteria. We can use these to test the search. If these articles come up in your search, it's a good sign you're on the right track.

Test 3: Peer Review

Ask a team member or colleague to check the search and offer feedback. A set of fresh eyes often reveals small errors or logic problems.

Final Step: PRESS Checklist

 

The PRESS Checklist provides a shortlist of search strategy criteria to assess that your search is complete.

It is a good idea to assess your search before you move on to translating your search as it can help you pick up critical errors. The table below is an abbreviated version of the checklist however the full version is linked below.

Criteria

Guidance

The research question has been correctly translated into search concepts.

  • Does the search strategy contain concepts which reflect the research question?

Correct Boolean and/or proximity operators.

  • Are all synonyms combined with OR, with no errors?

  • Are concept groups combined with AND?

  • Has use of NOT been justified?

  • Are there any proximity operators that can be used to increase recall and precision?

Correct Subject Heading Use

  • Are the included subject headings relevant to the topic? (check the scope notes)

  • Have any narrower terms been included? (use of 'explosion')

  • Are there any missing subject headings (check indexing of key papers)

Are keywords adequate?

  • Has each concept been accurately described using keywords?

  • Have all applicable alternative terms been included?

  • Are the keywords used on topic? (not too narrow, not too broad but are relevant)

Correct use of spelling, correct use of syntax and correct search implementation.

Review the search strategy for misspelled words and for errors in system syntax that are not easily found by spell checking.

Check each line number and combinations of line numbers to ensure that the search logic was correctly implemented.

Limits and filters: Assess whether the limits used (including filters) are appropriate and have been applied correctly.

Review the search strategy to see if limits that are not relevant to the eligible study designs or to the clinical question have been applied, as these could potentially introduce bias.

Chapter Summary

 

 

✅ Learnings

  • Database searches must be testing to ensure they return comprehensive and accurate results.
  • Reviewers can conduct testing including result review and pearl article searching.
  • All systematic review manuals recommend peer review of search strategies.
  • The PRESS checklist is a comprehensive tool to evaluate search strategies.

 

⚠️ I'm stuck!

My results are way, WAY off

Many databases will highlight your search terms in the article preview, so click to view articles that are way off, and figure out why they appeared in the search. This might lead to adjustments like removing particular keywords, changing keywords, or adjusting the structure of the search altogether by adding/subtracting a concept.

My pearl articles didn't appear in my search

If you are certain that the article is in the database (always do the first step!), then it's time to go through the article and identify why it didn't show up. It could be that a subject heading or keywords is missing, or that one of your concepts isn't present in the search. This might reveal whether you need a particular concept, or whether it should be changed. 

No-one is around to peer review my search

At the very least, a supervisor should give the search a lookover!

 

🛑 Stop and Review

  
  
  
  
Click the next button to continue.  ►