Expert review (method)

Expert review or expert evaluation is a method to evaluate survey questions from the perspective of one or more experts.

An expert review has two primary goals:

1. Identify potential problems related to data quality and data collection so they can be mitigated.
2. Group survey items by how likely they are to result in measurement errors, such as to classify interviewer effects.

An individual expert or a panel of experts such as survey methodologists, subject matter experts (such as sociologists) or other people familiar with questionnaire design review the survey, and they may be asked to suggest revisions. Because variation is found in the problems identified by each expert, a panel rather than one single expert is recommended.

The experts document their assessment and typically report the results in open-ended comments. They can also conduct a systematic review using a coding tool such as the Questionnaire Appraisal System that is supported by an item taxonomy of the cognitive demands of a question and a detailed set of potential problem codes document the features that may lead to response error.

When applied to translated surveys, the expert panel also includes translators and they usually produce individual reports and then convene a resolution meeting that generates an improved version of the survey translation.

Compared to empirical pretesting methods, an expert review is more time-efficient and less costly to complete. An expert review can also be combined with other methods to enable multimode pretesting.