This month in the IRB Corner, we want to discuss the timing of the IRB review in relation to field testing an instrument and running a pilot study. The federal human subjects regulations (45 CFR 46) and institutional policy require that an IRB review be conducted before any research activities involving human subjects begin, including pilot studies. We often get questions about field tests, pilot studies, and IRB reviews. Within the literature these terms are used interchangeably at times. What is important from an IRB review perspective is the research activities and whether or not these activities involve the collection of data from human subjects.
Timing of the IRB Review and Pre-testing Data Collection Instruments
Field tests and pilot studies are used as strategies to check the soundness or “validity” of a data collection instrument. In other words, does the data collection tool gather the information it was designed to gather? When the testing of the instrument involves the collection of data, an IRB review must take place prior to the test.
With quantitative studies, for example, to test a survey instrument, researchers collect data to statistically analyze the validity and reliability of the survey items. This process of collecting the data using a subset of the sample population and testing the survey instrument is typically called a pilot test. Since data are collected during the pilot test of the instrument, an IRB approved or exempt decision must be in place prior to the researcher conducting the pilot study.
In contrast, in a qualitative study where a researcher intends to collect data using a questionnaire with open-ended items, the researcher cannot statistically test the validity and reliability of questions. In these cases, the researcher will typically “field test” the instrument, which involves taking the instrument into the field to identify problems that could be experienced by respondents during the actual study. To field test an instrument, the researcher may strategically select a very small number of individuals, say 3-5 people, who have expert knowledge about the population and research topic to provide feedback on the appropriateness of the questions being asked and how the questions are being asked in relation to the study focus and with the proposed sample. These experts are not answering the questions on the questionnaire, so they are not providing data.
The experts used in the field testing of the instrument provide information to help the researcher refine and improve the questionnaire or interview questions. The field test may lead the researcher to revisions to avoid ambiguity and bias, to reduce repetition of the questions, or to adjust word choice to establish authenticity. Since data are not collected during the field testing of the instrument, an IRB approval or exemption decision is not required prior to the researcher conducting the field test.
Is Data Needed for the Pre-test?
It is not uncommon for a novice researcher to confuse the terminology and the practices involved in field tests and pilot tests (or pilot studies). It is also not uncommon for the use of these terms to vary by discipline and method. What is critical for IRB purposes is that the researcher distinguish whether data are needed to conduct the pre-testing activities. If data are needed from human subjects to test the instrument, then the IRB review must take place prior to the engagement of the human subjects in the research activities. If the researcher’s aim is to ensure questionnaire or interview questions are written appropriately for qualitative data collection based on experts’ review of the questions, then no IRB review takes place for the field testing of the questions.
The easiest way for novice researchers to remember the difference between field testing and pilot testing is the field test involves a strategy of selecting a few “experts”, not human subjects, who provide feedback on the quality of the questionnaire or interview questions in connection with the topic. In these cases, the experts do not provide data. In contrast, in a pilot study (or pilot test) data is collected from human subjects allowing the researcher to conduct statistical tests and determine the validity of the instrument.
As a reminder, all IRB guidance materials, including guidance materials regarding field tests and pilot tests, are available in the IRBNet Forms and Templates library, from the left menu once logged into IRBNet (http://www.irbnet.org). After reviewing guidance materials and discussing the project with your Chair (students), researchers are encouraged to contact the IRB Office at IRB@phoenix.edu, or through IRBNet, with additional questions.