Delphi Method

Delphi Method

Leader: Dr. Phil Davidson

Introduction -  The Delphi technique is a research design, usually considered a qualitative method, which was designed to forecast viable solutions to problems where data was missing or incomplete.  The object “is to obtain the most reliable consensus of opinion of a group of experts” (Dalkey & Helmer, 1963, p. 458) as to the best workable solutions to the problem.  The Delphi technique originated in the 1950’s as a research project funded by the United States Air Force and developed by the RAND Corporation. The original research was a classified military effort involving a group of multi-disciplinary experts whose goal was to try and forecast the effects of future warfare on the United States (Gordon & Helmer, 1964).

The defining characteristics of the Delphi technique are as follows:

  1. Participants are experts in their field.
  2. The technique uses a series of rounds or iterations where information is given back to the participants for review.
  3. Participants work anonymously.  They do not know who the other participants might be.
  4. Future focused
  5. The Delphi technique is a “consensus” research method.  In most cases, the goal is to approach a consensus among the expert panel as to future “best” solutions.  There are, however, exceptions to this, notably the Policy Delphi.

The original Delphi technique (“Classical Delphi”) was a research project to study the idea of achieving consensus among experts regarding complex military issues where little published information was available.  The process involved a series of questionnaires “interspersed with information and opinion feedback” (Helmer, 1967).

Without going into a lot of detail, the first Delphi studies sought numerical answers to complex problems.  For instance, how many bombs would the Russians need to drop on the United States to destabilize the U. S. infrastructure?  The first studies were primarily about achieving consensus among experts and trying to determine if this form of prediction was valid. For example, in one early experiment, participants were asked “What is the surface of the Moon in thousand square miles?” and “What is the area in square miles of Los Angeles County?” (Brown & Helmer, p. 3).  These were questions that most people would not know, although some of the participants might be able to make more educated guesses than others.  However, through a series of rounds, the study evaluated participant responses to 20 questions, and the primary focus was on how the participants responded to the answers from the other participants (all anonymous to one another).  Did the opinions of anonymous experts affect the answers of other experts, knowing that the ideal was to reach consensus?

From that early beginning in the 1950’s and 60’s, the Delphi technique has evolved and grown significantly.  Use of the Delphi method in medical research is the most common field of application, based on the number of articles published.  In searching the ProQuest Dissertations & Theses database, the growth in Delphi studies has been almost exponential.  There were 926 doctoral dissertations that used some form of Delphi study between 1971 and 2017, with more than a third in the last 10 years. There were 72 doctoral dissertations incorporating a Delphi study from the University of Phoenix in the last ten years. 

The Delphi technique offers a somewhat unique because of its focus on the future. There are challenges, however, with this research design.  Questions as to who is an “expert” need to be answered.  Is the Delphi technique valid?  Future posts will address some of those issues.

When do we use the design? –  Linstone and Turoff (2002), two of the earliest researchers with the Delphi technique, offered a number of common reasons why a researcher might select Delphi as the research design of choice:

  1. There is a lack of detail or information is incomplete in terms of the problem which makes precise analysis impossible, yet collective subjective judgments by experts might be of value.
  2. The potential participants needed to work on a complex problem might be very diverse experts with no history of regular communication.
  3. Researchers need to include more participants than could easily be accommodated in an on-ground setting.
  4. Money and time make frequent meetings, as a group, are impractical.
  5. Group meetings can often be a problem because of personality or strong differences of opinion, whereas anonymous communication could avoid that issue.

Types of problems appropriate for this design? This is the exciting part about the Delphi technique -- almost any problem is appropriate -- with some qualifications. The caveat here is that Delphi is ALWAYS future focused, so bringing together a group of co-workers to discuss best practices of some problem you are having is not appropriate. 

I just read an article by Jiang, Kleer, and Piller (2017) where they used the Delphi technique to study future economic and societal implications of 3D printing. While 3D printing may not be a topic on your list of things to consider, it is a technology that is growing rapidly and has application into medicine, tool manufacture, and even space exploration. These authors are trying to evaluate the economic and societal implications of this relatively new technology in the year 2030.  While the scale of their study was impressive, the number of participants (almost 300) would be very difficult for a doctoral dissertation. 

The point here is that almost any topic is appropriate. I have done Delphi studies on education and cyber security (Davidson, 2014). Healthcare is probably the most common area of application for Delphi studies, and this may be because 1) it is easier to qualify "experts" to use as participants, and 2) the field of healthcare is so complex that it offers many crossovers into other areas such as education (e.g. nursing education), infection control, and hospital administration itself. However, there are no limits. Strategic planning is also an area that uses the Delphi process in trying to be better prepared for future events. For example, Roßmann et al. (2017) studied how big data will affect supply chain management in the future. This is critically important for all businesses. In an older paper, Loo (2002) discussed the use of the Delphi technical are a tool "to help forecast the future for the purposes of strategic management" (p. 762).  

Theoretical framework/discipline background:  

The background of the Delphi Technique goes back thousands of years, specifically the Oracle at Delphi (Adler & Ziglio, 1996). As far back as 1400 B.C.E., the Oracle of Delphi was the place where royalty would petition the god Apollo through an intermediary – the Pythia – a female priestess of Apollo.  Everyone wanted to know the future, and that has not changed in 3418 years.  Flavius Josephus (1998), the ancient Jewish Roman historian (37 to circa 100 CE) who also spent some time as a priest of the Oracle in Delphi, noted that it was the priests who “interpreted” the meaning of Apollo’s future forecast that was spoken, often in very confusing verse.  It has been suggested that because Delphi was a central point of activity in the Mediterranean world, the priests were also very adept at listening to what people were saying and putting together a future forecast based on “expert” knowledge.

Prediction theory is the theoretical framework supporting research using the Delphi Technical. From a theoretical perspective in more modern times, one of the earliest words on prediction was by McGregor (1938).  McGregor examined the concept of predictions as a form of psychological inference.  “Any argument from a premise to a conclusion is an inference, and those that are expressed in the future tense are classified as predictions” (p. 179). Cantril (1938) elaborated on McGregor’s work, also dealing with the issue of validity and the certainty of predictions. The studies by McGregor and Cantril both focused on psychological variables related to predictions, such as bias and preference.

In the work of Kaplan, Skogstad, and Girshick (1950), the authors relied heavily on the work of McGregor (1938), and Cantril (1938), but focused primarily on the process of improvement of prediction.  Kaplan et al. focus on precision of prediction and how that might be improved.  The theoretical foundation of the accuracy of prediction (prediction theory) appears to depend on a wide set of variables, including the individual and that individual’s knowledge background, preferences, and biases.  There are also issues related to survey questions and structure, as well as timing.

Specific Characteristics –    

• Delphi is one of two primary consensus-focus research designs, the other being the Nominal Group Technique (NGT), and both are widely used in healthcare (Fink et al., 1984). 
• Delphi and NGT typically use multiple iterations of questioning to assist in reaching consensus (Hsu & Sandford, 2007), and there is no specific set number of rounds. 
• The researcher provides feedback to the participants between rounds to encourage participants to reconsider their earlier responses. 
• Delphi focuses on participant anonymity from one another (there are rare exceptions). 
• Delphi is focused on forecasting future solutions 
• Delphi works with problems that are poorly understand or missing data. 
• Delphi participants are experts in their fields.

Expertise and Number of Participants (“Sample size”) -  

This may be the most important and most difficult aspect of the Delphi Technique. Selecting and qualifying participants as legitimate experts takes a lot of time but is essential in validating the study. There are no specific criteria established for selecting Delphi participants, but in my own study for an article I am writing on this topic, the primary consensus is that an expert has the appropriate education background and work experience and is regarded by peers as an expert or someone to whom they seek out for advice. I look for people who are published or who present at professional meetings.

As far as the number, Delbecq et al. (1975) noted that “With a homogeneous group of people, ten to fifteen participants might be enough” (p. 89). However, the number can vary considerably, depending on the research goals, and we rarely have a homogeneous population when considering complex issues. Because of the effort required by the researcher and the participants, Delbecq at al. suggested that the number be “a minimally sufficient number of respondents and seek verification of results through follow-up survey research” (p. 89). Ultimately, the researcher must do due diligence in defining the number of participants and support the decision as to the number used.

Sampling Method –  

The sample is Delphi and NGT typically starts with a question or a questionnaire addressed to the panel of experts. That questionnaire provides a set of beginning data, and the researcher then combines and synthesizes the data from the panel. The research can also add new information from a literature review to add to the information garnered from the panelists.

For the second round, the researcher provides the first-round data back to the group in a way so that bias is not introduced. For example, do not indicate how often a response was made in round one. I typically alphabetize my answers so there is no hint of the order of importance. In round two, participants are asked to prioritize responses and explain why they answered the way they did (defend their position).

Round three (frequently the last round) varies considerably among researchers. Typically. issues have been resolved and the primary issues identified and prioritized. In this third round, participants express their individual judgements and preferences regarding the issues and how those issues can be “fixed” or implemented.

Data Analysis –    

for the first round is a matter of combining information, deciding upon which responses are the same as others that are not worded the same, and determining frequency of response rate. The first-round data is returned as noted above, without bias in the formatting, to the panelists.

The second-round analysis is much like the first, combining similar answers and calculating “importance” of the issues based on percentage of times that issues was noted by the panel.

The third-round data is textual. This is the round where the participants are asked to elaborate in detail as to the issues, their preferences, methods to fix the problem or help implement the solution(s). As this is textual, some sort of context analysis is appropriate.

Write up Results – 

When presenting the results from a Delphi study, we typically present each round. With round-one, we present a picture of the number of responses and the frequency with which suggestions are made. We would typically also indicate what was being sent back to the panel for the start of round two. Round two data deals with the initial prioritization, commonly associated with textual defense of their priorities. Priorities are condensed into those that are essentially the same and percentages calculated for the occurrence of each priority. Presentation of the data is commonly the priorities in the order (averaged) as suggested by the panel, with some commentary noted from the participants. Third round data would typically move into the category of textual analysis, looking for categories, sub-categories, and themes based on the textual responses of the participants. As issues and priorities were typically resolved at the end of round two, this third round can provide a very rich thematic underpinning to the data. When I present the data, I show an abbreviated list of categories and (maybe) subcategories, and then present the themes and explain their development from the categories and subcategories.

References

Adler, M., & Ziglio, E. (Eds.). (1996). Gazing into the oracle: The Delphi method and its application to social policy and public health. London, UK: Jessica Kingsley Publishers.

Baker, J., Lovell, K., & Harris, N. (2006). How expert are the experts? An exploration of the concept of 'expert' within Delphi panel techniques. Nurse Researcher, 14(1), 59-70.

Cantril, H. (1938). The prediction of social events. The Journal of Abnormal and Social Psychology, 33(3), 364-389. doi:10.1037/h0063206

Dalkey, N. C., & Helmer, O. (1963). An experimental application of the Delphi Method to the use of experts. Management Science, 9(3), 458-467. doi:10.1287/mnsc.9.3.458

Davidson, P. L., & Hasledalen, K. (2014). Cyber threats to online education: A Delphi study. Paper presented at the 2nd International Conference on Management, Leadership and Governance, Boston, MA.

Delbecq, A. L., Van den Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning: A guide to nominal group and Delphi processes. Glenview, IL: Scott Foresman Company.

Fink, A., Kosecoff, J., Chassin, M., & Brook, R. H. (1984). Consensus methods: Characteristics and guidelines for use. American Journal of Public Health, 74(9), 979-983.

Gordon, T. J., & Helmer-Hirschberg, O. (1964). Report on a long-range forecasting study. Santa Monica, CA: Rand Corporation. Retrieved from Retrieved from http://www.rand.org/content/dam/rand/pubs/papers/2005/P2982.pdf

Helmer, O. (1967). Analysis of the future: The Delphi method. Retrieved from http://www.rand.org/content/dam/rand/pubs/papers/2008/P3558.pdf

Hsu, C.-C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8.

Jiang, R., Kleer, R., & Piller, F. T. (2017). Predicting the future of additive manufacturing: A Delphi study on economic and societal implications of 3D printing for 2030. Technological Forecasting and Social Change, 117(April 2017), 84-97. doi:10.1016/j.techfore.2017.01.006

Josephus, F. (1998). The works of Josephus: Compete and unabridged (W. Whiston, Trans.). Nashville, TN: Thomas Nelson, Inc.

Kaplan, A., Skogstad, A. L., & Girshick, M. A. (1950). The prediction of social and technological events. The Public Opinion Quarterly, 14(1), 93-110. Retrieved from http://www.jstor.org/stable/2745902.

Linstone, H. A., & Turoff, M. (Eds.). (2002). The Delphi method: Techniques and applications. Retrieved from http://is.njit.edu/pubs/delphibook/delphibook.pdf

Loo, R. (2002). The Delphi method: A powerful tool for strategic management. Policing: An International Journal of Police Strategies & Management, 25(4), 762-769. doi:10.1108/13639510210450677

McGregor, D. (1938). The major determinants of the prediction of social events. The Journal of Abnormal and Social Psychology, 33(2), 179-204. doi:10.1037/h0062931

Roßmann, B., Canzaniello, A., Gracht, H. v. d., & Hartmann, E. (2017). The future and social impact of big data analytics in supply chain management: Results from a Delphi study [in press]. Technological Forecasting & Social Change. doi:10.1016/j.techfore.2017.10.005

 

Additional Resources 

Link to Discussion Thread  https://hangouts.google.com/chat/person/111966120928693716501