 |
Different Methods = Different Results
Do you think it doesn't matter which kind of "voting" or survey technique you use? Here's an example of how different survey research methods can give you different results.
 The wise man doesn't give the right
answers, he poses the right questions
|
|
Claude Lévi-Strauss
|
|
|
The President's Dilemma
The president of a manufacturing company decides to get the views of her 15 top executives to determine the optimal strategy for bridging a shortfall in manufacturing capacity. Financial constraints allow only one of the following three options to be selected:
- Construct a new plant (Plan A)
- Purchase a plant being sold by another company (Plan B)
- Expand the company's existing plant (Plan C)
The Vote
She decides to put the issue to an informal vote. Conferring with each one individually, she polls them on their views of the three alternatives. Of the fifteen executives, six recommend A, five choose B, and four opt for C. A appears to be the preferred alternative, followed by B, and then C. Or is this the case?
The Rating Scale
Because a lot is riding on the decision, the president decides to get some scores using the rating scale, and asks her execs to "rate" each alternative with scores from one (lowest) to five (highest). The results are as follows:
Six execs: A-5,B-1,C-2
Four execs: A-1,B-4,C-2
One exec: A-1,B-5,C-2
Four execs: A-1,B-2,C-3
Totals: A-39 pts, B-35 pts, C-34 pts
Seeing that the numerical data support the informal vote and feeling the support of her executives, she discusses in her next meeting the prospect of constructing a new plant. A lively discussion ensues, during which strong opposition is voiced against constructing a new plant, and no agreement is reached. The president is surprised, believing her data had indicated support for the new plant. What went wrong?
The Paired Comparison
The president returns to her data. When she uses the executives' scores to determine the preferences of the executives, the data show that six execs preferred A to C to B, five preferred B to C to A, and four preferred C to B to A. So, asked to choose between A and B, they would vote 9 to 6 for B. Asked to compare A to C, they would select C by a 9 to 6 margin. And by a 10 to 5 vote they would prefer C to B.
The paired comparison outcome of these decisions yields 19 points for C, 14 points for B, and 12 points for A.
Based on the comparative results, the president meets again with her team and floats the idea of expanding the existing plant, to the almost unanimous support of her executives.
In reality, then, the executives' clear preference is for option C, followed by B and then A — precisely the opposite ordering given by both the simple vote and the rating scale system!
What We Learned From This...
We believe this illustrates how the method chosen for survey research is not just a matter of preference, but can directly affect the outcome — and can give you an entirely different result.
So what is the right method? There is no single right method for every situation, but there is a right starting point — the decision you have to make. The choice of a survey method depends on the kind of decision you are trying to make. Most decisions are "whether" decisions or "which" decisions.
"Whether" Decisions
If you are trying to decide whether to dismiss an employee "for cause", or whether to split the company stock, or whether customers like a new package design for a product, then you have a "whether" decision. "Whether" decisions usually have binary solutions — yes or no, up or down, guilty or innocent. They are also called non-comparative decisions because the decision does not require comparison with similar issues or situations, and the outcome does not directly affect any other issue. Example: If you dismiss an employee for cause (a "whether" decision), the decision is unaffected by that person's performance compared with other employees, nor does it directly effect any other employee.
There are non-comparative assessment methods for "whether" decisions. The rating scale, checklists, behavior observations, critical incidents, even simple narrative evaluations can provide the input necessary for making these decisions. Voting is a method that may be applied to "whether" decisions, but only in the single case when there are exactly two choices, and a majority vote can decide the issue.
The President's Problem - The president in the example above tried to use voting with three outcomes (A, B and C) instead of binary (2) outcomes. The result was that the preference with the plurality of votes (A with 6) represented a minority of the executives. A majority of the executives (9) did not prefer A. She counted the votes correctly, but the method was not the right one for the task. When she applied the 1 to 5 rating scale to the problem, she counted the scores correctly, but again was applying a "whether" method to a "which" decision.
"Which" Decisions
If you are trying to decide which issue is more important, or which person should be promoted, or which skill is the most important to train, then you have a "which" decision. "Which" decisions have multiple outcomes greater than 2. They are also called comparative decisions because each outcome must be compared with all the others to decide "which" one of them is the right choice. Example: If you intend to promote 1 of 3 candidates to an executive position, each candidate must be compared with each of the others, one of whom is promoted, and 2 who are denied promotion, and may choose to leave the company.
The comparative assessment methods include ranking, paired comparison or Scaled Comparison. Voting would not be applicable to comparative decisions. Sometimes a process called "voting" is used with multiple candidates, where the voters are instructed to identify their first choice with a "1", their second with a "2" and their third with a "3". Then the first place votes are tallied, then the second, and the third. This is really a simplified ranking process, when you are only interested in the top of the ranking. True voting has no second or third place votes.
The president's problem was a comparative decision from the very beginning — there were multiple possible outcomes and she wanted to find out "which" plant capacity solution (not "whether") the executives thought was more advisable. Only when she applied a comparative process to her data did she obtain the answer that the executives had been trying to give her. It is important to realize that the executives were clear in their preferences and answering consistently when asked. Only the different methodologies made the situation appear perplexing.
The Paradigm of Limited Resources
A single variable can often change a non-comparative decision into a comparative one. That happens when resources (such as time, money, space, personnel, or intellectual capital) are scarce. A scarcity of resources may mean that
- although a new plant, a purchased plant and a plant expansion (A, B and C from our example) may all be viable options, the financial resource imitations required the president to determine which was the "better" option.
- although an organization has identified 10 competencies as important, only 6 can be adequately trained in the current budget cycle. Now the organization has to identify which are more important.
- although 3 middle managers are qualified and eager for promotion, the slower growth rate of the company results in only one new executive position, so it must be determined which of the 3 managers is better qualified for the position.
In all of these cases, and many more, finding out whether something is important is less of an issue than identifying which is more important given our current constraints. Only the comparative methods can reliably provide us with the data for difficult comparative decisions.
The reality of today's decision making is that we must work harder to ensure that our decisions are reflected in our strategies for data gathering. If we are not clear about the decisions we are trying to make from the data we gather, we will persist in asking inappropriate questions, and assembling them into inappropriate questionnaires.
Click here for a more technical discussion of the Scaled Comparison.
How can I get more information?
We are indebted to George A. Hazelrigg, Ph.D., of the National Science Foundation for the numerical example shown here.
|
|
|
