Rural Matters

According to the Department, that was because about half of the 882 individuals who showed up are school board members. By contrast, the survey sample was described as “random” and, by implication, representative of all Vermonters.

It is not.

It is not clear from the news release or the full survey report just who was surveyed, but it is very clear that the responses were not random. A national data service provided 4,000 names and addresses. These people were simply mailed a survey form and invited to respond. Only 301 did.

So the survey results consist of responses from people who decided to respond. This is called a “self-selection” response , and whether the invitation list was randomly selected or not, the responses are motivated, not random. These 301 are no more statistically representative of any larger group than are the 882 people who showed up at public meetings to which everyone was invited.

There are other sampling issues with this survey. In an attempt to make the survey a random survey of households, the survey architects required the national data service that drew the sample to include no more than one person from each household in the final sample. That gives each household an equal chance of selection.

The problem is that households do not have opinions. Individuals have opinions, and within households, these opinions sometimes vary. And that means that not everyone in the population (or on the voter lists) had an equal chance of being selected.

It also means the sample over-represents people who live in one-person households (who are never thrown out, if selected) and under represents people in households with more than one person (who will be thrown out if someone else in their household has already been selected).

In and of itself, these faults do not mean the data gathered is useless. It just means it cannot be described as representative of anything other than the opinions of 301 people who decided to share their views on school governance.

But there are other factors that call even that use of the data into question.

Those surveyed were provided “information” the report claims made them “more knowledgeable” than the general public.

The information was prepared by the Department whose Commissioner has taken a position on the issue being surveyed. The information included graphic representation of the Commissioner’s proposal and of the current school governance system. These charts were plainly designed to make the current system appear complex and unwieldy and the Commissioner’s proposal simple and streamlined.

The narrative provided also included patently biased statements describing the current system as producing “differing priorities” of school boards and “very different outcomes for the students” while suggesting that under the proposed system “policy direction is generally clearer.”

This information does not make the respondents any more knowledgeable about anything other than Commissioner Cate’s point of view. Saying it does is akin to saying that juries would bring better verdicts if they retired to decide the case immediately after hearing the prosecution’s opening statement.

The respondents might have been provided the perspective of the supporters of the current system who say its complexity is a function of its deep democratic design, while the streamlined look of the Commissioner’s proposal reflects the administrative power of a centralized bureaucracy. That point of view was not provided.

Finally, the report wrongly discounts school board members and school employees who responded to the survey, saying they are less favorable than others to Cate’s proposal because they have a “vested interest” in the current system. But nothing in the survey provides any basis whatsoever for describing the motive of any respondents.
It is plausible that this group has a lower opinion of Cate’s proposal than other respondents only because they are better informed. It is also plausible that they responded at a higher rate than did others in the sample because as elected officials and school employees they have an official and professional duty to care about education that others in the general public do not have.

This survey is nothing but a classic “push poll” designed to influence voter opinion, not to measure it. It influences opinion directly by giving respondents one-sided information before they answer the survey questions, and indirectly by misrepresenting the results as scientifically representative of the point of view of the general public. This is not the kind of service we need from the Vermont Department of Education.