Yesterday’s public opinion poll (and by extension, all of the ones we’ve done) has met with some professional criticism, to which I feel it’s important to respond. (Perhaps unsurprisingly, the only ones I’ve seen spiking political touchdowns with the response have been those who support the proposed anti-discrimination law.)
First, it was claimed that the questions we used were “unbalanced and inaccurate.” Specifically, the objection was made that we referenced “jail time” in our questions about anti-discrimination law, and that this punishment is “non-existent” in regards to anti-discrimination law. The purpose of this question was not to gauge reaction to any specific proposal or bill language, but to test the degree to which voters would be willing to use the violence of the state against a peaceful business owner. Further, a business owner who would be fined under the proposed law, and who objected to paying that fine, could easily find himself in a legal battle that escalates to the point of jail time. Thus, we felt it was not inaccurate to raise this as a possibility when discussing the state’s prohibition on peaceful behavior.
Second, the question order we used was questioned, the concern being that respondents were “effectively biased” as a result. This may certainly be a possibility, but our intent here was to address the fundamental principles involved in anti-discrimination law before drilling down into specifics. Our purpose was not to intentionally bias or sway any results, though I’ll defer to the professional pollsters that this may have been the effect of the question order.
Third, the use of automated calls (instead of in-person survey takers) was questioned by noting that such surveys do not have the ability to verify the individual’s identity or ask for a random voter. The initial question in this survey asked for the respondent’s age, and only those who indicated that their age was over 18 (to be eligible as a voter) proceeded with the questionnaire. But as this concern likewise noted, we did not weight the survey and thus the results were not as accurate as would have been produced by a weighted survey. And despite some speculation, the random selection of Utah voters was not limited to any geographical area; callers were selected at random from the master Utah voter list.
Fourth, it was noted that the survey data file we provided contained the phone numbers of respondents. This was a technological error resulting from our application’s export function, and was immediately corrected as soon as it was discovered. We have taken additional steps to ensure that this does not happen in future polls.
Which brings me to the fifth option. When we began conducting polls, we recognized that they would not be as “official” as professional, costly polls that cost $5,000 to $10,000 a piece. We wanted a way to more cost effectively gauge public reaction to certain issues that professional pollster’s aren’t covering. Still, attempts were made to solicit input from consultants to help ensure that the results would be statistically relevant enough to be more factual than, say, a public internet survey. As part of this initial advice we received, it appeared that we were incorrectly informed that non-profits were exempt from certain FCC restrictions, but yesterday’s survey prompted additional research into the regulations, including calls made directly to the FCC, to discover that this was not true.
As such, we will be suspending our polling activities for the foreseeable future while we take the time to better determine the relevant regulations (interestingly, it appears that numerous parties in Utah are violating these same FCC regulations with political robocalls during campaigns) as well as improve our processes such that the results will be more reliable.
As for the subject of yesterday’s poll, we remain confident that our theory would be likewise proven by any professional pollster—that being that if Utahns are better informed as to what the consequences are of violating anti-discrimination law (even if only referencing fines or loss of a business license), they would not be as supportive of it as proponents claim. In the end, what is most relevant is the sentiment amongst state legislators. Our guess is that the legislative proposal will once again be defeated—an outcome we encourage.
