Polling associations look to increase transparency with merger, but not everyone is on board
'Standards should be about accuracy, not process,' says Quito Maggi of Mainstreet Research
Two organizations representing market research firms and pollsters in Canada have announced they are entering into merger talks to speak on behalf of the public opinion polling industry with one voice and apply a uniform set of standards for their members.
But at least one prominent polling firm has concerns that some aspects of those standards could undermine the level playing field in the industry.
Formed in 2004 out of a merger of three industry bodies, the Marketing Research and Intelligence Association represents individuals and organizations across the market intelligence and survey research industry, including firms that conduct public opinion surveys.
Last year, a group of pollsters, researchers and media members who felt MRIA's disclosure requirements were not up to international norms, formed the Canadian Association for Public Opinion Research to create a code of conduct they felt had higher standards of disclosure.
Earlier this spring, however, MRIA announced a new set of standards that brought the association closer in line with their American and European counterparts. According to MRIA's press release, the "new standards are also very much in the spirit of the standards put in place by CAPOR."
That led to talks over the summer to bring the two organizations together. Once negotiations are concluded, the merger proposal will be put to a vote by members early next year.
Kara Mitchelmore, CEO of MRIA, says a merger will allow the industry to speak with one voice. She says there had been a consensus among its membership for the need for Canadian disclosure standards in public opinion polls — the most visible work the industry does.
Darrell Bricker, president of CAPOR and CEO of Ipsos Public Affairs, says that with these new standards it made sense to bring the two organizations together.
'The more exposure the better'
The new disclosure standards adopted by MRIA require polling firms to release more information when reporting new polling results than had previously been the case.
Among other things, the standards require polling firms to report, whenever any poll is publicly released, the size of the sample, the field dates of the survey, the method of polling and how any weighting was applied. These are details that polling firms have reported for years.
But additionally, the new standards require the reporting of the number of "call-backs" conducted (when pollsters call numbers again if they get no initial response), the proportion of undecided respondents, the "reasons for any 'rogue' polls or drastically different results between polls" conducted on the same subject, and both weighted and unweighted data tables.
"The more exposure, the better," says Mitchelmore, the idea being that if polling firms are forthcoming with the details of how they get their results, it will help foster public trust in their findings — something that has been lacking after a few high-profile polling misses in recent years.
Violations of these standards could result in censure, suspension or expulsion from MRIA.
There have been criticisms in the past that MRIA has not enforced its standards rigorously enough, but Mitchelmore says the new complaint resolution process will work better, more quickly and treat all parties fairly.
These minimum disclosure standards, says Bricker, make it possible to determine the quality of a poll.
With telephone response rates plunging in the industry and pollsters increasingly moving online to reach people, there are concerns that pollsters are struggling to gather representative samples. Weighting can correct for this, but there are limits.
The recent case of the USC/Los Angeles Times poll's extreme over-weighting of one unrepresentative black supporter of Donald Trump — enough to move Trump's national support in polls by a point or two alone — is one example of what full disclosure can reveal.
Standards for accuracy, 'not process'
But this push for more transparency, while generally welcomed by most pollsters, has raised concerns in some corners of the industry.
Quito Maggi, president and CEO of Mainstreet Research (political pollster for the Postmedia network of newspapers), feels that these disclosure requirements may force polling companies to reveal proprietary information.
Maggi says that publicizing unweighted data tables would allow other polling firms to reverse-engineer Mainstreet's weighting algorithm, which Mainstreet has put a lot of time and resources into developing. Correctly weighting a sample according to who is likely to vote in an election can make the difference between nailing and missing a call.
"Standards should be about accuracy, not process," says Maggi.
Maggi says that his company's unweighted tables are shown to its clients and are hosted at the Fields Institute at the University of Toronto (Forum Research, a firm that also uses interactive voice response technology, also houses its data at the university). But according to Maggi, a wider dissemination of Mainstreet's unweighted tables would lead to a loss of competitive advantage.
Bricker says that kind of disclosure is not good enough, and that if a company like Ipsos — one of the largest polling organizations in the world — can adopt these disclosure standards, others can as well.
But according to Maggi, the law is on his side.
"We had previously sought our own legal opinion which affirmed to us that the standards that were being proposed by CAPOR did not comply with Canada's Competition Act and we filed a competition complaint against CAPOR at that time," says Maggi.
Bricker says that CAPOR has not yet heard from the Competition Tribunal regarding this complaint.
But Maggi believes "the new MRIA reporting guidelines are similarly non-compliant with Canada's Competition Act as it pertains to industry associations" and is waiting to receive word from the MRIA on how it will address his concerns.
Mitchelmore says MRIA is sensitive to worries about proprietary information, but is moving toward a full disclosure model. For her, the priority is ensuring that widely reported polls do not mislead the public.
"There's no doubt that polls influence public opinion," says Bricker. "This level of influence places a special responsibility on pollsters to be accountable for their work. Accountability starts with being open and transparent about your methods. That's why there needs to be a set of unified professional standards and disclosure rules for all of Canada's ethical pollsters to follow."
ABOUT THE AUTHOR
