An opinion poll was widely reported last month as evidence that 55% of Syrians think President Bashar al-Assad should not resign. But does the claim stand up to scrutiny?
The world is watching Syria, where every day there are new scenes of horror as the violence between protesters and the regime’s security forces continues.
Against this backdrop, some commentators have picked up on a striking statistic - that 55% of Syrians want President Assad to stay in power.
In a column in the UK’s Guardian newspaper, the statistic was used to suggest that the Western media was mis-reporting the situation in Syria, suppressing “inconvenient facts” for the purposes of propaganda.
The statistic has been reported widely elsewhere, from the New York Times, to Al Jazeera (in Arabic) Iranian owned Press TV, and Syrian news sites.
• More than 1,000 people from 18 countries in the Middle East responded to YouGov Siraj’s internet poll question: “In your opinion, should Syria’s President Assad resign?”
• 81% of answered Yes
• 55% of respondents in Syria said they thought the president should stay
• Only 98 respondents were actually from Syria
• Only 18% of people in Syria have access to the internet
So what was this poll and who carried it out?
It was an internet survey of the Arab world by YouGov Siraj in December. It covered just more than 1,000 people in 18 countries in the Middle East and North Africa.
The central question was: “In your opinion should Syria’s President Assad resign?”
Across the whole region, the overall finding was that 81% of people polled thought President Assad should go.
But the polling company also stated: “Respondents in Syria are more supportive of their president. 55% do not believe Assad should resign.”
Looking closely at the survey report, it does not say explicitly how many of the 1,000 people who responded were from Syria. But it does say that 211 were polled in the Levant region, 46% of whom were in Syria.
Doing the sums, this suggests that only 97 people took part. When the BBC checked with YouGov Siraj for the exact breakdown, the company said that in fact there were 98 respondents from Syria (the difference arising from the fact that averages given in the survey report were rounded).
This is a very low sample according to the managing director of survey company ORB, Johnny Heald, who has been carrying out polls in the Middle East for many years.
“When we poll and we want to find out what Libyans think, or what Syrians think, we would rarely do anything less than 1,000 interviews,” he says.
“One thousand is the generally accepted industry minimum to be able to speak confidently about what people from a particular country think about an issue.
“If you say that this poll covers people from 18 countries, then that’s fine. But you need to be very careful when you interpret the findings.
“It is not good to say that 55% of Syrians, for example, think that Assad should stay when only 97 people were asked that question.”
But he has another criticism - according to UN figures, only 18% of people in Syria have access to the internet, which means that the sample polled is biased towards those who can get online.
The people who conducted the survey at YouGov Siraj, the Dubai-based arm of a UK polling company, say the poll was not intended to be representative of all Syrians.
They too say the sample was too low for this and that internet penetration in the country is not good enough.
This is why they referred to “respondents from Syria” rather than referring to “Syrians”, they say.
However, the Doha Debates TV programme, which commissioned the poll and published its findings, were not as sensitive to the distinction.
The figure is described on its site as: “Syrians are more supportive of their president with 55% not wanting him to resign.”
In a statement, the pollsters at YouGov Siraj said that with hindsight they wish they had been clearer: “To the layman, there seems very little difference between the two expressions but for researchers, the difference is huge.
Johnny Heald Managing director of survey company ORB
Journalists have jumped on (the statistic) and ran with it, without thinking about the science behind how they came to that figure”
“I think we should have stressed the difference much more to our client (or simply not shown the Syria data, as there was always a chance it might be misinterpreted).”
When we asked the organisers of the Doha Debates about the statistic, they insisted that despite the small sample size, the result was “of interest”.
They say the figures and polling data are freely available for people to draw their own conclusions.
Is it OK to put out a figure based on such a low sample?
Johnny Heald thinks it is acceptable for pollsters to pull out data from a broader poll, because often it is interesting.
But he says: “What you should always do is say: ‘Caution - this is a low base size.’
“The problem comes when people interpret it to be representative of a country.
“And I think in defence of YouGov, they don’t claim the poll is nationally representative of what Syrians think.
“They have just pulled out the Syrian numbers and because it is an interesting story and somewhat controversial, I think the journalists jumped on it and ran with it, without thinking about the science behind how they came to that figure.”