Evaluation and social media
When evaluating public campaigns aimed at social or policy change, we are often asked to provide insights on social media strategies. Clients may provide large quantities of data relating to online petitions and levels of engagement with posts on Twitter and Facebook and so on, with an expectation that we will be able to determine whether particular strategies have been successful. In itself, this is a reasonable enough request, but it is one which tends to come with certain assumptions – that campaign targets are sensitive to social media activity and that more – hits, likes, followers – is better. It is these assumptions behind the data, rather the data themselves, which are the proper focus for scrutiny in a campaign evaluation.
A direct causality between a number of clicks and outcomes of a campaign can’t rightly be assumed. A brief review of petitions on change.org reveals that there is no simple correlation between number of signatories and campaign impact. A total of 37,000 people signed a petition as part of a successful campaign calling on the Bank of England to keep a woman on bank notes, but a petition signed by nearly 500,000 people demanding that the-then UK Work and Pensions Secretary Iain Duncan Smith be forced to live on £53 a week after saying this was possible did not achieve its goal. More recently, 1,276,628 signatures did not manage to ban Donald Trump from entering the UK although it may have helped raise some doubts about whether it would actually go ahead.
In all these cases, signatories on the petition were anyway only one part of the influencing equation.
Social Media strategies can help produce public pressure when campaigns aim for significant changes. But a sophisticated campaign would not, of course, hinge on sheer weight of numbers. It would be grounded in a thorough understanding of the forms of pressure that campaign targets are susceptible to and work back from this to identify audiences, tactics and activities, including social media, to bring influence to bear upon them. In turn, this would be broken down into specific objectives for different media channels – how far each contributes to mobilisation at any of several stages:
- getting more people to know about an issue; or
- getting more of the people who know to agree; or
- getting more of the people who agree to care; or
- getting more of the people who care to act; or
- making sure that the actions of those who act are visible to decision-makers and influencers.
This still leaves the question as to how to measure whether progress is being made across these stages and how important social media is to that. There may be scope to draw comparisons with an organisation’s own previous social media engagement, taking into account differences between campaigns and the scale of investment of resources. Or it may be possible to measure against social media activities by others, whether on a similar issue or by those of a similar profile and size.
In this, any or all of the following indicators may be useful:
- Trends in the adoption and use of campaign hashtags on Twitter;
- Levels of activity and profile of Twitter ‘super-users’ – how many connections they have, how active they are on the campaign issue;
- Basic engagement metrics (retweets, likes, comments, etc.);
- Webpage visits, including to compare visits on regular days with visits when a campaign is at its height;
- Mapping spikes in Facebook reach, top-performing Instagram posts etc. against levels of campaign activity;
- Monitoring the reaction of campaign targets on their own social media channels or in the press – how often they post content in reaction to a campaign may be an indicator of how engaged or even “annoyed” they are by it.
It is important not to reify data derived from these sorts of indicators, but to find the link between the data and campaign outcomes. Data have meaning only when understood in context and given qualitative flesh by information harvested from interviews with those involved in, or targeted by, a campaign. Two examples are instructive here. A review of the Occupy Oakland campaign drew on quantitative Twitter data and qualitative information from interviews and participant observations. It established that both physical places and online spaces were important to the work of activists. Although social media are used within the Occupy movement, physical space remains a prominent component, and critical consideration, for this movement: the physical and digital are not distinct aspects of the movement, occurring in isolation, but are interlinked.
As a second example, the Oxfam corporate campaign Behind the Brands combined evidenced-based advocacy with digital mobilization, media, offline and online stunts, advertising, investor and shareholder activism, case studies, and collaboration with allies and influencers. Companies made significant concessions, both because of the campaign’s direct engagement of them and because of actual or threatened public mobilization.
As these examples suggest, the meaning of social media activities comes only from the context in which they are delivered. Social media outreach is a tactic whose effectiveness depends on the deployment of other, complementary tactics. Without reflection on the wider context and without considering social media outputs as part of overall campaign strategies, data for the number of clicks, likes and such like are only random numbers.
Elena Lucchi, with thanks to colleagues in the Hub for the discussion and the relevant inputs
 Engagements: Total number of times a user interacted with a Tweet. Clicks anywhere on the Tweet, including Retweets, replies, follows, likes, links, cards, hashtags, embedded media, username, profile photo, or Tweet expansion – https://support.twitter.com/articles/20171990
 Occupy Oakland and #oo: Uses of Twitter within the Occupy movement by Sky Croeser and Tim Highfield. First Monday, Volume 19, Number 3 – 3 March 2014. Available at: http://firstmonday.org/ojs/index.php/fm/article/view/4827/3846#author
 Coe, J., What the data can tell us. May 2017, http://www.coeandkingham.org.uk/evaluation/what-the-data-can-tell-us/