Audience surveys, segmentation and OpenAudience

How do you get to know more about what kinds of people attended an event?

Two fairly common quantitative approaches to this question are surveys and segmentation. Regarding the latter, we’re talking specifically about geodemographic segmentation, rather than sales or marketing based segmentation.

What if someone was able to do both and compare the difference in results?

Or is this more like a Ghostbusters ‘don’t cross the streams’ kind of thing? Read on and find out.

Practically speaking, both surveys and segmentation require some effort in data collection although segmentation can typically be easier given that you only need to rustle up a list of postcodes, usually from a box office. On the other side, if an event doesn’t have a box office, perhaps being free to attend, you may have to do some further thinking. Both surveys and segmentation could also be described as empirical methods, attempting to make some kind of valid, repeatable, unbiased observation, either directly or indirectly.

Geodemographic segmentation is based on the assumption that people who live in similar physical areas share similar demographic (age, ethnicity, employment…) characteristics. Many methods along these lines use the national census as a key source of data. The national census can be viewed as a kind of survey, albeit one with a 100% sample.

Census (n.) from Latin census, originally past participle of censere “to assess” (see censor)

The logic goes that if you know where someone lives, you can assume their characteristics are similar to those of other people in that physical location. Therefore you can infer their characteristics to a degree of accuracy without having to directly ask them yourself. I am not sure how quickly census statistics ‘age’ but at the time of writing, the most recent stats are 7 years old (2011).

What sort of physical location are we interested in, though? A house? A neighbourhood? A city? The national census uses a range of terms to describe location, and while they collect data at a household level, they do not publish the data at this level for various, fairly obvious, reasons. (Hello, neighbour…) The smallest/finest level of detail that is published in the UK are called Output Areas (OAs)

The precursor to OAs in the national census were Enumeration Districts (used from 1961-1991). These ‘EDs’ were based on the amount of work each individual census-taker (enumerator) could do, presumably in one shift or day. While they were presumably ‘fairly similar’, there was no real statistical basis for their existence so you could not fairly compare one ED to another.

OAs addressed this, starting with postcode blocks, with a target for each OA to contain 125 households (averages still vary quite a bit). OAs have a minimum limit of 40 households and 100 individuals, for confidentiality reasons. Therefore an OA in Scratchy Bottom, Dorset should have roughly the same number of households as an OA in Brown Willy, Cornwall. (Thanks, list of rude place names in the UK). An OA in a rural area may cover a large area, in a densely populated urban area, an OA will cover a smaller area. Actual OAs have thoroughly dull but technically expedient names like ‘E00141220’. There are around 175,000 of them acrosss England and Wales.

To reinforce the point: segmentation is based on the assumption that individuals in a given area share characteristics in a way that is ultimately useful for decision making. Think about your immediate neighbours and neighbourhood. How much would it be fair to assume about the surrounding 124 households based on your own characteristics? In my particular neck of the woods, I estimate that’s something like: 14 semi-detatched households each side of the street = 28 households each street = about 4-5 streets. Of course, you may (probably will?) be biased in this analysis, especially if you can’t stand your neighbours. On the other hand, how accurate do we *really* need this to be? Some data is better than no data, right?

For audience research purposes, we’re not particularly interested in any one individuals’ personal characteristics, but we are, probably, interested in the aggregate characteristics of the whole group. So, we take all our individual assumptions and build one big assumption out of them. Both surveys and segmentations have similar assumptions here; you might call it ‘the law of large numbers’. Even if your measurement tool is wrong some of the time, the more you use it, the more the total effect of those errors is smoothed out in the end.

BUT BIAS! What if our tools are broken and our assumptions are wrong and other such things? Well, along the lines of post-positivism or critical realism I believe they are accurate to the degree that they allow us to make useful observations and decisions, and I do not actually expect them to reflect the full entirety of the world. A postcode isn’t a person and neither is a completed questionnaire. Hooray for multiple methods, triangulation and so on.

Surveys or segmentation?

Say you are a typical, handsome, intelligent and well-read events manager. (Have you done something new with your hair, by the way?). You might be thinking, do I really need to survey my audience or can I ‘reverse-engineer’ a decent enough view of them through geo-demographic segmentation?

The short version is: surveys and segmentation are two different approaches and, really, there is nothing to stop you using both if you like.

The long version is: well, keep reading.

You can read more about my work with Leicester Arts Festivals (LAF) here. In short, we collected a lot of surveys from a variety of festivals in Leicester, and I’m going to compare some of the results here to the results from an open-source segmentation tool called OpenAudience.

“Open Audience uses open data to analyse the demographics of your audience or users within England and Wales. Paste in a list of postcodes and see what type of people live there. Try it at openaudience.org”

This was developed by Thomas Forth and IMActivate, who are based in Leeds. You can read more about how it works here: https://github.com/thomasforth/openaudience. In short, it takes lots of official stats and data from sources like the National Census and matches this up with the postcodes (and therefore relevant Output Areas) that you provide. If I knew more about coding I would probably like to mess around with it or add in data that is more localized to Leicester. Maybe one day. I think it is technically very handy but more importantly the open-source nature of it is something arts, events and the cultural sector at its very broadest could really stand to benefit from and do more of.

The LAF survey collected a variety of data, so we’re just going to pull out the few bits that can be compared with OpenAudience. Age and ethnicity were collected in the survey in line with national census categories. We’ll also look at deprivation and social class, but these were reverse engineered from survey postcode data.

The festivals have been anonymized as A,B,C,D and E. The survey samples for each range from around 100 to 300, average 165. Completely coincidentally, they were shuffled up and labelled in ascending order of sample size, so if you like, we might assume that the results for E have a better margin of error than those for A and so on. As OpenAudience just needs postcodes, this varies only slightly as some people who filled in a survey may not have given a postcode, or only given a partial one etc.

Ethnicity

We’re going to look at Ethnicity first as this is the simplest variable to compare (in this fairly limited analysis). The survey used national census categories (18 + a number of ‘other please state’) which have been collapsed to form a comparable variable with OA’s Black and Minority Ethnic (BAME).

What can we see? The overall trend, aka ‘which festivals had a more ethnically diverse audience’ is roughly the same, though perhaps in the case of B & D the segmentation results are slightly more varied (6% difference) than the Survey results (3% difference). Or put another way, if you compared the segmentation of B with the Survey of D you would see a 19% difference. In one case (C) did segmentation suggest a higher level of ethnicity than the surveys.

Some thoughts: Does this mean that people from BAME backgrounds are more likely to participate in surveys, or at least, more likely to give a response to the relevant question, rather than skip this question? Does this mean that people from BAME backgrounds are more likely to attend arts festivals of the sort selected here (more likely than geodemographic segmentation would suggest?) Does this suggest that many of the festivals had greater BAME representation in their audience, again, than segmentation alone would suggest? Also, in this particular case, it might be that the population of Leicester, specifically, has an influence on the relative effectiveness of data from both surveys and segmentations: (BBC: Census 2011: Leicester ‘most ethnically diverse’ in region) How accurate is the census when it comes to Ethnicity? (Self-reported so, presumably, 100%?) If you hadn’t already guessed, I have no particular answers to these questions!

A quick chi-square test also tells us that all of the differences here (A/A, B/B etc) are statistically significant (at 0.05), taking into account the marginally different sample sizes. Given the differences in technique, this is not a huge surprise but confirms what we expected.

Age

Here are the results for Age, comparing the survey results to the segmentation results.

Note, age categories did not line up perfectly but have been fudged together a little. The OA categories, collapsed here, were actually: 0-12, 13-18, 19-30, 31-59 and 60+. Only a few years either way.

At first glance, they look pretty similar and again, if we were only comparing the results to each other, we would probably come to the same conclusions: eg, which festivals had an older or younger audience than the others.

Rather than just present the data as is, we can instead look more directly at the differences between each. Essentially in the chart below, if there are no bars above or below the line, we would be able to say that the two sets were the same. Where the bar is above the line, this means the survey reported a higher percentage, where it is below the line, segmentation had a higher percentage.

We’ve certainly exaggerated some of the differences here, or at least made them easier to spot. The age brackets are not evenly sized either, but it certainly seems that the 30-59 group is overrepresented in the surveys while 60+ tended to be underrepresented. General assumptions about arts audiences and the context of these festivals seem to be reflected in the data. Although we can’t say much about the programme of specific festivals (being anonymous) it is very interesting (for me at least!) to see A as, presumably, the most representative, in terms of age whereas D & E have a clearer skew towards specific age groups. The more detailed look at this we’ve done with the LAF data would also confirm this, in some cases being even more skewed towards even smaller age brackets, for example 45-50.) More broadly it doesn’t feel like a stretch to suggest that many festivals probably do succeed by virtue of attracting quite a specific demographic.

Using another quick Chi-square test (2×4 cells this time) all of the above differences between surveys and OA results were significant (at 0.05), with the exception of A, which is broadly what we’d expect but does make the case of A all the more surprising (or perhaps it’s just a fluke.) Non-significance is sometimes significant!

Class / Deprivation

In the final comparison, we are not really comparing equal variables, but they are at least, thematically similar. As mentioned, the survey collected postcodes, which were processed through Geoconvert (UK Data Service) to establish, among other things, the Indices of Multiple Deprivation (IMD) of survey respondents. We took the decile value (1-10) and took an average for each festival. Go and read more about IMD if you like, but the short version is that 1: most deprived and 10: least deprived, in this analysis. It takes into account a range of separate measures, though it is balanced more towards economic or financial factors. It is also a relative measure (eg: only works in relation to itself) rather than an absolute measure (eg: not a measure of your exact income for example).

OpenAudience provides comparison to Approximated Social Grade, which you might be familiar with as “ABC1C2DE” or the NRS (National Readership Survey) social grade. Grades are determined depending on the individuals job, so As and Bs are managerial and professional jobs, while Ds are semi-skilled or unskilled workers and Es are non-workers (including pensioners). Incidentally you might also be aware of NS-SEC (National Statistics Socio-economic Classification) but from what I can tell, this is also related to occupations and relies on a more detailed consideration under the Standard Occupational Classifications or ‘SOC codes’. See this Arts professional article by Dave O’Brien for more and an example of how you could phrase this question.

The above is about the best way I could think to display this all at once. The IMD is the value in the white box whereas the bars are proportions of the NRS classifications. There is no real direct comparison to be made here but you can see for example that where the IMD is lower (4.3) there are more people in DE occupations and vice versa. It seems that B, D and E were all relatively similar in terms of age and relatively similar here in terms of class/deprivation. It’s uncomfortable to say but A & C had the highest levels of ethnic diversity and also are slightly more deprived and working class here… but having said that, if an IMD of “5.0” is average for the whole country, we could also say that none of the festivals attract an especially deprived or working class audience. How does this compare to other events, other festivals, or other venues? How does it compare to the average values for Leicester as as whole?

I probably should have added this into the chart above but for reference the NRS across the whole UK was (in 2016): AB (27%) C1 (28%) C2 (20%) and DE (25%).

Oh go on then, let’s have another chart:

Some moreso than others, but my reading of this is that all of the festivals are relatively representative in terms of class. I am not really sure what a + or – 10% difference either way makes in terms of class, maybe it is more significant that I’m initially considering it to be. Equally the position of pensioners in grade ‘E’ with casual workers and the unemployed might not make this as straightforward as we like. Not debating that some pensioners have it very tough, but just because your annual income is quite low, you might still be comparatively asset rich (eg paid off mortgage).

And whatever you are looking at, always bear in mind the potential for a misleading Y-axis (relevant xkcd)

 

Conclusions

Should I use surveys or segmentation? Well, to say they are equivalent is not right but equally neither do they seem to give completely different results. Segmentation can be easy (thanks to the work of the wonderful OpenAudience) but then surveys do not have to be particularly complex either and at least you can start to triangulate your findings rather than view them only in isolation.

The underlying considerations end up at the type of reliability vs accuracy questions you can ask of many scientific experiments. You could conclude that segmentation is more reliable but less accurate while surveys are less reliable but more accurate. (valid/accurate)

https://i1.wp.com/www.reality-check-online.org/Images/ReliableValid.jpg

Segmentation usually refers to pretty concrete and official data and can make true, but very broad, observations. Surveys try to take what we hope is a ‘good quality’ sample and can make true but potentially narrow and therefore easy to skew observations. One is a cheap and low magnification pair of binoculars and the other is a fragile and easy to knock over microscope. Actually that’s not a great analogy for either but hopefully you get my exaggerated point.

There are similar faults that could befall either: maybe your box office postcodes include people who didn’t actually show up, or the person who booked them gave them to someone else as a present, or the person in the group who booked them lives in a different part of town to the others. Meanwhile, maybe a survey could be completed by a general passerby, or a volunteer at an event rather than an actual attendee, maybe the questions are difficult to understand and so on and so on…

For one thing, it would be interesting to know more about how accurate censuses actually are, and particularly whether countries with mandatory participation do any ‘better’ in this respect. (Australia: two weeks to complete or you are fined $180 per day for not participating and $1800 for giving misleading information… and they do it every 5 years?!).

Back in the UK specifically, they have run a linked but separate Census Quality Survey, where another 4,500 household interviews are carried out shortly after the main census to check the accuracy. In short, this found that the more subjective or memory-intensive questions were less reliable (What year did you last work? How many rooms does your house have? How is your general health?) than the more objective questions (Did you work last week? What is your date of birth?). Related to our investigation here, Date of Birth had an agreement rate of 98.4% and Ethnicity was 94.7%. So, pretty accurate, I would say, but it hasn’t stopped the Census coming under criticism for being too expensive, out of date and so on, but we are still getting at least one more go, perhaps the last one, in 2021. But people still lie on surveys! From a similar family to the survey or census, is the political poll, which despite getting a bit of a hammering in the media of late, is apparently still as accurate (if not moreso) than it’s ever been. Maybe we should be more sceptical of media reporting of surveys and polls than of the surveys and polls themselves.

Either surveys or segmentation can clearly benefit from a more open model for investigation. This includes actual open-source software but the more general principles of transparency that should be at the core of any scientific investigation. (Yet another plug for Open Data Kit) The reproducibility crisis in science is just one aspect where more open methods would help.

Surveys AND segmentation. Why not both?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.