Counted

[liv]

A couple of days ago, I got the chance to take part in an opinion poll. It's something I'd always wanted to do, cos I like being asked my opinion about stuff, and I sort of want my opinions to be part of the national scene.

So a chap showed up at my door about 7 pm, and said he was from Ipsos MORI. I have somewhat warm fuzzy feelings towards Ipsos MORI because shreena used to work for them, and I didn't have any very urgent plans for that evening, so I asked him in.

The whole thing was much less smooth and professional than I expected. The interviewer didn't entirely know what he was doing; he mentioned being recently trained. And he said it would take half an hour, but it was more like an hour, because he was chatty and because he was really slow finding the way round his survey program on his laptop. I quickly got the impression he had taken the job mainly because he's somewhat lonely, and this was rather confirmed when he told me a rambling story about how he'd lost his job as a postman because after going through cancer treatment he was no longer physically fit enough to do it. I'd just been teaching the med students about how to tell the difference between jaundice and variation in skin tone, and with this guy I wouldn't have been prepared to bet either way.

I think a lot of people would probably have been uncomfortable with the amount of personal stuff he was rambling on about, especially in the situation of a woman alone in the house being visited by a strange man. I mainly got a little extrovert buzz from the encounter, I'm interested in people and their stories. But I was still a little bit annoyed, because I felt he was imposing on my time and politeness a bit, and going beyond the bounds of what is expected as part of his job. I'm happy that he was a little chatty and personal, and didn't just robotically read through his questions, but I think he went somewhat too far. He also wasn't very good at avoiding influencing my answers or expressing his own opinions about some of the questions. He even stood in the doorway chatting about nothing much once the interview was over; I only managed to persuade him to leave because my mother phoned while he was still babbling about some benevolently gender-essentialist thing he'd read.

For some of the questions, he was allowed to show me the computer screen, and my goodness the program is clunky as all get-out! I understand that you can't really give your researchers fancy iPads or fast, shiny laptops, because you'd be setting them up as a target for mugging. But even a slow computer can have a decent UI, and this really didn't.

The questions were an utterly random mix, I had expected that a proper polling org would ask mostly political questions, but there were about equal numbers of political ones and more market-research type ones about brands and shopping habits. Having been exposed to qual research recently, I have ideas in my head about survey design and such, and I would say that the questions I faced were above average but not as great as I might have expected from an expert polling company. A few were confusing, such as using multiple negatives or very unclear definitions.

The computer completely choked on the concept that I am married, but live alone. The interviewer himself was totally fine about it, accepting that the problem was in his software, not my reality, but still. Silly narrow-minded demographic categories! So anyway, now my views about elected police commissioners and switching between banks are part of the pile of data that people will look at to decide things, and that's quite cool, even if it was a bit more of a waste of evening than I would really have liked.

Comments

[redbird]

I've done opinion polls, though not recently, but they were all either over the phone or online. Maybe American polling companies think (or know) that we won't invite random strangers in to ask those questions?

I am amused by their computer system choking on your living arrangements; even if it has never occurred to them that someone might choose that, they should have noticed that it happens, in part for job-related reasons.

[liv]

Yeah, I think a lot of people wouldn't let a stranger in. And the demographics of who is prepared to undergo an interview in those sorts of conditions must be really skewed.

My instincts are kind of weird; a few weeks ago I came home, as it happened in the middle of the day because my job is sometimes like that, to find a guy in a hi-vis vest poking about around my house. I challenged him, and was immediately suspicious when he told me that they'd detected a gas leak and it was a potential emergency, but not to worry, getting it fixed wouldn't cost me anything. That seemed like the perfect scam to get into someone's house: look officialish, mention a scary, panic-causing emergency, and in the next breath suddenly jump to talking about a free service. So I went through the rigmarole of calling the phone line to verify his ID, and he checked out, and I did eventually let him in. Thing is, those guys are legally authorized to force entry without a warrant, because preventing gas explosions is more important than the legal protections on private property. In fact, it wasn't so much getting verification from the phoneline that reassured me; after all, I'd called the number on his badge, the phoneline people could theoretically have been in on the scam. It was the fact that he didn't try to pressure me, hurry me up, or otherwise talk me out of gathering more info before I was prepared to let him in, and a scammer / burglar would have been more pushy.

Conversely, with the pollster, his relatively low level of competence made me more inclined to trust him. He introduced himself by saying "I'm from Ipsos MORI," with no indication of what that meant. Then I said he could do his survey but didn't invite him in, and he was really embarrassed and apologetic about the fact that he had to come in, he couldn't conduct the survey standing on the doorstep. The fact he didn't have a polished spiel about why it was totally in my interests to let him in made me less likely to think he was a scammer.

Yeah, of all people the biggest demographic researchers in the country should definitely know that there's such a thing as married couples who live apart! But I was really surprised in general by how poor their software was, in lots of ways.

[antisoppist]

I did one of these but the researcher was a woman in her fifties, who I felt comfortable about letting in (and seeing the state of the toy-ridden living room).

[liv]

That makes sense. I wonder what the gender balance is in that kind of job. I imagine it appeals to a lot of women, in that it's piecework with flexible hours, and primarily requires interpersonal skills. But at the same time, it involves walking about on your own through all kinds of neighbourhoods, and inviting yourself into strangers' homes, which a fair number of women wouldn't be willing to do for money.

[khalinche]

This more or less exactly describes my experience of being surveyed by IPSOS-MORI, down to feeling well-inclined towards them because Shreena used to work for them. My interviewer was not a postman recovering from cancer, but he was talkative to the point of me getting annoyed for him taking up my time to tell many stories about his life, rather than actually getting on with the survey. It was a long hour which was mostly me listening to his opinions and family anecdotes, and as you say, the software he had was made out of plywood and Formica. I'd probably still do it again, even so.

[liv]

*smile* Nice to know that your experience was similar. I would do it again, but in future I'll say no if I actually only have half an hour free rather than an hour. I generally feel more positive than uncomfortable about over-sharing strangers, which is good because mentioning that I work on cancer tends to attract a lot of those, as does being a religious leader. I guess I naively imagined that professional pollsters would be trained not to babble too much about personal stuff.

[zhelana]

Sorry your wish didn't end up awesome. But I am surprised the postmen have to be physically fit. I mean, they just drive around, right?

[evilsusan]

Ours cycle from the next village.

[liv]

In this country, the final part of postal delivery, actually distributing the letters to individual customers, is mostly a mixture of walking and cycling. I think maybe they drive in really remote areas, but not normally. The polling involves a fair amount of pounding the streets as well, but he could spread it out over the whole day, rather than needing to cover his area in just a couple of hours very early in the morning.

[lovingboth]

Lugging heavy bags around... Both here and in London, they're typically dropped off by van into caches, but 'from there on in, you walk'. Some here cycle, but they're the ones going to the extremities and there's still a significant amount of lifting and walking.

[pretty_panther]

I hate when people go off track too much like that. It becomes awkward. I don't like letting people like that in because I have social anxiety but my mum does it sometimes and the variation in how different people conduct themselves is very interesting.

[liv]

Gosh, social anxiety sounds like a really good reason not to take part in this kind of survey. Even for someone like me, being extremely outgoing and also a huge extrovert, it did feel a bit awkward to hear that much about his personal life and health information and so on.

[tig_b]

[soapbox starts]
I don't have warm fuzzy feelings about most market researchers - having battled too many times trying to explain why 'my' data and analyses were better indicators than any 'facts' from than self-selected respondents, poorly designed instruments with leading or biased questions, and excessive interviewer bias.

I do know that some of them do a good job, but s long as we have journalists and politicians who are almost innumerate the bad stuff will get headlines.

[soapbox ends]

[liv]

Definitely agree that innumerate politicians and irresponsible journalists are a big problem with how the results of polling and market research get reported to the public! But I'm intrigued how you ended up in a position to dispute market research methodology like that, with better data of your own?

[tig_b]

I worked as a Govt Statistician.

One of my first small projects for big boss was to draft a note, to be attached to all copies of a monthly publication, explaining why their 'survey' data shouldn't be used.

I had to visit a couple of the policy people to explain further, as they continued to use the figures for a while.

The 'survey' was of their own members, and with a response rate of between 8% and 12%, but they wrote it up as representing the UK business population.
Very naughty.

Official Govt statistics are not perfect, but we do publish the methods, coverage, etc. and the caveats.
and then we watch the spin drs and others producing catchy headlines - groan.

IPSOS- MORI

[(Anonymous)]

A. The mix of questions is a necessary feature of good surveys, to avoid leading questions.

B. I am mildly concerned by the apparent amateurish style, though your pollster was almost certainly genuine. Did you check his credentials?

Southernwood

[lovingboth]

The other reason for a mix of questions is that people can pay per question to have them included.

My experience is that it's much easier to say 'let me read it, yes, one, c, d, e, a..' It takes a fraction of the time and they're paid by completed interview, not per hour.

[liv]

That totally does make sense! This researcher was really conscientious, he said that some questions he was allowed to show me the computer screen, others he had to ask me verbally and I couldn't look at his answers. I agree it would have been quicker if I'd been able to read for myself, but might have lost some of the subtleties, I guess.

[shreena]

Heh. I'm amused that you were well disposed to them because I used to work there!

To respond, slightly haphazardly to a few things in your post/comments -

Actually, having been out with interviewers a few times (researchers do this from time to time to quality assure/gain experience/etc)and from having seen a LOT of fieldwork stats in my time, I'm not convinced that the demographic that let the interviewers in is that unrepresentative. A good random probability survey will get over 50% in response rate, sometimes as high as 65%. A surprisingly high number of people will let interviewers into their homes (though some will make the interviewer stand outside and do it like that.)

Interviewers are more likely to be female than male - the Ipsos MORI fieldforce is about 70% female.

The computer system is clunky but it's quite difficult to make it less so for a few reasons - one is that it's all quite complicated, the interviewers work from home and need to be able to upload the questionnaire scripts, the questionnaire scripts need to be able to export the data sensibly to be able to do quite high level things, the interviewers need to be able to update every day or so on how they're doing, how many interviews they've got/what demographics, etc; and the other thing is that a lot of the very good interviewers (the personable ones that get good results) are, frankly, poor at computers and need something simple.

The interviewer really shouldn't have been leading your answers/etc. If you were to write in/complain, he'd almost certainly get some further training/advice/help from his supervisor.

Questionnaire design - some of it will be poor work from Ipsos MORI staff, no doubt, but, in my experience, a lot of it is that clients are often wedded to asking questions in a particular way and simply won't listen to advice from the researchers.

On questions jumping around - it sounds like to me that you were doing an omnibus survey, i.e. one where different clients have bought questions. Those almost always sound a bit weird as you're going through, the ones which are all one client generally flow better.

One of the issues with interviewers generally is that they have to be personable and good with people to get the interviews but that sometimes leads to them being a bit inappropriately personal with interviewees. It's a fine line.

[liv]

Ooh, thank you for all this, it's really interesting to get an insider perspective. I'm somewhat pleased my guess was wrong about who lets interviewers in, people are more trusting than I cynically imagined.

What you say about the profile of interviewers does fit in with my experience. I can well imagine that the sort of person who wants a job like that is naturally personable but not necessarily so great at technical skills. And that the failure mode is that they influence the answers and / or overstep boundaries a bit; someone who was very good at being objective and managing the technology might well not build a rapport with the subjects, so they'd get less responses overall.

Also really interesting to know about omnibus surveys, particularly that that the clients might have designed the questions rather than Ipsos MORI people.