Lesson video

Hello, I'm Mrs. Lashley and I'm gonna be working with you as we go through the lesson today.

I'm really hoping you're ready to try your best and make the most of this lesson.

This lesson, we're going to be looking at evaluating various real life questionnaires and identifying bias in questioning.

This lesson is all about biassed questioning.

On the screen, there is a definition of bias.

It might be that you wish to pause the video so that you can read that and check that you're okay before we move on.

Questionnaire is a new word for us this lesson, and it's a set of written questions designed to collect information from a respondent.

So the first learning cycle is about evaluating questionnaires and then we're going to look at biassed questioning.

So let's make a start at evaluating these different questionnaires.

So sometimes as part of a data collection process in the statistical inquiry cycle, a series of questions are answered in the form of a questionnaire.

When the questionnaire is being written, part of the planning stage, some considerations need to be taken into account in order to retrieve reliable information.

So can you think of any reasons that may deter respondents from answering the questions both honestly and accurately? Pause the video whilst you think about that.

What reasons might stop you from answering a questionnaire honestly or accurately? And then when you're ready to check with me, press play.

So you may have thought of, and you may have got different to me, but you may have thought of things that are too personal.

You maybe are deterred from answering a question that's asking about personal data.

It might be if the question's too complicated and you don't understand it, or it's got words that you're not sure of, or a context that you're not sure of, then you probably are not going to answer it at all.

Or if you do, it's not gonna be particularly accurate for the data collection.

It might be that you're not able to.

The question just may not have been written well enough for you to be able to, not in terms of complication or a level of complexity, but in terms of the responses that they've provided.

It might not be relevant, it might just have no relevance to you as an individual.

And therefore, your answer, if you do answer something, wouldn't be accurate, wouldn't be honest.

So this question is part of the questionnaire collecting data about various elements of a school pupil's life.

So here is the question.

If you had to get to school by, and the options are bike, bus, train, aeroplane, or a Formula One car, which would you choose? You should consider annual rainfall, your budget, and whether you enjoy travelling with others.

So firstly, is this question too personal? No, not really.

Is the question too complicated? Yes, the considerations, so you should consider your annual rainfall, your budget, and whether you enjoy travelling with others, sort of complicates the question.

Without that part, the question's not complicated.

But with that additional bit, it might confuse some.

Does this question allow all people to answer, all respondents that potentially will be answering this questionnaire, can they all answer? Yes.

So although you may be selecting something you do not really want to due to the limited options.

So the accuracy of the data collected might be reduced because of the limitations on the options.

Is this question relevant to the targeted respondents? So this was a questionnaire about school pupils' life.

So the data is gonna be collected from school pupils.

So school pupil age.

No.

So although it might be cool to travel to school by aeroplane or a Formula One car, a school pupil cannot drive.

So a Formula One car only has a single seat and therefore, they couldn't come to school in a Formula One car.

So it's just absurd.

And they also do not have a budget to consider.

Alex, Jun, and Izzy have been asked to complete a questionnaire.

So remember, a questionnaire is a set of questions.

So this is just one of the questions.

In your opinion, do you not agree that people who don't like maths are not wrong? I'll read that again.

In your opinion, do you not agree that people who don't like maths are not wrong? Their responses are this.

Just confusion, question marks.

What does that even mean? And there are so many negatives.

So questions should avoid having multiple negatives.

The question becomes over complicated to understand, and the responses, therefore, might not be accurate, or respondents just might become despondent and choose not to finish your questionnaire altogether.

So if they meet a question that they just cannot understand, just seems nonsensical, then they might just decide enough with this and not finish the questionnaire.

For you to get honest and accurate data, you need them to finish your questionnaire.

So here is a check.

This question is part of a questionnaire taken when exiting a store.

Do you agree or disagree with this statement? I received brilliant customer service from all of the staff today.

Why might you disagree with this statement? So A, you've received poor customer service from one member of staff.

B, you didn't interact with all members of staff.

Or C, you didn't require any customer service.

So pause the video whilst you consider that and then when you're ready to check, press play.

Actually, all three of those are reasons you would disagree.

So because the question says I received brilliant customer service from all of the staff today, by putting the word all in there indicates that you have spoken to all of the staff, which probably you haven't.

So that's part B.

If you receive a poor customer service from just one member of staff, then you cannot agree that you've received brilliant from all of them.

So that's the A.

And part C, you didn't require any customer service.

On exiting the store, I've received brilliant customer service from all of the staff today.

If you didn't receive any customer service, then you can't agree with that question.

So it's a poorly written question for them to really gauge any understanding about customer service within the store.

Here's another one.

So this is another question from the questionnaire.

Does it not feel impossible to avoid buying unneeded items when you go shopping? What makes this question challenging to understand? So pause the video and then when you're ready to check, press play.

So too many negatives, which makes it a complicated to answer.

Does it not, there's a negative, feel impossible, impossible is sort of a negative connotation, to avoid buying unneeded, so things that you don't want, again, sort of negative, when you go shopping.

There's so many negatives within that question that you, it's just complicated to understand what actually the right answer is.

If you say yes, does that mean that it's easy to avoid buying unneeded items? Or if you say no, is it hard to, what does that actually mean? So continuing with the questionnaire that Alex, Jun, and Izzy were completing, here is another question.

How many pets do you have? And there's some response boxes.

Izzy says, "I can't answer this question." So why can Izzy not answer this question? So have a look at it.

Maybe you're like Izzy and you cannot answer this question.

And why is that? There's not an option for anyone that doesn't have pets.

So if you have no pets, what do you tick? And if this was a digital based form, this questionnaire was digital instead of paper-based and it was constructed that you couldn't move forward until you had responded to this question, then you're gonna get false information in your data collection because Izzy would have to select something.

So she might just randomly select one of them, even though that is not true for her.

So a poor written questionnaire can cause data that is poor.

So here's another one from that same questionnaire.

What is your favourite genre of film: horror, romance, sci-fi, or comedy? Alex says, "Action films are my favourite." And Jun says, "Period dramas are mine." So that's not one of the options on their question.

So again, if this was digital and they couldn't move forward until they give an answer, they're gonna have to lie.

They're gonna have to give an inaccurate response to this question.

So what response option could be added so that Alex and Jun both could answer this question? Can you think what that might be? Maybe you've seen one in a questionnaire yourself.

By giving an option of other, it then allows any that you've missed out can be ticked by the people that have a different answer to the ones that you chose.

You need to be mindful though with the use of other, because if you haven't given enough responses that are different to each other, it might mean that most of your data is other, which isn't very helpful when you're trying to do any analysis with that.

So make sure you've given enough responses.

And then other is for any sort of more obscure answers.

So here is a check.

Here is a question from a questionnaire.

How many flights have you taken this year? What should the response boxes be? So there are five response boxes and what should they be in order to mean that everybody can answer this question? Pause the video whilst you make a decision and then press play to check.

So you may not have got the exact same as what's on the screen, but the important things that you must have, you need to have a none or a zero option.

'cause maybe somebody hasn't taken any flights this year.

You need your options to not have overlaps.

So if you look, I've done one to five and then six to 15.

If you've taken five flights, you'd tick the one to five box.

And if you've done six, you'd do the six to 15 box.

There's no ambiguity about where you should tick and having a more than option because there are going to be a few individuals that have to fly frequently.

Maybe you are a flight attendant and therefore, you'll be on aeroplane most days.

So more than 30.

So we're up to the first task.

So each of these questions are part of different questionnaires.

Comment on whether the questions are well written or not.

Pause the video whilst you're working through question one.

And then when you press play, we'll go to question two.

So question two, quite an open task here.

Might be good for you to discuss if you've got a partner next to you or sat next to somebody or sat with people that you can discuss your questions.

So question 2A, write a poorly written question for a questionnaire explaining why it is a poorly written question.

And then part B, write a similar question in terms of its context and the intent that is improved and written well.

So pause the video.

You can have lots of fun and creative licence on A to make it a poor question, but you also need to explain and justify why it is poor.

Maybe you could annotate it.

And then part B, you're gonna rewrite that question.

So similar context, similar intent, but make it better and improved and written well.

Press play when you are ready to go through the answers.

So question one, we're gonna go through each question and we were discussing whether it was a well-written question for a questionnaire or not.

So part A, do you feel welcomed by every member of staff when you go to the cinema? So this question infers that you have interacted with every member of staff at the cinema, which is highly unlikely.

So the majority of respondents would have to say no despite the fact that they may have felt welcomed by the staff that they encountered.

So it's not a well-written question because you're going to end up with probably a lot of nos to this question, but that might not actually be a reflection of how welcoming the staff members at the cinema are.

Part B, does it not feel impossible to you that the moon landing never happened? So this one had multiple negatives, which makes understanding it very difficult.

It might put your respondents off or respond without thought and that will make the data that you are collecting less reliable.

Remember the whole point of a questionnaire is part of the collecting stage of a statistical inquiry cycle for you to then go on and process and present and interpret and come up with some conclusions and evaluations.

So if the data is unreliable because of poor questions, then the whole process sort of falls down.

Part C, in your opinion, how would you rate the speed and accuracy of your work? Amazing, excellent, great, good, bad, or poor.

So the response options are subjective and there are more positive options than negatives.

So how you rate your speed and accuracy is probably different to somebody else rating your speed and accuracy.

So it's a bit subjective and you're gonna expect a probably a positive skew, especially with there being more positive options than negative.

D, where do you get your information from for current events? The radio, TV, the newspaper, or internet.

So the options do not allow for other mediums. There isn't an other option.

So it's very restrictive in terms of what it is suggesting where you get them from.

Question two is where you had open licence.

So your poorly written question could be over complicated, it could have too many negatives, have response boxes that do not allow all respondents to answer.

Or it might be asking just a too personal question.

So you are gonna have to check through your question to, and you should have been explaining why it was a poorly written question.

Part B, your well-written question should not have any of those issues.

So it shouldn't be over complicated.

It shouldn't have too many negatives.

It shouldn't have response boxes that would miss out certain respondents.

So not having a zero, for example, or not having a more than option and it shouldn't be asking personal information.

So we're moving to the second learning cycle of the lesson, which is about biassed questioning.

So questionnaires are written for many different purposes and one of those is market research.

Some of you may go on to look at market research in your future careers.

So here is a question from a market research questionnaire.

The best feature of this product is the price, the ease of use, the style, and the effectiveness.

Who do you think wrote the question? The product's company, a neutral party, or a rival's products company.

So look at that question.

Who do you think is the author of that question in this questionnaire? So it's positive throughout this question.

The best feature, all of the options are positive things.

The price, the ease of use, its style, the effectiveness.

So the likelihood is that this is written by the company who made the product.

It has a bias towards positivity and getting the best feature.

There is not an option for anyone to disagree that this product is poor, that it doesn't do what it should do, that the price is too expensive or it's hard.

So there's an element of bias because of maybe who wrote the question.

So many factors cause bias in a single question or a whole questionnaire and some of the bias is completely unintentional, but other times, it might be a bit more intentionally added.

So one of them is it being phrased positively or conversely negatively rather than written from a neutral point of view.

So we just saw that on that market research question, it was completely positive.

So that's one thing, but the other thing is the response options can be unbalanced.

For example, how was the service you received today? The question itself is not positive or negative.

It's asking you how was your service that you received today? The options are okay, good, amazing, unforgettable.

And that's where there is an element of bias because there's nowhere there to say it was poor, it was terrible.

The words they've used also are subjective.

So they're not very quantifiable.

So somebody's okay could be equivalent to somebody else's amazing.

But the idea of this is that actually there's a bias to it because all of the responses are positive, it's unbalanced.

Also a question like this, which would make respondents agree with a statement regardless whether they agree or disagree.

So will you continue to shop in our amazing store? If you say yes, you've agreed it's an amazing store.

If you say no, you've still agreed it's an amazing store because the question is about continuing to shop and they have phrased it as an amazing store.

The way that the question has been phrased has been phrased very positively even if you say no to that question.

So here is an example of biassed or leading question.

Reading is proven to make you smarter.

Do you read books? So the respondent might not be truthful in their answer when a question's been framed like this because if you say no, you've just sort of suggested that maybe you're less smart than somebody that does read books because they gave you a statement, a fact of reading is proven to make you smarter.

So this might lead people to lie because they are embarrassed by their response.

So what are the implications of respondents not being truthful when completing surveys or questionnaires? Pause the video and think about that.

What implications are there if the responses you receive in your data collection are not truthful responses? The data that is collected is just not reliable.

The bias is introduced into the responses and it may lead to false conclusions.

So the question that we can still see on the screen, if lots of people say yes, you might think, "Oh, great, the population is reading lots of books", but they might have said yes because they didn't want to say no.

And that doesn't mean that the yes is a truthful response.

So another place where unreliable data can be collected is when the questions are about personal information.

And this can lead to a bias in the data collected.

So in some circumstances, personal data is necessary and relevant for your data collection, but it needs to be considered in the planning stage of the statistical inquiry cycle whether it can be avoided or reduced.

So it does need to be really considered when you are planning your statistical inquiry whether it can be avoided or reduced.

But sometimes it is a necessary factor.

So what personal information may people be unwilling to share in a questionnaire? So pause and think about that.

When we talk about a question being too personal, what might that be about? So I've come up with these eight, you may have had different ones.

So age.

People tend to, especially as you get older, might be less likely to be happy to share their age.

Younger toddlers and sort of infants are quite happy to tell you how old they are.

But on the most part, as people get older, they're less likely to be willing to share.

Heights.

Some people feel quite conscious of their heights.

Weight is another one.

Gender, religious beliefs, political views, relationship status, whether you're married or divorced or single, and sexual orientation.

All of these things are personal to individuals which they may not wish to disclose in a questionnaire, but it might be that you need them to disclose it.

And so we need to think about how we collect that personal information.

So data's gonna be collected about a new mattress and the perceived comfort level by a sample of users.

Within the questionnaire, it's important to know the weight of the user.

So if the question was how heavy are you, what issues might there be? So think about that.

So because it's a new mattress and the perceived comfort level, the company wants to work out whether maybe heavier individuals find it more comfortable or less heavy find it more comfortable.

So actually knowing the weight of the users is an important factor.

But we know that weight is personal.

So some users may respond with answers like very or not very heavy, which are subjective.

It's not very useful for you as a data collector and to process that data.

So a question of how heavy are you just open text might get you some quite vague and subjective responses.

So that's an issue.

Some users may not respond to the question and feel offended to be asked about their weight.

So it could damage the brand's reputation.

So this is another thing that you need to be considering when you're writing a questionnaire is what are the respondents going to perceive about the brand that is collecting the data? Some users may not know their weight or lie about their weight so that reliability of the data then comes into question.

So if we reworded that question to how much do you weigh in kilogrammes, that avoids the very heavy, not that heavy response, but what issues might there still be? So again, pause the video, think about that, discuss it if you can, and then when you're ready, press play.

So it still might be that people won't want to respond, feel offended, and damage the brand's reputation.

So there's still an element because it is about personal information that some individuals may dislike that question completely, even with the addition of how much you weigh in kilogrammes.

Some users still might not know their weight or lie about them and they're more likely to lie at this point now that there's also a unit.

They can't just say, "I'm heavy." So how could the question be written to collect the data about the weight 'cause we know we need to or it would really help us with our data collection to know the weight of the users and their perceived comfort.

Can you think of a way to do that? Maybe you came up with this.

If we used grouped response boxes such as 50 to 65 kilogrammes, so we've still got our unit of measure, this reduces the amount of personal information that's actually being recorded.

And it would also allow the responders that don't know their weight to estimate their weight.

So it's much more likely on a personal question that if you've got grouped response boxes, that you're gonna get truthful responses because you don't actually know their weight, you just know the sort of range that their weight is within.

So here's a check.

A group of pupils are asked to fill in a questionnaire.

One of the questions asks how tall are you? One pupil's response is I'm very tall.

Another pupil's response is I'm taller than Sofia, but shorter than Lucas.

Why are these responses unhelpful for your data collection? Pause the video and think about that.

Come up with different reasons and press play when you're ready to check.

They're unhelpful because they're relative to other pupils and those pupils may not be known to the research team.

So I'm taller than Sofia, well, how tall is Sofia? I'm shorter than Lucas.

How tall is Lucas? And that I'm very tall, again, is just subjective.

Why might pupils want to keep their answers vague? So pause the video and think about why they may not have actually said I'm one metre 52 centimetres.

Press play when you're ready to check.

So they may feel conscious of their height and not wish to disclose it in the questionnaire.

So it does come under one of those personal factors.

So a grouped response box here would be really helpful because they could tick the group they're within rather than giving an actual value.

So onto this last task, task B.

So there are three questions to this task.

So question one is on the screen.

So for each question, explain why the question is biassed or why the data collected may be unreliable.

So pause the video, read the question, think about where the bias might be coming in or why the data collected could be unreliable.

Press play when you're ready for question two and three.

So question two and three are both on the screen now.

Question two, Oakfield theme park is striving for a five star customer satisfaction rating.

How satisfied are you with the theme park? Extremely satisfied, very satisfied, satisfied, less than satisfied, not applicable.

So part A is comment on the issues with this question and part B is to rewrite a question that Oakfield theme park could use to gauge customer satisfaction.

And then question three, write your own biassed question explaining what elements of it make it biassed.

And part B, write a similar question without the bias.

So pause the video whilst you work through two and three.

When you press play, we're gonna go through our answers.

So question one, you needed to explain why they were biassed or why the data might be unreliable.

So question A is how old are you? So people are often reluctant to share their age.

It's deemed to be very personal.

So the data collected may be unreliable.

Part B, how old do you feel? So this could lead to responses such as very old, but it's also very subjective.

So it's unreliable in terms of the data that you collect.

Part C, what do you dislike about the shampoo? Nothing.

It smells amazing.

It is affordable.

My hair is left smooth.

So this is a very leading question as it does not allow the respondent to give any criticism of the shampoo at all.

So it's a poorly written question.

It says what do you dislike about the shampoo? And really the only answer you can respond there is nothing.

The others don't make sense as a response to the question.

And so it leads you to say that there is nothing to dislike on the shampoo.

Part D, Taylor Swift is the best music artist to have ever set foot on the planet.

Do you like Taylor Swift? Again, the respondent might feel less likely to disagree with the statement because of the leading statement putting an opinion in the mind.

So your data that you're gonna collect may have a bias and be unreliable.

Question two, you need to comment on the issues with the question from Oakfield theme park.

So the responses do not actually include any star ratings.

It says that they're striving for a five star customer satisfaction rating.

And then it was talking about how satisfied.

The responses are unbalanced, there's more positive satisfaction there.

Extremely satisfied, very satisfied, satisfied, and then there's less than satisfied or not applicable.

And the question is also leading.

It's telling you that's what they're striving to achieve.

And so people might be more likely to give them a positive answer than a negative.

So rewriting the question, this is one response that you could have given.

How would you rate your satisfaction with the theme park: naught stars, one star, two star, three star, four star, or five stars.

It doesn't have that leading nature of saying we're striving to get five star customer satisfaction.

It allows them to rate it poorly or positively depending on how they actually felt with the theme park.

Question three, you needed to write your own biassed question and explain what elements were making it biassed.

And then you needed to write a similar question.

So the same sort of context or intent, but having it without that bias.

This was open licence for you to be creative.

So I haven't given you a sort of example question here, but your question may be biassed because of the response options, be it unbalanced rather than objective.

So they might be more positively phrased rather than a balance.

It could be that you're asking a question that's gonna be too personal in its nature.

And so that would add bias to the data collected 'cause people might lie and it might become unreliable.

It might be biassed by use of coercive language.

So the way that you phrased it might lead you to get biassed data.

So your question should not be leading or too personal.

So your rewritten question without bias should not have any of those poor factors to it.

So in summary of today's lesson about biassed questioning.

So questionnaires are used in a variety of different jobs and situations and sometimes that leads to an element of bias.

On designing the questions for a questionnaire, the data that is being collected needs to be purposeful.

So you're not just gonna ask questions that are not relevant to the respondent.

You've got to think about why you are asking the question, what data you are hoping to collect, and design the questions so that you're going to get the data that you are hoping to collect.

The questions should not be biassed and questions that are overcomplicated, too personal, leading or restrict the respondent or are not relevant can potentially reduce your completion rates, give negative impressions, or introduce bias into your responses.

And then the reliability of the data and the conclusions you can draw from that will be limited.

Really well done today.

I hope you've enjoyed that lesson.

Next time you are looking at any type of questionnaire, really make sure you are thinking and being analytical about whether it's a well-written questionnaire.

I look forward to working with you again in the future.