How can social desirability be avoided in UX research?
Social desirability bias occurs naturally in humans. It is instinctual to conform to socially acceptable behaviours that help us to be seen favourably. When undertaking UX research, this bias needs to be considered throughout the design, wording and execution. Below, we’ll cover simple steps that can help ensure that your resulting study data is valid and useful to you by reducing this bias.
Table of contents
1. What is social desirability?
2. What is the effect of social desirability on studies?
3. Steps to avoid social desirability bias
4. Record and measure objective data (or at least pretend to)
5. Final thoughts
1. What is social desirability?
We all want to be seen in a good light, whether that’s by downplaying our so-called “bad habits” or by exaggerating our good ones. This can be done by outright denying, claiming certain behaviours or simply giving incorrect answers. We might not want to admit that we don’t always regularly sort and recycle our household waste or file our taxes on time. We might want to show that we understand something or that a task was easy for us to undertake. It is most commonly seen around topics such as personal ability, personality, and socially taboo subjects. These are areas that can affect how others perceive us therefore we often try to make a good impression.
The social desirability bias is based on wanting to be seen as a good person. This is a natural part of living in a community with others where we rely on their support and connection. Also, it can be connected to our own self-esteem or wanting to present ourselves as a certain type of person. It can be an innate response to personal questions, rather than trying to mislead or provide false information. It is a normal human behaviour that people may not even be aware is affecting them at the time of undertaking your study.
2. What is the effect of social desirability on studies?
When conducting research, this cognitive bias can lead respondents to answer questions incorrectly, from outright lying to slightly inaccurate answers. This can pose a serious issue with the validity of your study.
You can also see the impact of social desirability in UX research. When asking how easy users found navigation, if they struggled with any features or how they feel about their experience, social desirability bias can show up by respondents not wanting to provide negative feedback. If they are not confident, are embarrassed that they struggled or didn’t understand what they were supposed to do, they may downplay these facts in their responses. Respondents may overstate certain favourable personality traits or not admit to others.
3. Steps to avoid social desirability bias
Some topics encourage social desirability bias in UX research more than others. Topics such as personality traits, income level, medical issues, eating behaviours and even intellectual achievements are especially prone.
No matter what the research topic, social desirability bias often results in research that doesn’t truly represent the problem, solution, or approach. Here are some of our top tips when it comes to mitigating the impact of social desirability bias in UX research.
3.1 Anonymous surveys
An easy way to remove the self-consciousness that leads to social desirability in UX research is to allow users to be anonymous when completing surveys. This gives respondents the freedom of knowing that their answers will not be connected to them. If revealing personal information, admitting not knowing something or behaviour that’s not acceptable, respondents will be more likely to provide honest answers if they know the answers won’t be recorded against their names.
3.2 Communicate that the results are anonymous and confidential
Making surveys anonymous isn’t enough, it must be clear to respondents that their answers will remain anonymous. Providing information on how anonymity is created is important to provide this reassurance. If receiving requests for studies via email or text message, people may think that their answers will be connected back to them via their email address or phone numbers. In face-to-face studies, there is also a presumption that they will be connected back to their answers.
Explaining to participants the process you undertake to keep results anonymous and confidential can help reduce this anxiety. This can include steps like assigning numbers to each participant as well as removing email addresses and identifying data. This can be included in surveys, communicated face-to-face or provided in links to privacy policies.
3.4 Keep the purpose of your survey vague
Social desirability bias in UX research can manifest itself in a few different ways. One of the most common is by trying to please or impress the authors of the study. By informing respondents of the intentions of the authors, you can lead them to provide certain answers.
Using an external agency like TestingTime to conduct research for you can help place a layer of anonymity between your organisation and respondents. This can increase response rates from a wider pool of users that your organisation may not be able to contact or elicit responses from.
3.5 Be careful with the wording of questions
How you present your questions can affect how your respondents answer them. This framing bias can easily be avoided by ensuring questions are asked in a non-leading way. Rather than presenting a preference or implying that there is a correct way of answering, ask questions in non-biased language. Asking “Did our amazing support team solve your problems today?” can lead respondents to answer more favourably than a simplified version of “How did you find our support team today?”
3.6 Self-completion process
There are many ways to conduct surveys or research and each has its own pros and cons. Sitting down with a researcher may allow you to dig deeper into respondents’ answers but this comes at a cost. It increases the likelihood of social desirability bias in UX research as respondents will feel the impact of the bias more having a person sitting asking them questions. This can be even more heightened where there is a power imbalance between the researcher and the respondent. This may be real or just perceived in the respondent’s mind.
Instead, conducting research in a self-completion mode allows respondents the privacy to answer questions in a more honest and full way. They can take their time to consider their responses which can lead to more nuance. They also do not have to worry about what the person sitting across from them thinks of them, dramatically reducing the chance of social desirability bias.
3.7 Online surveys
Using online surveys is the most common and cost-effective way of completing studies. It also comes with added benefits of reducing social desirability bias. The freedom and privacy of your respondents’ own homes (or wherever they choose to complete your study) help reduce the pressure to present a certain way of being to the researchers.
Using external agencies to create online surveys for you creates a multitude of benefits that help to reduce social desirability bias. It helps to achieve the self-completion goal and it allows surveys to be anonymous. It also means that you can keep the purpose of the study vague as the request for the response will come from a third party. They will also help ensure questions are worded to reduce the bias too.
3.8 Indirect versus direct questions
Direct questions seek to get answers from respondents about themselves. This could be by asking them about their experience, personality trait or behaviours. An indirect question would be to ask how a “typical user” would experience it, what traits they have or behaviour they would have. This allows respondents to make presumptions about how others would answer, rather than having to use their own personal experience.
To prevent social desirability bias in UX research, you can ask how a “typical user” would engage with a website or service. Some respondents may not wish to say that they don’t understand how to navigate or complete a necessary task. But by talking about other users, they can be more open and honest in their feedback.
3.9 Find the right participants
Finding the right profile – not just any profile. Finding the right participant profile will have a huge impact on the quality of data that you collect from your study. Using an external provider such as TestingTime allows you to perfectly specify the test user profile you need, including age, gender, language, place of residence, device skills, etc.
3.10 Empathise with participants
When conducting research in a lab or testing situation, participants may be nervous. It might be the first time they have been in such a location, and they will often want to make a good impression. Simple actions such as communicating open body language, showing genuine interest in your participants and removing physical barriers such as desks, screen can reduce the feelings of nervousness that lead to social desirability bias.
When conducting research via digital methods, you can build rapport in the introduction or through the messages you send requesting their responses. Letting respondents know that there are no wrong answers, that their personal information won’t be shared and that their responses will be anonymous helps to reduce any social desirability bias.
3.11 Covert observations
Covert observations are situations in which either the identity of the researcher, the nature of the research project or the fact that participants are being observed are concealed.
We all know how intimidated we can feel when someone is watching us complete a task. This could be filling survey questions, undertaking UX research or testing products. Depending on the type of research you are doing, it may be possible to give participants time alone – also known as ‘Unmoderated Testing’. This allows participants to complete the necessary tasks without feeling like the responses or actions are being watched and gives them more freedom to act naturally. They may not be comfortable asking questions but be okay with trying things out by themselves.
If you decide to use this method, encouraging participants to speak out loud while completing the task will help you collect relevant and important data about the study.
In certain situations ‘Moderated Testing’ where a real-time interview with a participant who is testing your product or service might be necessary, for example testing a product or feature that is technically difficult. This method can yield high-quality results, but mitigating social desirability bias will be more difficult
4. Record and measure objective data (or at least pretend to)
Social desirability can be measured directly (by questioning) and indirectly (without people knowing what they are about). Today, there are various measurement paradigms, which are presented below.
4.1 Bogus pipeline
A bogus pipeline or BPL is a research technique that attempts to discourage false responses from self-reported data through a pseudo lie detector machine that purports to highlight dishonest participant answers.
The Bogus Pipeline method aims to reduce social desirability bias by encouraging participants to respond truthfully, in the belief that dishonest answers will be exposed.
4.2 Physiological reactions
Physiological methods where data collection is done by biosensors is becoming more and more popular in UX research. Methods such as eye-tracking, facial recognition and electrocardiography allow for the measurement of reactions and emotions that are involuntarily produced and independent from the self-reporting.
These methods bypass social desirability bias by providing objective information, whilst simultaneously allowing participants to fully concentrate on the task.
4.3 Implicit associations
The Implicit Association Test or IAT measures attitudes and beliefs that people may be unwilling or unable to report. It is a controversial assessment intended to detect subconscious associations from past experiences that influence how we feel or behave towards something in the present.
The widely accessible computerised IAT requires participants to categorise two concepts with an attribute, as quickly as possible. For example, the concepts “young” or “old” with the attribute “foolish”. The IAT suggests that the quicker the concept and attribute are paired, the stronger the internal association.
We have already looked in detail about how and why Individuals may conceal or amend answers under the pressure of social desirability. In addition to this, participants hold a wealth of subconscious beliefs and attitudes that they are not aware of, but will affect their responses in research studies. With these factors in combination, asking participants to objectively state their deep-held associations is a near impossible feat.
As the IAT does not ask the participant a direct question on the topic, it enables the researchers to bypass many subconscious biases and receive more objective results from their research.
5. Final thoughts
When conducting studies, you want to ensure that you’re getting quality data back. Social desirability bias is one key factor that needs to be considered in your planning as respondents will often not be aware of it themselves. Considering things like wording, the environment that the testing takes place and how you reassure your respondents all helps to reduce the social desirability bias. These few simple actions are not difficult to execute but just simply take a little thought into the design of studies. Spending this time on the steps above can help to ensure that your study produces valid results. This way your study will be successful.
This article was originally written for the team at Testing Time and published here.