Why should we care about tracking responses by subgroup?
Often when you are running a study, you may be interested in getting a certain proportion of responses from a certain group. Your product or service likely has usage by a variety of groups, and it can be helpful to make sure you get input from the most relevant ones, so that the insights you’re drawing aren’t skewed to one population.
For example - you may want a certain proportion of respondents to be split between:
Men and Women
Three different age groups
People who work at small businesses vs. large enterprises
Pre-screening participants
At a high level, there are two ways we can “pre-screen” participants of Outset interviews before they even get the link to the interview.
If you’re recruiting outside of Outset - you can pre-screen participants before they even get to the Outset survey -
e.g., we know these people are customers of product X because we have internal customer data
e.g., we know these participants are from Japan because we’re working with a panel that specializes in recruiting Japanese participants
e.g., we have participants take a screening survey with another survey platform, and then direct qualified participants to Outset
If you’re recruiting with Outset - our built-in survey panels (Prolific and User Interviews) have various filters that can be applied to participants to “pre-screen” participants so that only people who meet those criteria according to data they’ve filled out on the panel platforms will be directed to the Outset interview -
e.g., Women, in the United States, Aged 40+
e.g., Men or Women, Asian ethnicity, who currently have children aged 4-12
Screening participants
Then, once participants are directed to the Outset interview, they complete a screener survey (if you decide to set one up for your interview).
With that screener survey, you can set up additional questions that you weren’t able to answer with your pre-screening. This could be either because you didn’t have the information available from your other panel (if recruiting outside of Outset), or because Prolific or User Interviews don’t include those data fields in information they track about participants (if recruiting with Outset).
e.g., Have you stayed at a Hyatt resort in the past 12 months?
e.g., Are you currently considering refinancing a loan?
💡TIPS
We generally recommend using pre-screening methods if available, before participants take your screener survey. With Prolific, you’re charged a small fee for each participant who takes the screener survey and doesn’t qualify, so if you’re able to exclude them with pre-screening filters then you reduce the total recruiting cost of your survey.
Methods for recruiting desired number of responses by subgroup
There are two main methods to doing this:
METHOD #1 - Create duplicate studies, and target each study to specific subgroups using pre-screening filters
METHOD #2 - Use one study and set up a quota within your study so that you keep recruiting participants until you get a set number of participants in each subgroup. If a subgroup's quota has already been filled, anyone else who takes your screener survey and shows up as that subgroup will be screened out as "over quota".
| Method #1: | Method #2: |
Example | Study #1 - filters to women only using pre-screening, N=50
Duplicate Study #2 - filters to men only using pre-screening, N=50 | One study - with N=100 and a split of 50% women, 50% men via quotas |
✅ Pros | Studies are already pre-segmented to your subgroups of men vs. women, so it’s easy to review differences between the two groups without any custom report segments
| No need to duplicate the study |
❌ Cons | If you have many sub-groups (e.g., 2 genders x 4 age groups, you have to set up 8 duplicated studies) | If for some reason one of your sub-groups is very low incidence, then the study will stay live and people will keep taking your screener survey until you find enough qualified candidates |
One important factor that could weigh into your decision to use method #2 (quotas) is which panel you're recruiting with:
With Prolific - recruiting with quotas could become costly if you have a challenging-to-recruit-for subgroup because each participant who takes an interview and does not qualify charges a small fee to your account
With User Interviews, there is no fee for screened-out candidates, but the survey may take quite a while to fill, and people will unnecessarily keep taking your survey screener if they're part of a subgroup that's over quota
In the example above, many topics wouldn't have a problem filling quota for 50% men and 50% women. But let's say your research topic is interviewing Lululemon superfans...if you have a quota with a subgroup of men over age 60, you might be waiting a while for that subgroup's quota to fill!
Ultimately, you should pick the method that best suits your budget, study implementation, and analysis needs. Feel free to reach out to [email protected] if you have any questions about which method may be right for your study.
📖 READ MORE!