Lessons From My Online Dating Experiment
by Evelyn Gosnell | Jul 16, 2015 | attraction, Behavioral Economics, Experiments, Online Dating
“Speak with data,” ran the company mantra at an apparel company where I used to work. Any time I suggested a new idea or proposal to my boss, he would always cut me off by saying, “Are you sure customers want this? Go run a survey.” As a result of this, we became a very survey-happy culture, as many companies today are. Our customers were surveyed all the time from topics such as interest in new products and services to the price they’d be willing to pay for them.
Theoretically, those surveys were not a bad idea. We all know that having more background information before you make a decision is a good thing. From the behavioral standpoint, however, there are a lot of issues with surveys that make their results less reliable than we might like.
To illustrate let’s look at a fun example from real life (we’ll link back to how this is relevant in a company setting shortly). I often hear male friends say they prefer women who don’t wear makeup. Hearing this never made sense to me as it was inconsistent with my experience. So, in the spirit of speaking with data, I ran a quick survey on Qualtrics with about 60 men.
Here are the results: 78% of men said they preferred women with the no-makeup look, vs. 28% who preferred makeup. Pretty strong opinion! In a company context, product managers would take that as a clear preference. They’d run off and make product changes accordingly.
But let’s slow down for a second. The basic challenge with this type of survey is that it assumes that people’s preferences are fixed, and that they are able to accurately predict them. We know from behavioral science that this is definitely not the case. The better way to understand behavior is to test it, not to ask about it.
So that’s what I did. Pushing my discomfort with the idea aside, I created two profiles for myself on the online dating website OKCupid. For one profile photo, I wore makeup, and for the other, I wore none. Everything else about the two profiles was exactly the same.
In addition, I did not respond to any messages on either account, and I also kept the amount of time logged into each account the same.
Drumroll for the results? After one week , the profile with makeup received messages from 34% more men. So while my survey showed that men say they preferred women without makeup, they are actually more likely to contact a woman who is wearing makeup than one who isn’t. And that’s a truer gauge of attraction, isn’t it? (Oh, for a funny side note: 10 of the men messaged both profiles. And of these 10, 7 sent the same message to both. Not that I blame them. Christian Rudder, founder of OKCupid actually writes in Dataclysm that that that’s a good strategy. I wrote about his other findings before, but in case you missed it, find it here).
So what’s the point of all this? Run experiments instead of surveys when you can! Surveys are not all bad; they have their place, but in general you’ll get better insights when you test true behavior instead of stated preferences. This is especially important when you are basing major business decisions on these results.
Here are some quick tips on experiments:
- Learn from others. This is a great HBR article that summarizes the how-to’s of running experiments in companies. It covers the basics of setting up a control group and a feedback mechanism, the two cornerstones of an experiment.
- Keep it simple. Running an experiment does not have to be a massive endeavor. Even splitting your email list and sending different content to each group is an experiment. Make sure you split the list randomly of course.
So yes, speak with data. Just make sure you’re looking at the right data!
Originally published at behaviorly.com on July 16, 2015.
