What tools does Upworthy employ to test its headlines?
In his answer to What advice does Upworthy have for making viral videos?, Adam Mordecai mentions that he writes 25 headlines for every post, and that his testing tools help focus his write ups. Since he's A/B testing headlines, each headline is likely different enough from the others, which suggests that he might adapt a different pattern for each. What are those patterns?
8 Answers
We have a custom click testing system that I call the magic unicorn box which we built internally. I can't say how it works. All I can say is that I put my headlines in the box, and the magical scores come out, and then I make decisions based on that.
The reason we write 25 headlines is that it's an old strategy from my boss's old gig, The Onion. At the Onion, they would "crap out 25 headlines" as fast as possible. Then in the pitch room, they'd assume that 20 were crap, and the remaining 5 had an opportunity to be an article. Then once they had their headline, they'd finish their piece.
The reason it's always 25, no less, is that it forces you to think waaaay outside the box when writing. You get desperate around headline 21, and do something so out of left field that it's not the typical headline. The key is to not overthink your headlines and make every sentence perfect. Remember 20 will be crap. So just get them all out.
As for patterns to my headline writing, there isn't really a distinct one. The following tips are for people who want to reach the widest audience possible. This can dilute the strength of your voice, so you have to work twice as hard not to sound bland, but if you care about a wide audience it works pretty well. The main structural rules for headlines in my mind are as follows:
Again, these rules are for getting the widest audience possible. If you have a targeted audience that might not like these rules, then don't ruin their experience by being like us. Our voice is not for everyone.
Also, just FYI, I've broken every single rule on this list at some point. I learned by experimenting and failing. So test on your audience, have fun with it, assume you will make mistakes, and win the Interwebs.
And write 25 headlines. (I can do it in 15 minutes now that I've practiced enough.)
UPDATE: An important factor that I seemingly left out it that your content should not ever be like this. Your content should challenge people, make them think, and be more than just clickbait. However your Headlines will be more effective with these rules.
Voiceiness in your content is actually really important. People refrain from sharing headlines and images that make them uncomfortable. However they will share content that does. As long as they won't be judged for the packaging on Facebook.
So make your content safe for Facebook to share, but live on the edge with the content you present on the other side of the click.
See also: Adam Mordecai's answer to What advice does Upworthy have for making viral videos?
The reason we write 25 headlines is that it's an old strategy from my boss's old gig, The Onion. At the Onion, they would "crap out 25 headlines" as fast as possible. Then in the pitch room, they'd assume that 20 were crap, and the remaining 5 had an opportunity to be an article. Then once they had their headline, they'd finish their piece.
The reason it's always 25, no less, is that it forces you to think waaaay outside the box when writing. You get desperate around headline 21, and do something so out of left field that it's not the typical headline. The key is to not overthink your headlines and make every sentence perfect. Remember 20 will be crap. So just get them all out.
As for patterns to my headline writing, there isn't really a distinct one. The following tips are for people who want to reach the widest audience possible. This can dilute the strength of your voice, so you have to work twice as hard not to sound bland, but if you care about a wide audience it works pretty well. The main structural rules for headlines in my mind are as follows:
- Tell a story in your headline, but don't give it all away. (This is what makes us so successful, though some people really hate it.)
- Give away as much as humanly possible. The more transparent you can be with the subject matter the better.
- If you've got some heroes and villains, play them off each other.
- To optimize shareability you want to make sure everyone can feel comfortable sharing it. Think to yourself, "Would my mom share this headline?" If not, do something different. Unless you are only targeting a subgroup and don't care about shareability.
- Don't depress people so much that they want to give up on humanity. Negative headlines breeds negative shares.
- Don't curse in your headlines. Moms hate it (and are the biggest sharers on the internet by a significant margin. Mothers With Children Under 5 Most Active on Social Media [STUDY]
- Don't make people take positions they might be uncomfortable with. For example, "I Really Hate All White People" is going to not get shared, whereas, "An Open Letter To Pasty People" is far less hostile and more likely to get shared.
- Don't use terms that overwhelm, polarize or bore people. I never use Social Security, The Environment, Immigration, Democrats, Republicans, Medicare, Racist, Bigot, etc... You can talk about issues without giving away what they are. Most people aren't going to want to look at a Immigration video. Once they get to your site and hit play, they may reconsider. (Though immigration is particularly challenging. )
- Don't be shrill and judgy, let the facts speak for themselves. Anytime I've made that mistake, the content dies a horrible death. As an example, when Todd Akin came out with the cartoonishly awful idea that women's bodies stop them from getting pregnant from rape, I started with the headline, "Meet Todd Akin. He's A Horrible Human Being. Share This So Everyone Knows." No one clicked or shared. When I changed it to "A Congressman On A Science Committee Doesn't Understand How Science Works" it did waaaay better with people across the political spectrum. Because people didn't have to be afraid of sounding partisan.
- Don't oversell. We've worked hard to tone ourselves down, as occasionally our headlines would veer towards THE BIGGEST THING EVER, when it was actually THE PRETTY INTERESTING BUT NOT BIGGEST THING EVER. It's not worth dragging people to your site if they feel ripped off after they get there.
- Don't be afraid to talk like a human. People like human conversation.
- Don't make obscure pop culture references. 90% of the American audience has never seen half the shows you are referencing. Instead of Jennifer Lawrence Talks About What It's Like To Be Judged, it's "That Lady From The Hunger Games" Nobody knows who your favorite character is played by.
- Always test. No matter how clever you think your headline is.
- Speaking of which, don't be too clever. People won't get it and they won't click and then you lose.
- Have fun. Dry headlines bore people.
- Always write 25 headlines. Or the terrorists will win.
- BONUS UPDATE: Try descriptive headlines. We've found that they usually can tie vague ones. There's nothing I love more than giving it all away.
Again, these rules are for getting the widest audience possible. If you have a targeted audience that might not like these rules, then don't ruin their experience by being like us. Our voice is not for everyone.
Also, just FYI, I've broken every single rule on this list at some point. I learned by experimenting and failing. So test on your audience, have fun with it, assume you will make mistakes, and win the Interwebs.
And write 25 headlines. (I can do it in 15 minutes now that I've practiced enough.)
UPDATE: An important factor that I seemingly left out it that your content should not ever be like this. Your content should challenge people, make them think, and be more than just clickbait. However your Headlines will be more effective with these rules.
Voiceiness in your content is actually really important. People refrain from sharing headlines and images that make them uncomfortable. However they will share content that does. As long as they won't be judged for the packaging on Facebook.
So make your content safe for Facebook to share, but live on the edge with the content you present on the other side of the click.
See also: Adam Mordecai's answer to What advice does Upworthy have for making viral videos?
Jeff Lesser,
Views
This is complete speculation but given the answer from Adam Mordecai I would assume that their magic unicorn system that they built utilizes MAB testing a.k.a. Multi-armed bandit testing to serve the winning experiment more until it reaches a tipping point and completely takes over.
Kimberly Munoz,
Views
Although Adam has a great answer above, I also want to point to his answer about how Upworthy tests their headlines:
What are good ways to run an A/B test for a page title on a social media site?
tl;dr: They have a secret proprietary system, but there's other tools to test your headlines.
What are good ways to run an A/B test for a page title on a social media site?
tl;dr: They have a secret proprietary system, but there's other tools to test your headlines.
Shana Carp,
Views
I'm repeating something I said elsewhere
They're posting 25 times-and you couldn't run the same test even if you wanted to.
If you scrape out every link on upworthy, you'll see that they are appending numbers to some links and not others. Each number represents a title and a different set of OG metatags.
They then seed with partners, the core of people who hand out on the site, and some paid support.
Once an answer is reached - they switch over all the links inside facebook.
The reason they are able to do this without penalties in the newsfeed for duplicate linking/repeat posting of the same material : they have scale, and it is unlikely they're using a standard AB test. They're probably using a MultiArmed Bandit, and switching over before statistical significance is reached.
As to why I know this - my startup, BayesianWitch, actually offers a title testing tool that have about 85% of Upworthy's Black Box (aka the Unicorn Box). I did research into how Upworthy's title testing works in order to do positioning, and because I found their testing interesting.
If you want to try the plugin, you can sign up here:
Bayesianwitch for Wordpress
Not On wordpress or want to do something different? - We also have an API here:
API Documentation
They're posting 25 times-and you couldn't run the same test even if you wanted to.
If you scrape out every link on upworthy, you'll see that they are appending numbers to some links and not others. Each number represents a title and a different set of OG metatags.
They then seed with partners, the core of people who hand out on the site, and some paid support.
Once an answer is reached - they switch over all the links inside facebook.
The reason they are able to do this without penalties in the newsfeed for duplicate linking/repeat posting of the same material : they have scale, and it is unlikely they're using a standard AB test. They're probably using a MultiArmed Bandit, and switching over before statistical significance is reached.
As to why I know this - my startup, BayesianWitch, actually offers a title testing tool that have about 85% of Upworthy's Black Box (aka the Unicorn Box). I did research into how Upworthy's title testing works in order to do positioning, and because I found their testing interesting.
If you want to try the plugin, you can sign up here:
Bayesianwitch for Wordpress
Not On wordpress or want to do something different? - We also have an API here:
API Documentation
Stephane Allard,
Views
There are a few other media companies using predictive analytics to help their journalists write great headlines. Buzzfeed of course, but Melty in France too.
We're currently working on a predictive solution (http://www.wisemetrics.c
Testing headlines,images and content in general before publishing on social accounts generates great results, you should definitely do it!
We're currently working on a predictive solution (http://www.wisemetrics.c
Testing headlines,images and content in general before publishing on social accounts generates great results, you should definitely do it!
