[This blog post is written by BizLab Fellow Cynthia Yue, sharing the results from her Summer 2018 experiment.]
As outlined in my last blog post, I spent my summer at WBUR following an iterative design process to design an experience that compels users to purchase books. In this blog post, I discuss the findings from an experiment to determine if this experience drives affiliate marketing revenue.
Experiment Design
After designing a new user experience for a digital bookshelf, featuring the books WBUR’s programming talks about on-air, in order to evaluate its effectiveness as a revenue channel, I compared how it performed in comparison to WBUR’s other webpages that have book content. To do this, I made some minor improvements to two additional webpage types by adding book imagery and more links to Amazon. Then I launched Facebook ads to all three types of pages, targeted at people who like WBUR and books.
Condition 1 – Articles about Books
Anytime a book is mentioned on-air or an author is interviewed, there is an article on our website with this content. I selected four of our recent articles about books and added some additional book imagery and links to Amazon, and created a Facebook ad that appeared similar to a news article in a Facebook newsfeed.
| Before | After | Facebook Ad |
|---|---|---|
 |
 |
 |
Condition 2 – The Summer Reading List
The On Point team published a Summer Reading List in June 2018 that contained several lists of books recommended by media personalities. We had seen this page get notable traffic in June, and because it has many links to books, it is the most similar page to the bookshelf, but it lacks the rich content and additional information shown on the bookself page. Before launching Facebook ads for this page, I added more book imagery to this page as well.
| Before | After | Facebook Ad |
|---|---|---|
 |
 |
 |
Condition 3 – The WBUR Bookshelf
The design of the bookshelf was based off of several phases of user feedback. The main inspiration for the bookself design was to replicate the experience of browsing for books that were hand selected by hosts of shows. The bookself we launched is a high-fidelity prototype in the sense that none of the content is dynamic. This is important to note because to launch the bookself in a sustainable way, we will need to invest time and resources into building it from scratch.
| Facebook Ad 1 | Facebook Ad 2 | Facebook Ad 3 |
|---|---|---|
 |
 |
 |
We ran three variations of Facebook ads for the bookshelf, to maximize traffic.
Results
The Facebook ads pointing to each page type ran for approximately 7 days, and during that time the different pages got thousands of pageviews. We anticipated, because the bookshelf had the features users were looking for, that it would get the most traffic, interaction, and revenue. This is not what happened. While the bookshelf got the most traffic, indicating interest, it did not generate as many click throughs or purchases as we’d hoped for.
| Condition | Pageviews | Clicks Through to Amazon | Amazon Items Purchased | WBUR Earnings |
|---|---|---|---|---|
|
Articles about a book |
1,611 pageviews | 25 clicks (1.55% CTR) |
2 items (8.0% conversion rate) |
$2.35 |
|
Summer Reading List |
2,664 pageviews | 1,585 clicks (59.5% CTR) |
90 items (5.7% conversion rate) |
$100.08 |
| WBUR Bookshelf | 4,247 pageviews | 468 clicks (11.02% CTR) |
10 items (2.1% conversion rate) |
$9.11 |
The standout winner in this comparison of the page types is the Summer Reading List page. This page had an extremely high click through rate over to amazon (59.5%) and the conversion of those clicks into purchases was also higher than the bookshelf (5.7%). In our 7 day experiment, the list earned WBUR $100, while the bookshelf earned $9.11 and the book articles earned $2.35. The conversion rate is significantly higher (at a 99% confidence interval) when compared to the bookshelf.
Discussion and Recommended Next Steps
These results have generated a lot of new ideas for where to take this concept and how to translate this into actionable next steps for WBUR. As a designer, I have many ideas for how to improve upon the bookshelf design. I’m also a practitioner and understand the realities of limited time and budget, so want to balance my recommendations for the best design direction versus the most practical next step for a public radio station.
Beyond this experiment, there is a larger context: the WBUR site has hundreds of links to purchase books and over the years the institutional knowledge of how to create Amazon affiliate tags had been lost. When I started my fellowship, I worked with WBUR staff to ensure affiliate tags were on all of the Amazon books links found on our site. This change in workflow impacted many pages, and will continue to have an impact. In the last two months since I’ve joined BizLab, WBUR raised $2,488.41 through the Amazon Affiliate’s program! Average earnings before I arrived were $410 a month.
What should WBUR do going forward, based on what we learned from the prototype, the experiment, and the change in tagging?
Here are my recommendations on how we can improve.
-
We know that the embedded Amazon search widget (that allows searches for Amazon books on our site) is beneficial. In interviews, our users appreciated knowing that they can support their favorite radio station while shopping on Amazon, and this mitigated their hesitation over supporting Amazon instead of a local retailer. The metrics from our bookshelf page confirm active engagement with the search widget — it accounts for 40% of purchases and 10% of click throughs from the bookshelf page. My first recommendation is to include this widget on all WBUR pages related to books, accompanied with messaging about their purchases benefitting WBUR. For example:
- I also propose that we create more book lists and present them in a list-like format while retaining some important visual elements from the bookshelf. Users liked seeing book covers and photos of hosts on the page. A/B tests can be conducted to confirm whether or not it will impact Amazon purchases. As long as these elements are not preventing Amazon purchases, I think it is beneficial to include them because it’s what our users want.An advantage to creating book lists on a periodic basis, versus building a bookshelf that automatically aggregates books, is that the technical development effort is much lower. On a quarterly, or monthly basis, a radio programs team can create a quick list using existing tools, and likely replicate the impressive revenue increase we saw this summer.
Although adding book cover images to the book articles and reading list did not increase Amazon purchases, I think more testing should be done before we can draw conclusions. I say this for 2 main reasons:
- Due to time constraints, the new / old versions of the articles and reading lists were not tested simultaneously. The original versions without book cover images were launched first, and then entirely replaced with the new versions. Timing can be impacting conversion rates here.
- During interviews, our users expressed a preference for pages with more book cover images.