A/B Testing, COntent Strategy, Product Design
Scribd: Home Experience
Project: Redesign the homepage to let customers indicate what kind of format (books, audiobooks, magazines, or documents) they're interested in. From prior user research, we had found that format is not fully interchangeable - they often serve different needs, and each may be appropriate for different situations, depending on the customer.
Platforms: Mobile (iOS + Android), Responsive Web
My role: Product designer working end-to-end (paired with a lead product designer). The team was made of a product manager, lead product designer, data analyst, lead back-end engineer, mobile+web front-end developers and myself.
Defining the problem problem space
To start out the project, I worked with the PM and the lead product designer for a hypothesis driven approach -- where we worked to determine what impact were we aiming for with this updated homepage.
Hypothesis #1: By allowing users to specify a single content type to view, they will see more of what they are interested in.
Hypothesis #2: By using a Multi Armed Bandit system on a per user segment basis and removing hardcoded positioning, each user segment will get the best experience based on what's resonating within their user group.
Mapping out the Reader’s Journey
After defining what we wanted to achieve, the lead designer and I paired during a working session to define:
The reader's journey on home & reflect on the multiple motivations + use cases that a customer has when they first land on the homepage
Defined what are the “key ingredients” for discovering what to read
How were other competitors solving for this problem?
We also researched what’s out there in the industry -- noted some common strategies for how have other products handled a multiple-formats experience.
Defining the content strategy
Before diving into the interaction design & wireframes -- I took some time to work through the content strategy on home. I worked with the backend developer who was creating the Multi Armed Bandit system — that optimizes the module type for every position of the home page for each user segment.
Researched in tableau to see how our current recommendation rows were performing. Did an audit of the 'bestselling' rows we could start migrating to the home page.
Defined the current pool of content recommendation rows on home that we were keeping -- and through that was able to find new recommendation opportunities.
Worked with data + product to further help define the user segments
Ideating with the team
These are some of the concept level sketches I created used to meet with the team to align on the general direction.
Designing the web experience
Led the design for translating the design onto the web experience.
Given that the homepage was relying on an algorithm that reordered rows based on activity -- I worked throughout the design process to ensure we had a flexible design so any two modules could stack near each other -- due to the module reordering that we were introducing based on a multi-armed bandit.
Along with landing on a final design for how users can select a “format” - I also created a new user education framework for introducing this new homepage on Scribd (and designing for future user education opportunities).
Full Web Experience:
Results and onward!
We went 100% with the test -- it was successful in driving reading engagement for subscribers.
Discovery successes (defined by reading the first 15 mins) from the home page increased by 43%
Each user group used their primary content type home tab more than any other content type home tab.
Subscribers read 74% more audiobooks from home page recommendations and almost 6X more from the home page library module.
The new modules we introduced (Top_category) performed well for the books/audiobooks/magazines user segment.
One unexpected discovery (that led to us digging more into saved items behavior) was seeing customer's wanted quick and immediate access to the titles they've recently saved -- we saw that customer's first successes increased 424% for that saved titles module on the home page.