How this Leading Brand Overcame the Fear of Holiday Optimization [Case Study]
Optimizing your eCommerce site during the holiday season doesn’t have to be scary. In fact, there’s now a risk-free way to unlock the true revenue potential of your customer experience, backed by data from leading merchants, SIs, and tech providers in the Mobile Optimization Community.
In eCommerce, customer experience optimization during the holiday season is typically off limits. Companies fear that the risks associated with testing new ideas during the high revenue season will outweigh the benefits. Recognizing that online consumer sentiment and preferences are constantly changing—especially during the hectic holidays—The Timex Group put aside their concerns and extended an existing adaptive optimization campaign that was running on their product detail page during the high-stakes season. The test started during the pre-season and continued throughout the peak- and post-holiday period.
Much to their surprise, and delight, the experiment yielded +20.40% in new revenue while uncovering a winning ensemble of optimization treatments that had +36.35% lift.
More importantly, the Timex Group campaign is an industry game-changer that debunks conventional wisdom—pre-holiday results, in fact, may not work as expected during the holiday season. Therefore, it is imperative for eCommerce sites to continually optimize the customer experience throughout the holiday season for maximum revenue.
The Challenge: Fear of Holiday Experience Optimization
Conventional wisdom among eCommerce companies is to run A/B or multivariate tests during the pre-season to optimize online customer experiences. They then freeze the code and do not actively experiment during holidays. After years of experience and positive results with HiConversion’s adaptive experience optimization capabilities, Timex Group management was confident to continue an optimization experiment that was marginally negative during the weeks preceding the holidays.
The Solution: Adaptive Experience Optimization
The Timex Group optimization campaign, which started weeks before the holiday season, had four experimental variables from their product description page and miniCart template layout. All were simple changes introduced to experiment with the existing page elements. The goals were to detect the importance of individual elements and their optimal page locations. The test included the following treatments:
- Cross Sells: Move product recommendations higher on the page
- Photo Share: Make user created content more prominent
- Mini Cart: Modify layout and the content of the mini cart
- Distractions: Remove several auxiliary elements like reviews or social links from the product selection area
Considering that the test ran over an extended period of time, including both the pre- and peak-holiday season, conventional wisdom would conclude that the cumulative campaign results are ‘evergreen’ and should work all the time. Common practice at this stage would be to recommend a permanent implementation of the winning treatments. After all, the campaign had reached the required sample size with nearly 100% statistical confidence.
As we will see, following conventional wisdom can often be costly.
The Insight: User Behavior Varies With Time. Retailers Must Change Optimization Practices.
When using time as a segment, however, Timex Group management discovered that the holiday season skewed the test results. The revenue lift seen over the full campaign period was entirely due to the strong performance of all treatments during the peak holiday season.
Let’s examine the impact of time on consumer shopping behavior before, during and after the holiday season.
Time Segment: Pre-Season
The initial test results that ran in November did not look very promising. However, Timex Group management agreed to continue the optimization campaign through peak season, knowing that consumer behavior can shift dramatically during the holiday shopping crunch time.
Time Segment: High Season
Timex Group’s patience paid off. Treatments that performed negatively before the holiday season dramatically improved in performance during the busiest period of the holiday season. The difference in results between the pre-season and peak-season time periods compelled Timex Group to further extend this optimization campaign into the post-holiday season.
Time Segment: Post-Season
The post-season results confirmed that visitor behavior during peak-season is dramatically different than during off-season. What was once a strong leader during the peak-season became one of the many negatively performing ensembles once the timeframe being observed was adjusted. What worked well during the holiday season brought in a lot of additional revenue, but even with 99% statistical confidence, was not guaranteed to continue to perform strongly in the post-holiday period.
Conclusions
The Timex Group campaign clearly demonstrates the need to continually optimize online customer experiences—especially during the high-stakes holiday period. What works well during the peak holiday season, may in fact, not perform well before or after.
In today’s experience economy, the only sustainable solution is to go beyond cumulative results and statistical significance. Online merchants must deliver a continuous optimization of the online buying experience. This requires a new methodology that can eliminate the risks of traditional testing by automatically detecting and adaptively optimizing the customer buying experience in real-time.
