Setting Up a Great Multivariate Test
Part 5 in my MVT series will focus on the key considerations and decisions for setting up your multivariate test so that you get the most learning and best results possible. I’m not going to get too technical here since I am not a developer however I want to share what I feel will ensure happy testing.
JavaScript Implementation
The first and often most difficult part of setting up MVT is getting the javascript implemented on the pages to serve and change the test content and getting the proper tracking in place. I have seen it done in days I have seen it done in weeks and I have seen it done in months. You probably know your site architecture and IT department well enough to know how long this will take. Plan accordingly. The good news is that once that JavaScript is in place it can be used for future tests and content targeting. Every vendor has implementation guides and most of the implementation with JavaScript based testing tools is very similar.
No Ad-Side Variables
Ads can be quite influential on landing page performance, so they are a great (and rather easy) thing to test. In fact MVT done across ads and landing pages most often you will see that ad elements haves the higher factors of influence on conversion than page elements. Because of this, just like if we were working in a science lab we want to control as many variables as possible to prevent noise from getting into our test results. For a multivariate test on a landing page this means having only a single ad that sends people to the page.
Clearly Defined Channel Sources
It’s incredibly important to look at your results by segments. Channel segments like Natural Search, Paid Search (down to the keyword level), Email and others can have be very high impact and may deliver different results. Make sure that your tests are set-up to recognize the proper channel attributions (usually URL parameters) so your results can be sliced and diced and your learning can be many.
Segmentation (Profiles)
As soon as people are coming into your environment you should have your campaigns set-up so they can create non-PII (Personally Identifiable Information) rules based profiles. These profiles can create constant real-time segment creation based on behavior, source (mentioned above), temporal and environment. This profile create no only lets you look at your data in more helpful ways but it allows you to create test and target rules for these visitors either in-session or on a return visit based on what you know about their intentions and affinity.
Tagging the Funnel
Understanding how your test campaign users navigate through your experience can be very useful information for optimization and future test ideas for content and routing. You want to make sure that each step of your conversion path is being tracked. I have seen horrendous conversion funnels that have nullified some great landing page test results. Keep this in mind. Every page has a goal -- and every user has a goal on that page. The goal of the landing page is to meet or exceed the interest created in the ad and get people into the funnel. Keep this in mind as you develop testing success metrics discussed in Part 3.
Value Attribution (AOV, RPV)
Higher conversion rates with lower orders values is usually not a good thing. All this testing work is about two things. How much money did I spend and how much money did I make. Therefore it is imperative that you feed back price attribution into the test results so you can look at your results by what matters. With Omniture Test&Target this is a pretty simple process. This also allows you to do what is many times the most important testing you can be doing, price testing. If you are a publisher site you can create your own value attributions or scoring systems to help quantify testing results. This is a great idea that is unfortunately hardly ever used.
Proper test set-up is an essential part of testing. Make sure you allocate the time and manpower necessary for code implementation and QA. If your site uses redirects you want to take extra precaution to make sure that the first party cookie is passed consistently across domains throughout the user experience.
Once you are set up you can go live and start the best part of testing, monitoring the results. The next part of this series will look at results monitoring and will attempt to answer the question I get asked the most, how do I know when my test is over?
Adobe is 
