Some site designs are more efficient and productive than others, whether your goal is to maximize pages per visit, time on page or even revenue per visitor. Problem is, how do you efficiently test possibilities? Sure you can A/B test two designs, but what if you could test dozens or hundreds of designs simultaneously to quickly identify the best layout for your goal?
This type of testing is known as multivariate or Taguchi testing, and it can be quite mind-numbingly complicated. Imagine keeping track of dozens of variables simultaneously!
That’s why I’m really intrigued to be running the Ezoic UI testing system…
I like experimenting with things, whether they’re technological or just recipes in the kitchen. There’s something about exploring possibilities and stumbling upon something that’s an improvement to the baseline. Heck, even knowing that you tried something and proved that it was worse than what you have now is a success because it helps confirm that what you have is good and helps you keep refining your results.
The problem is, it’s darn hard. That’s why I signed up for Ezoic and am letting them deliver the layout of the pages here on AskDaveTaylor.com. I specified that my goals were to maximize time on page, pages per visitor and revenue per page and they do the rest of the work, parsing my pages to identify blocks of information then rearranging them — and redesigning pages on the fly — to improve these particular results.
Thing of it is, I don’t actually know what your’e seeing at any given time, and given the nature of Ezoic, there are also dozens of different designs competing in the race at any given time.
It’s best demonstrated with this fascinating graph of designs and results over time:
You can see how bad designs spike in percentage of visitors who see it then quickly vanish, to be replaced by more successful ones. The winners are those that show up and stay around for a while, eventually supplanted by yet another design that’s even better at attaining the desired results. Right now, dark purple seems to be winning, doesn’t it?
Looking along the top, you’ll see that Ezoic is also working simultaneously with mobile, tablet, desktop and iOS app (if you have one) results. For me, it’s just the first three, but it’s really interesting to be able to pull out mobile design, for example, and see how those perform versus desktop visitors.
I can, of course, drill in further, at which point you can see some of the statistical results by design:
Here you can see that Macaw has the best page views/visit value, ties with Ling for time on site (TOS), and is the most valuable in terms of effective value per thousand visitors (EPMV, where “M” is the Latin “mil”, for thousand), and that just about all of them have a sufficiently high degree of statistical confidence (e.g., have collected enough data to be statistically meaningful) that the system has moved on the next designs.
In reality, this graph shows me that Garfish was doing great, with 429k visits, but it was supplanted by Ling, which has 85k visits. Macaw should be replacing both of them, however, since it has better results in all the important measures.
Like I said, very cool mad scientist results.
The designs, however, are not always particularly aesthetically pleasing because they seek to maximize specific results, not balance everything, and that’s something that I have had to get used to. Here are a few examples. Let’s start with the official “real” site design:
Pretty attractive, I think.
But push it through the Ezoic black box and it can end up looking like this:
Notice more white space and, of course, more advertisements.
Why the white space? Here’s what Ezoic had to say when I asked the team: it’s a large ad unit – called a 970×250 ‘billboard’ ad – especially for large monitors – which will get filled with the highest paying ad, no matter what the size – but if there isn’t an advertiser to to take that bigger slot, it’ll get filled with a smaller ad. As the first ad of a user session is the most valuable, the system has done the right thing and followed Google best practice for best revenue, and filled that with another ad slot – which does create white space, but it’s not necessarily representative of what a typical user is actually seeing. Fuller explanation on why bigger ad slots get filled with smaller ones can be found in the AdSense help area.
A different user might well see something that’s a hybrid of those two and no white space at all, just a tight, attractive layout like this:
Or how about this version, as displayed on an Apple iPad?
One more, an iPhone:
This one ends up showing a lot of ads, but that’s just Ezoic doing the right thing when I tell it to maximize my ad revenue. What’s a bit hard to fully understand is that there are dozens of different versions of the site design being tested at any given moment, so without going into the Ezoic backend and previewing them all, I have no idea exactly what you’re seeing when you visit this page.
Truth be told, I don’t often look at my Web site so I can just pretend it’s all good, but with its mono-focus on the statistical results that are important to me, Ezoic can sometimes product page layouts that aren’t particularly attractive. But then again, my site’s seeing substantially improved revenue, pages-per-visit and time-on-page, along with a substantial increase in Google traffic (it doesn’t hurt that Ezoic is one of a handful of certified Google AdSense partners, I’m sure).
One person doing design, even tapping best practices, can only accomplish so much in terms of improving a Web site. On Ezoic I can even see how the basic “standard” design is doing too, and it’s nowhere near as good as the Ezoic designs. Which leads to what is perhaps the most important question: how much are you willing to sacrifice in terms of traffic and revenue at the altar of aesthetics and design?
Disclosure: Ezoic sponsored this post, but I wouldn’t have written this article in the first place if I didn’t find that Ezoic was producing beneficial results for my site.