The homepage nobody had ever tested

The homepage had accumulated years of undocumented changes, each shipped on somebody's judgment, none of them measured. Two things followed. Nobody could say which parts of the page were doing the work. And every redesign proposal turned into an argument between opinions, because there was no shared evidence to argue from.

So the safe answer was always to change nothing — while the page kept quietly underperforming.

Testing one "new homepage" against the old one tells you that something changed, never what. So I built variants that differed along a controlled dimension: two designs with identical block structure, differing only in header design and visual styling.

Holding structure constant meant the result would speak to styling alone — whatever won, we would learn something transferable instead of a one-off answer.

Google Analytics answered what happened — conversion, revenue, per-session value, statistical confidence. Hotjar answered where — which of the three steps between the page and a finished order each variant lost people at, over the same days.

187,131 sessions across 43 days, three variants live at once, segmented by new users and by mobile traffic. On a funnel this size a test read too early produces a number that is confidently wrong, and one wrong call would have ended the team's appetite for testing altogether.

Both variants beat the control. The winner lifted conversion by 6.73% relative at 100% probability of outperforming the original. Per-session value rose with it.

Then the part designers usually skip: I converted the result into money, calculated what it would cost the company not to ship it, and put that figure in front of the stakeholders. I built it on the new-user segment only, deliberately conservative — the floor rather than the ceiling, because a number nobody can argue down is worth more than one that invites argument.

Guidelines instead of a relaunch. The findings became rules applied to the live page round by round, each change carrying its evidence — which is why the control kept improving underneath the later tests. An awkward result to report, and the useful one: the page was never relaunched, only improved against a measurement.

A principle, not just a page. With block structure held constant, styling alone produced a real, measured lift. What the team kept was not “styling matters” but the habit: change one dimension at a time, and the answer transfers to the next page. Redesign proposals stopped being a contest of taste and became a contest of evidence — a better argument to be having.

Testing as the default. The company went from shipping on opinion to shipping on measurement. By the end I was running that loop as Product Owner across a ten-person team, and facilitated the company's entry into the UK market.

The strongest finding of the programme was one the company did not act on. I documented the cost after the fact rather than making the case forcefully enough beforehand.

The lesson was not about analysis — the analysis was right. It was that a result is only worth what the organisation does with it, and getting it shipped is part of the designer's job, not somebody else's.

• The exact revenue figures, and the forgone-revenue calculation as shown to stakeholders
• The GA Content Experiments table for the decisive test, and the round-by-round account of the other three
• The Hotjar funnels behind each conclusion — variant against control, step by step, round by round
• The homepage as it actually changed — the original, the prototype and the directions that followed — and the working board they came from

It lives at alex.zhovnir.com/work/uvoteam, behind a password — client agreements, not theatre. Send me a note and I’ll open it: alex@zhovnir.com.