Experimentation at Qantas
Continuous improvement at speed: Partnering with Australia's largest airline to improve the experience of buying flights.
Background
Qantas recognised that years of feature creep had left their flight booking journey cluttered and confusing. To simplify the experience and make purchasing a flight easier, they partnered with Deloitte Digital's experimentation team.
My goal: iterate towards an elegant and simple booking experience, optimised to perform.
The Research
Before launching straight into A/B testing, we undertook extensive research to uncover problematic areas of the site and opportunities for improvement. Our research involved both quantitative and qualitative analysis.
The Quant
Our team's strategist and data analyst dove deep into the data from the various analytics tools used by Qantas. They looked at different audiences (e.g. desktop/mobile, domestic/international) and different channels (e.g. paid search, email, social) to understand how customers behave differently based on who they are, how they arrived on the site and the type of flight they're looking for. They also performed a journey analysis to see where the largest fallout points are — typically a good indicator of a problem.
The Qual
Our qualitative research involved three parts:
User interviews: I talked to real people and asked them to navigate the site—booking a flight for a holiday or a business trip. I dug deep into their thought process to understand how each decision is made, which parts of the journey are confusing and how the experience compares to other airlines.
Heuristic evaluation: here I applied my own expertise in conversion design and methodically evaluated the site against a set of criteria to identify friction points that could be impacting purchases.
Category research: I analysed about a dozen different airlines—some premium, some low-cost—to understand how they solve for problems we've already identified in the research, and to see where Qantas might be breaking away from industry norms.
The Deliverable
The end result of the research is more than 200 different test ideas backed by both qualitative and quantitative data. From there we worked with Qantas to prioritise based on value and effort, and created a roadmap for the next 12 months of testing.
Experiments and Outcomes
For the last two years the team and I have designed and executed experiments at a pace Qantas had never been able to achieve before. Working closely with their product teams and designers, we've been able to test over 30 ideas and put many into production.
Our experimentation program has had significant impact on revenue for Qantas. Since we're accurately tracking flight purchases in our tests we can say with confidence that our work has paid for itself more than ten times over!
Not every test wins—in fact, many lose—but we're always learning more: what works for Qantas customers, what doesn't, how they make their decisions and what makes their life easier. Even losing tests lead to insights that inform future design projects and can help Qantas make informed decisions.
Now for the part we're all here for: the tests! Here are a few select examples.
Example 1: Utter declutter (flight search results)
We hypothesised that the amount of elements fighting for attention on the screen was preventing customers from selecting their flights.
I designed a new version of the page that reduced the clutter, brought important elements higher up on the screen and focused attention where we wanted it to be: on the flights.
Current Experience
New design
This was a massive test that involved a huge amount of design and development. Our efforts paid off—in our sample size of around 50k people, we saw a +20% increase in the number of users selecting flights and moving to the next screen.
The wins didn't stop there though — we saw double digit improvements all the way down to flight purchases.
A big swing, but a big win.
Example 2: Beauty isn't everything
During our user interviews we noticed that people tended to prefer airlines that included photos of the destination while browsing flight deals and sales.
Qantas's deals page is devoid of imagery, making it feel a bit utilitarian:
Current Experience
We wanted to explore whether or not adding photos made a measurable impact for customers—perhaps it would provide an emotional lift to nudge them towards a destination. We tested two variations:
Variation 1
Variation 2
Conclusion
For more than two years I've been working with Qantas to iterate and improve their website. We've had huge success in changing user behaviour and making their life easier when searching for and purchasing flights.
I'm proud of the work my team and I have achieved together—the site is looking and performing better than ever. The learnings from our tests are rigorously documented and can be referenced at anytime by anyone. This reliable and comprehensive data informs the design direction of the site moving forward.
