Introducing Stefan Rodricks
Stefan is a product manager at Coles looking after enablement of personalisation across web and app. Prior to moving into the personalisation space Stefan spent 6 years as an experimentation practitioner. Working with brands like Qantas, Jetstar, Bunnings, Carsales, NIB, Pet Barn, Newscorp, Dodo and Coles (more recently) to build their experimentation capabilities.
I spoke with Stefan about the interplay between experimentation, personalisation and optimisation, and how organisations can set themselves (and the talented folks in their teams) up for success.
As a starting question, how did you find your way into the world of experimentation and personalisation?
By accident!
I came from digital design and UX design, and was first exposed to testing through my days in an agency focused on email messaging. They spoke a lot about testing content, subject lines and send time - and this first exposure to AB testing I still wasn’t really thinking about what it meant or why it was important.
When I joined The Lumery I was tasked by my manager, Ben Fettes, to basically own the testing practice, focused on ecommerce brands. With a crash course in Google Analytics and testing programs used, I fell in love with data and experimentation. I realised we had a way of finding objective truth, and using numbers to be clear about which designs customers love.
Shout out to Ben. His love of experimentation really rubbed off on me and led me to this path.
I also realised that as a designer, I was free to explore so many more solutions and let customers tell all of us which solution works best, instead of being pigeon-holed into a single solution based on a brief. That was an incredibly freeing feeling.
This world of experimentation helped me realise that there are many solutions for one problem.
Not enough designers find themselves diving into data (and vice versa)! How did your work in analytics influence your design?
Being able to dig deep into data helps me validate assumptions or redirect me to what the actual truth was. It enabled me to do on-the-fly heuristic analysis, because I then had all this data that showed me what customers were doing when I looked at the page.
I could do a quick heuristic analysis about why behaviours might be happening by bouncing back and forth between the website and analytics. Like, that makes sense because I'm seeing that this component here it's got too much copy and the CTA is way down the bottom and this right here is where customers are dropping off. So that also makes sense in the data.
It allowed me to marry the what (quant) and the why (qual) really quickly, which made me a better strategist. At the end of the day, a design can go so many varying ways depending on the brief, but being able to understand the core customer problem that needs to be solved allowed me to take a strategic approach to designs that might work. . It's a strategy thing first and foremost. If you have the right strategy to solve the problem, then your designs and your creative and execution can be guided in the right way.
Your journey is one that spans consultancies and brands with exposure to how a lot of companies are operating. With that broad spread of experience, how do you rate the maturity of experimentation in the Australian market?
Walking back from the initial question, it’s probably best to first call out what I think makes a mature experimentation practice.
Mature experimentation is embedded as a practice within, at minimum, product teams if you're a digital product business. The most common approach stopping maturity is thinking “We’re under a time crunch, we’ve just got to fix this”. As opposed to finding a true solution to the customer problem. It’s treating the symptoms. Experimentation should help us solve the cause of the problem.
The ability to do a bunch of tests and learn that then informs the optimal experience so that you're going to market with less of an MVP, more of a MLP, a more lovable proposition. Then being able to then do your optimizations and your CRO on on top of that feature. Learning from what customers are telling us from those interactions so that it feeds back into the next evolution of your product or your experience. At this point experimentation is baked into the life cycle of your product evolution or your key performance metric.
That's mature experimentation where it is part of the living, breathing life cycle of product and the evolution of the experience. That doesn't mean that we all do it properly though.
Experimentation can be hard because you come up against the realities of living deadlines, stakeholder expectations, market pressures, and competitive pressures.
It can be really hard to do in its entirety.
As for who is doing it well in Australia, there are a number of organisations that actually really value it. The industries where I've seen it done the best actually sit outside of E commerce. I think airlines do really well. I think banking does really well.
These businesses can do well as their customer life cycles are really staged. You've got your acquisition going to on-boarding, consideration, purchase, post-purchase and then retention. And I think where it can be really easily delineated between “stage” of the funnel or product line, I think of experimentation. It goes really well in those environments.
Obviously there are challenges for lots of brands in building and scaling their experimentation practices. What do you see as the success factors for brands seeing results from experimentation?
At minimum, the leadership in the organisation are talking about experimentation and there is a desire to focus on experimentation beyond “it’s expected from us” or “our competitors and peers are doing it”. When leadership actually genuinely embrace experimentation and they're building organisational KPI's around it or they're investing in ways of working that support it.
That’s a necessary condition, but it’s not sufficient.
I've been in this digital space for almost 10 years and I've done experimentation in a number of different permutations. Organisational structure around it is where things either fail or ultimately succeed, so I think if organisations are setting up their teams to experiment and to provide time to focus on unpacking customer problems that will then also result in improvements to key business metrics.
You’ve got to set up your organisation and your teams to have the time to sit with the problem and have the breathing space to explore the problem. This enables them to go after solving customer problems that are the biggest slices of pie. That is where you get the wins.
I personally haven't seen it implemented here in Australia (it might exist, I just haven’t seen it yet!), but I've always really liked the idea of like Growth Product Managers. While product teams own the baseline experience and they own all the other OKR's or key metrics or KPIs for that experience, they're concerned about delivering the road map. Onto the next feature. Growth Product Managers with their own teams sit either within their squads or sit alongside the squads and they're like, “Cool, you delivered this feature, it's now the new baseline. I'm going to go and grow this feature.”
In the process, Growth PMs contribute back to the businesses KPIs around NPS, revenue as key business outcomes. Feed their learnings back to the product squad. The product squad then goes and identifies how to then evolve the experience in partnership.
Experimentation being embedded into the product lifecycle, to drive iterative growth, is key. You've got someone sitting on your team as a companion saying, “Hey, we're going to do that discovery work coming up in your roadmap, feed it back to you with quick and dirty AB testing” and then they can take those learnings, put it into the next quarterly planning cycle and to evolve the experience. I think that organisational structure will really help drive better outcomes for customer problems.
That sounds like a key role for brands in the larger enterprise space to take advantage of, though lots of the companies experimenting in Australia have much smaller teams that might not support a Growth Product Manager. What kind of advice would you give to practitioners now looking to grow their practice in the business?
1. Attach experimentation to an existing roadmap
It really depends on the size of the organisation you're in and what type of organisation it is, but I think if you're an individual building out a roadmap of experimentation your roadmap should be an extension of someone else's'.
That way, when experimentation is deployed and the learnings are disseminated to the business it's more likely to be adopted if it's more relevant and accessible to many teams at the same time. As long as your experimentation is part of a 12 month product roadmap where you're helping to grow and get towards the goal, it's likely to be far more successful and far more adopted in the business because it's applicable.
At the end of the day, comparing a business that relies on that roadmap to deliver revenue or deliver customer experience, it's going to focus on revenue.
2. Allow space for your ideas to be tested
At the same time, I think every experimentation practitioner should have a few ideas that are just their own.
"One to two to three things in a year that they deliver that is just for them to learn, grow and specialise.
"
Those could be tests that you believe are a gap that no one's looking at. That you know, you could go and use to learn more about the discipline to develop yourself. You could test to learn more about the organisation or learn more about the specific customer problems that keep coming up. Or it could be something that you use to actually tell the story and sell experimentation through the business because it's an interesting test. It’s the Pixar model: one for the box office, one for the story telling.
No one is thinking about it, and it doesn't have a place on anyone's roadmap, but it is the focussed problem solving that you do that is a major bang for buck. And I I think the success that I've found in the experimentation I've run was where it was a problem that no one was thinking about, but I knew it'd be valuable at some point because it applied across many teams: marketing, digital, growth.
3. Measure seriously: celebrate revenue
If we talk about the organisation, the ability to speak to revenue by experimentation is super important. The ability to confidently stand by your numbers so that when you want the organisation to take experimentation seriously and you want them to invest - having bulletproof numbers is invaluable.
What you do has to be entirely measurable. You need to have a strong measurement framework around everything. I find mapping input metrics to output metrics, with output metrics being representative of your KPIs/OKRs, is a really strong way to build out a measurement framework. That also then enables the structuring of very methodical testing strategies with clear focus pillars of tactics to test. Then ensuring that your tagging is accurate when it comes to deploying tests so that you can speak to the numbers with confidence and communicate the value that GMs think of, which is usually dollars. That will make or break an experimentation program.
