What goes into an operating model for Experimentation
A conversation that’s come up quite a bit recently has been with teams who are looking to kick off building an operating model for experimentation. Whether they’re looking to run that as part of a hybrid team partnering with us or something they’re running entirely themselves, the question has still more or less been the same.
“Where do we start?”
The question hasn’t always been explicitly asking about an operating model per se but at the end of the day that’s more or less what they’re asking about. The conversation happens with organisations that understand having buy-in and acceptance for experimentation is important and that it can’t just live with one small team if they’re going to have success.
So what are we talking about with an operating model, especially within the context of experimentation?
Primarily your operating model is how you set up for success, priming your team to know how they’re going to work and setting expectations for everyone else.
This isn’t yet about setting your strategy but it is about making it so that your experimentation program is set up to meet your organisation’s strategic needs.
It’s the step in between that takes the high value goals and strategy of the business and gets them ready to be acted on and operationalised.
A quick important note to remember that you don’t need to have accounted for every detail when you’re starting. Your operating model is a living and breathing thing. It’s going to (has to) evolve over time as your maturity does and as your teams change shape. Otherwise, it’s going to fall behind and lose its utility.
So how do you get started? Well, like all good plans - with a little bit of (organisational) introspection!
This is far from an in-depth guide but it is here to equip you with a starting point for that internal cognitive deep dive with an aim to provide some context on what the operating model will do for your team and how to balance short and long term planning.
There’s really two questions you need to ask yourself to kick off.
Who are the other teams that are going to be interested in doing this with me?
What are the complementary skillsets to help run great testing?
That doesn’t necessarily mean those other teams or people are needed straight away but being able to answer those questions now gives you an opportunity to start planning how you’re going to champion the program in the future and how you'll need to communicate with cross-functional team members.
For instance, if your team's success is measured on sales or customer acquisition, how do you balance out your test roadmap with a team who cares more about customer loyalty or engagement? Or if your product is living room furniture and you’re working alongside a product team who wants to drive kitchen appliances?
As you progress, working with more teams means you have access to more skillsets and more passionate team members who can help solve problems. If you’re light on developers in your team, partnering with your IT or engineering teams and helping to help them meet their goals is going to open up more support for running your experiments.
Having more teams who are interested in joining you means you have more hands to build more tests, supplement each other's skillsets, and get more buy-in from your organisation to back the operating model you’re aiming to build.
While your operating model isn’t necessarily the place to start putting your processes down, it is the place to be mapping out what processes are going to be needed.
How ideas get generated and how do they come to life. What kind of requirements you’re going to include in your roadmap. Communicating ideas to developers, approving designs, quality assurance. Sharing results with those other teams that should be involved over time.
These are all things that you don’t need to create at this stage but start getting your plan down, so you know what kinds of processes and documentation you’ll need in short order when you’re kicking off.
What does success look like for you?
I don’t just mean “improve conversion rate of my favourite metric” - as we talked about with teams, those metrics might look different depending on who you talk to.
Communicate the vision for how experimentation helps every team. Absolutely, that they’re going to be able to help improve the metrics they care about is important to that.
But when we’re talking about the vision it’s just as much around having a team that is constantly asking questions, knows how to validate the answers to those questions, and is actually doing the thing that everyone loves to talk about in digital - making data-driven decisions. It’s having coherent and consistent way to approach
Start sowing seeds early to go beyond just revenue actions or conversion rate. They’re more easily measurable and testing across the line but aren’t what makes it stick and get adopted more fully, finding value by experimenting for a wide variety of reasons.
This is a really important consideration because in most cases, you aren’t going to get to your ideal state straight away. Whether that’s because you don’t have all the skillsets in place that you’re going to want, or because some of the team you need are already incredibly busy, or just because you know you’re going to need some time to prove the value first - there’s a good chance you’re going to have to start learn and build up to your ideal state over time.
Team structure is a great example here. Many experimentation programs start with a centralised team or even just a few people who are passionate to get the practice up and running. They're doing everything from planning and strategy to delivery and measurement for all of their experiments.
Even when that's the case for your team, it’s worth having an understanding of what kinds of models other organisations run as you consider how your practice expands and who it could benefit six months or two years down the line.
If I understand, for instance, that I eventually want to have a centre of excellence for experimentation, then even if that isn't feasible yet then I can set in place some checks along the way to gauge maturity and see when we're ready to start that conversation.
When getting started folding experimentation into their teams, sometimes that means hiring a full-time specialist, sometimes it’s hiring an agency, other times it’s trying to teach existing teams about the fundamentals to start testing.
In any of those cases, at the start don’t treat running experiments like it’s some totally new, separate thing. Testing is there to improve and validate many of the things you’re already doing and folding that into your operating model is an important principle.
Your team is already writing copy, now they can come up with a few variations and see what resonates best with customers. They’re already developing new features that are competing for time and budget, creative minimal versions to see if customers show an interest before deciding what comes next.
You’re already finding ways to solve customer problems, use experiments to do that even more effectively.
There’s no way you’re going to know everything you’ll need into the future and that’s alright - that’s again why your operating model is a living, breathing thing so you can update it as you learn more and evolve.
With processes as an example, to start with you’ll just want to get the basics in place that we spoke about above. But as you evolve you might need to put a bit more detail into how folks in different departments can submit ideas, you might need to re-evaluate how you prioritise experiments, you might need to add in some additional steps as part of a QA process. This can take all sorts of different shapes but in any case, don’t worry about getting everything down on paper right now.
Do make sure you leave space for change as you mature down the line and build into your operating model check-ins, reviews, retros or other ways to take a step back and look at what’s working and what isn’t over time.
It’s too easy to spend so much time planning for every scenario that it blocks us just getting started. At the same time, it's also an easy trap to set processes in place at the beginning and get so used to running experiments that way, we don’t evaluate if they’re as effective as they could be. Having set times to evaluate and iterate how you operate helps avoid both of these challenges.
Starting to build an operating model for something you may never have done before, or only done in bits and pieces, sure ain’t easy.
You’re not going to be able to do this all on your own, so leverage the people around you for support. You’re going to need inputs from senior leaders, from cross-functional teams, and from the people in your team who will be doing much of the work. Not only is that going to be valuable to help you plan, its a great thing because it gets them involved early in helping to make decisions and figure out what’s going to make sense for them - meaning you’re many steps closer to building a program that’s going to be utilised and effective.
Take some time to think about what’s going to make sense right now and into the future so that you can just get started in the short term, while working towards a plan in the future. Just don’t think so hard about it that you don’t get moving.
I guess what I’m saying is do a bit of planning for your operating model, try it out, then measure and iterate on it. And, by the way, that’s your very first experiment for your new program!