Optimization culture
@guido | x@gui.do​
a merchant's story
“ Sadly Thursday wasn't as good as we wanted in
terms of volumes...​
Standard deveation
...and we are now 20 orders behind target.​​
Issue seems to be related to CR. We are bringing more
traffic than last year but it's not converting as well.
= 46.9 orders
200 Employees
35 Online
4+2 Optimization
Traditional roots
Online & Offline
B2C & B2B​​​
Started CRO
24 months ago​​​
Great KPI levels
for experimentation
Every test
UX research
UX design
Frontend developer
Web/data analyst
Psychologist
E-commerce manager
Optional
​Customer Service
Business intelligence/
Finance
Business sales (florists)
​External
People/skills involved
Customer journey phase
optimization couples:
1x Optimization team
1x Business user​​
Survey​
Q1: Company type​
Startup, online
Traditional, online + offline
Q2: How old is your
testing program?​
< 1 year​
1-3 years​
3 > years​
You work at Booking (10 > years)​
Q3: MGMT support​
CEO is really enthusiastic about
testing
It's more a small (side) project
Q4: How many experiments
did you run this year?​
< 10
10 - 50
50 >
You work at Booking ( 10.000 > )
Platform
People
Process
Why do you need culture anyway?
Experimentation doesn't
come natural to everyone
Lack of trust/understanding about
data/stats/experiments undermines
your authority
Silo’s don’t let you connect the dots
Successful optimization cultures...​
Broad data literacy, leadership & trust
Inquisitive, questioning culture
Iterative, learning culture
Bottom-up (Anti-HIPPO)
Share learnings widely
Clear north star goal
​Things that didn't work
Own internal "Which Test Won"
Tell once, expect everyone to "get it"
Getting older, offline people on-board
Employee form for experiment ideas
​Things that do work
Take care of your shit umbrella
Explain all research/experiment learnings
Transparent workflow tool
Calculate ROI in kr./€/$
Shared cross-team KPIs
​Things I hope will work
Onboarding for people
outside the online team
Move away from "CRO" and "A/B testing",
move to "Idea validation"
​Quantifying every step in
the customer journey
Final thought(s)
Create the shortest single RegEx that
shows all order confirmation pages
regardless of language, platform and
TLD. Exclude error pages.
RegEx challenge
“ Last week I did the challenge and I came up with
.*.../((en|EN)/)?(Pages/)?(c|C)?onfirmation.*
...but I couldn't stop thinking about it, got a
breakthrough and made it much shorter:
[a-zA-Z]/(c|C)onfirmation
What do you think?
​@guido
x@gui.do​
gui.do/onculture

Guido Jansen -How to Involve the Whole Team in Optimization

  • 1.
    Optimization culture @guido |x@gui.do​ a merchant's story
  • 2.
    “ Sadly Thursdaywasn't as good as we wanted in terms of volumes...​ Standard deveation ...and we are now 20 orders behind target.​​ Issue seems to be related to CR. We are bringing more traffic than last year but it's not converting as well. = 46.9 orders
  • 3.
  • 4.
    Traditional roots Online &Offline B2C & B2B​​​
  • 5.
  • 6.
    Great KPI levels forexperimentation
  • 7.
    Every test UX research UXdesign Frontend developer Web/data analyst Psychologist E-commerce manager Optional ​Customer Service Business intelligence/ Finance Business sales (florists) ​External People/skills involved
  • 8.
    Customer journey phase optimizationcouples: 1x Optimization team 1x Business user​​
  • 9.
  • 10.
    Q1: Company type​ Startup,online Traditional, online + offline
  • 11.
    Q2: How oldis your testing program?​ < 1 year​ 1-3 years​ 3 > years​ You work at Booking (10 > years)​
  • 12.
    Q3: MGMT support​ CEOis really enthusiastic about testing It's more a small (side) project
  • 13.
    Q4: How manyexperiments did you run this year?​ < 10 10 - 50 50 > You work at Booking ( 10.000 > )
  • 14.
  • 15.
    Why do youneed culture anyway? Experimentation doesn't come natural to everyone Lack of trust/understanding about data/stats/experiments undermines your authority Silo’s don’t let you connect the dots
  • 16.
    Successful optimization cultures...​ Broaddata literacy, leadership & trust Inquisitive, questioning culture Iterative, learning culture Bottom-up (Anti-HIPPO) Share learnings widely Clear north star goal
  • 17.
    ​Things that didn'twork Own internal "Which Test Won" Tell once, expect everyone to "get it" Getting older, offline people on-board Employee form for experiment ideas
  • 18.
    ​Things that dowork Take care of your shit umbrella Explain all research/experiment learnings Transparent workflow tool Calculate ROI in kr./€/$ Shared cross-team KPIs
  • 19.
    ​Things I hopewill work Onboarding for people outside the online team Move away from "CRO" and "A/B testing", move to "Idea validation" ​Quantifying every step in the customer journey
  • 20.
  • 21.
    Create the shortestsingle RegEx that shows all order confirmation pages regardless of language, platform and TLD. Exclude error pages. RegEx challenge
  • 22.
    “ Last weekI did the challenge and I came up with .*.../((en|EN)/)?(Pages/)?(c|C)?onfirmation.* ...but I couldn't stop thinking about it, got a breakthrough and made it much shorter: [a-zA-Z]/(c|C)onfirmation What do you think?
  • 23.