Technology

A/B testing #eMetrics London

Description
1. A/B Testing Alexandra Paun @ToastingIdeas 2. Pre Test Significances A/A Testing 3. User Base Control Test 75% 25% 4. Revenue Test Control 5. Revenue Test Control 6.…
Categories
Published
of 50
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Transcript
  • 1. A/B Testing Alexandra Paun @ToastingIdeas
  • 2. Pre Test Significances A/A Testing
  • 3. User Base Control Test 75% 25%
  • 4. Revenue Test Control
  • 5. Revenue Test Control
  • 6. Pre test significances results biased
  • 7. The Viral Gate Test
  • 8. Revenue Control Test
  • 9. Daily Active Users 1,500,000 1,450,000 1,400,000 1,350,000 1,300,000 Control Test 1,250,000 1,200,000 1,150,000 1,100,000
  • 10. Daily Active Users 7 Day Moving Average 1,440,000 1,420,000 1,400,000 1,380,000 1,360,000 1,340,000 Control Test 1,320,000 1,300,000 1,280,000 1,260,000
  • 11. 3 Day Retention 18.60 18.40 18.20 18.00 17.80 17.60 17.40 Control Test 17.20 17.00 16.80 16.60 16.40
  • 12. Viral gate
  • 13. Too good to be true?
  • 14. The Opportunity Cost Test Could This Be Making More Money?
  • 15. User Base Cheap Plasma Expensiv e Plasma 10% 90%
  • 16. 3/6/13 3/8/13 3/10/13 3/12/13 3/14/13 3/16/13 3/18/13 3/20/13 3/22/13 3/24/13 3/26/13 3/28/13 3/30/13 4/1/13 4/3/13 4/5/13 4/7/13 4/9/13 4/11/13 4/13/13 4/15/13 4/17/13 4/19/13 4/21/13 4/23/13 4/25/13 4/27/13 4/29/13 5/1/13 5/3/13 5/5/13 5/7/13 5/9/13 5/11/13 5/13/13 Revenue Cheap Plasma Expensive Plasma
  • 17. Daily Active Users 700,000 600,000 500,000 400,000 Cheap Plasma Expensive Plasma 300,000 200,000 100,000 0 2/28/13 3/7/13 3/14/13 3/21/13 3/28/13 4/4/13 4/11/13 4/18/13 4/25/13 5/2/13 5/9/13
  • 18. Plasma bubbles sold 200,000 180,000 160,000 140,000 120,000 100,000 Cheap Plasma Expensive Plasma 80,000 60,000 40,000 20,000 0 3/6/13 3/13/13 3/20/13 3/27/13 4/3/13 4/10/13 4/17/13 4/24/13 5/1/13 5/8/13
  • 19. Fire bubbles sold 100,000 90,000 80,000 70,000 60,000 50,000 Cheap Plasma Expensive Plasma 40,000 30,000 20,000 10,000 0 3/6/13 3/13/13 3/20/13 3/27/13 4/3/13 4/10/13 4/17/13 4/24/13 5/1/13 5/8/13
  • 20. Question everything
  • 21. Long Run Testing
  • 22. How long should I keep this test running for? Long enough
  • 23. 4/29/13 4/27/13 4/25/13 4/23/13 4/21/13 4/19/13 4/17/13 4/15/13 4/13/13 4/11/13 4/9/13 4/7/13 4/5/13 4/3/13 4/1/13 3/30/13 3/28/13 3/26/13 3/24/13 3/22/13 3/20/13 3/18/13 3/16/13 3/14/13 3/12/13 3/10/13 3/8/13 3/6/13 3/4/13 3/2/13 2/28/13 Daily Active Users 1,400,000 1,200,000 1,000,000 800,000 600,000 Test Control 400,000 200,000 0
  • 24. 4/29/13 4/27/13 4/25/13 4/23/13 4/21/13 4/19/13 4/17/13 4/15/13 4/13/13 4/11/13 4/9/13 4/7/13 4/5/13 4/3/13 4/1/13 3/30/13 3/28/13 3/26/13 3/24/13 3/22/13 3/20/13 3/18/13 3/16/13 3/14/13 3/12/13 3/10/13 3/8/13 3/6/13 1 Day Retention 22.5 22 21.5 21 20.5 20 Test Control 19.5 19 18.5 18
  • 25. 4/29/13 4/27/13 4/25/13 4/23/13 4/21/13 4/19/13 4/17/13 4/15/13 4/13/13 4/11/13 4/9/13 4/7/13 4/5/13 4/3/13 4/1/13 3/30/13 3/28/13 3/26/13 3/24/13 3/22/13 3/20/13 3/18/13 3/16/13 3/14/13 3/12/13 3/10/13 3/8/13 3/6/13 Revenue Test Control
  • 26. 4/29/13 4/27/13 4/25/13 4/23/13 4/21/13 4/19/13 4/17/13 4/15/13 4/13/13 4/11/13 4/9/13 4/7/13 4/5/13 4/3/13 4/1/13 3/30/13 3/28/13 3/26/13 3/24/13 3/22/13 3/20/13 3/18/13 3/16/13 3/14/13 3/12/13 3/10/13 3/8/13 3/6/13 Revenue Test Control
  • 27. Could this be better?
  • 28. Why did I not test this?
  • 29. Daily Active Users 1,200,000 1,000,000 800,000 600,000 400,000 200,000 0
  • 30. 1 Day Retention 25.00% 20.00% 15.00% 10.00% 5.00% 0.00%
  • 31. New Users 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0
  • 32. New Users Segementation 25,000 20,000 15,000 Featuring 10,000 5,000 0 Requests
  • 33. Make your life easy. Run an A/B test
  • 34. • A/A Testing • Too good to be true (?) • Could this be better? • Long run tests • Test everything
  • 35. Questions? @ToastingIdea s
  • We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks