Documents

A-B testing

Description
A/B testing 1 A/B testing In marketing and business intelligence, A/B testing is jargon for a randomized experiment with two variants, A and B, which are the control and treatment in the controlled experiment. It is a form of statistical hypothesis testing with two variants leading to the technical term, Two-sample hypothesis testing, used in the field of statistics. Other terms used for this method include bucket tests and split testing but these terms have a wider applicabilit
Categories
Published
of 4
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Transcript
  A/B testing1 A/B testing In marketing and business intelligence, A/B testing is jargon for a randomized experiment with two variants, A andB, which are the control and treatment in the controlled experiment. It is a form of statistical hypothesis testing withtwo variants leading to the technical term, Two-sample hypothesis testing , used in the field of statistics. Otherterms used for this method include bucket tests and split testing but these terms have a wider applicability to morethan two variants. In online settings, such as web design (especially user experience design), the goal is to identifychanges to web pages that increase or maximize an outcome of interest (e.g., click-through rate for a banneradvertisement). Formally the current web page is associated with the null hypothesis.As the name implies, two versions (A and B) are compared, which are identical except for one variation that mightaffect a user's behavior. Version A might be the currently used version (control), while Version B is modified insome respect (treatment). For instance, on an e-commerce website the purchase funnel is typically a good candidatefor A/B testing, as even marginal improvements in drop-off rates can represent a significant gain in sales. Significantimprovements can sometimes be seen through testing elements like copy text, layouts, images and colors, but notalways. The vastly larger group of statistics broadly referred to as Multivariate testing or multinomial testing issimilar to A/B testing, but may test more than two different versions at the same time and/or has more controls, etc.Simple A/B tests are not valid for observational, quasi-experimental or other non-experimental situations, as iscommon with survey data, offline data, and other, more complex phenomena.A/B testing has been marketed by some as a change in philosophy and business strategy in certain niches, though theapproach is identical to a between-subjects design, which is commonly used in a variety of research traditions. A/Btesting as a philosophy of web development brings the field into line with a broader movement towardevidence-based practice. Common Test Statistics Two-sample hypothesis tests are appropriate for comparing the two samples where the samples are divided by thetwo control cases in the experiment. Z-tests are appropriate for comparing means under stringent conditionsregarding normality and a known standard deviation. Student's t-test are appropriate for comparing means underrelaxed conditions when less is assumed. Welch's t test assumes the least and is therefore the most commonly usedtest in a two-sample hypothesis test where the mean of a metric is to be optimized. While the mean of the variable tobe optimized is the most common choice of Estimator others are regularly used. History Google data scientists ran their first A/B test at the turn of the millennium to determine the optimum number of results to display on a search engine results page. While this was the srcin of the term, very similar methods hadbeen used by marketers long before A/B test was coined. Common terms used before the internet era were splittest and bucket test .As with most fields, setting a date for the advent of a new method is difficult because of the continuous evolution of a topic. Where the difference could be defined is when the switch was made from using any assumed informationfrom the populations to a test performed on the samples alone. This work was done in 1908 by William Sealy Gossetwhen he altered the Z-test to create Student's t-test.  A/B testing2 An emailing campaign example A company with a customer database of 2000 people decides to create an email campaign with a discount code inorder to generate sales through its website. It creates an email and then modifies the call to action (the part of thecopy which encourages customers to do something  € in the case of a sales campaign, make a purchase).ããTo 1000 people it sends the email with the call to action stating, Offer ends this Saturday! Use code A1 ,ããand to another 1000 people it sends the email with the call to action stating, Offer ends soon! Use code B1 .All other elements of the email's copy and layout are identical. The company then monitors which campaign has thehigher success rate by analysing the use of the promotional codes. The email using the code A1 has a 5% responserate (50 of the 1000 people emailed used the code to buy a product), and the email using the code B1 has a 3%response rate (30 of the recipients used the code to buy a product). The company therefore determines that in thisinstance, the first Call To Action is more effective and will use it in future sales. A more nuanced approach wouldinvolve applying statistical testing to determine if the differences in response rates between A1 and B1 werestatistically significant (that is, highly likely that the differences are real, repeatable, and not due to random chance).In the example above, the purpose of the test is to determine which is the more effective way to impel customers intomaking a purchase. If, however, the aim of the test had been to see which would generate the higher click-rate  ã thatis, the number of people who actually click onto the website after receiving the email  € then the results might havebeen different.More of the customers receiving the code B1 might have accessed the website after receiving the email, but becausethe Call To Action didn't state the end-date of the promotion, there was less incentive for them to make an immediatepurchase. If the purpose of the test had been simply to see which would bring more traffic to the website, then theemail containing code B1 might have been more successful. An A/B test should have a defined outcome that ismeasurable, e.g. number of sales made, click-rate conversion, number of people signing up/registering etc. Segmentation and targeting A/B tests most commonly apply the same treatment (e.g., user interface element) with equal probability to all users.However, in some circumstances, responses to treatments may be heterogeneous. That is, while a treatment A mighthave a higher response rate overall, treatment B may have an even higher response rate within a specific segment of the customer base.For example, the breakdown of the response rates by gender could have been: OverallMenWomen Total sends2,0001,0001,000Total responses803545Treatment A50 / 1,000 (5%)10 / 500 (2%)40 / 500 (8%)Treatment B30 / 1,000 (3%)25 / 500 (5%)5 / 500 (1%) In this case, we can see that while treatment A had a higher response rate overall, treatment B actually had a higherresponse rate with men.As a result, the company might select a segmented strategy as a result of the A/B test, sending treatment B to menand treatment A to women going forward. In this example, a segmented strategy would yield an increase in expectedresponse rates from 5% ((40 + 10) / (500+500)) to 6.5% ((40 + 25) / (500+500)), constituting a 30% increase.It is important to note that if segmented results are expected from the A/B test, the test should be properly designed at the outset to be evenly distributed across key customer attributes, such as gender. That is, the test should both (a) contain a representative sample of men vs. women, and (b) assign men and women randomly to each ‚ treatment ƒ (treatment A vs. treatment B). Failure to do so could lead to experiment bias and inaccurate conclusions to be drawn  A/B testing3from the test.This segmentation and targeting approach can be further generalized to include multiple customer attributes ratherthan a single customer attribute  ã for example, customer age AND gender, to identify more nuanced patterns thatmay exist in the test results. Acceptance Many companies use the designed experiment approach to making marketing decisions, with the expectation thatrelevant sample results can improve positive conversion results. It is an increasingly common practice as the toolsand expertise grows in this area. There are many A/B testing case studies which show that the practice of testing isincreasingly becoming popular with small and medium-sized businesses as well. A/B testing tools comparison ABtastyClickThrooContentExperimentsSiteSpectVisualWebsiteOptimizermaxymiserPlanOutCost 29 „   $95FreePaid$49PaidOpen Source(free) Email campaigns ……†………… Multivariate testing …††………… Within-subjectsdesigns ††††††… Target platforms Client-sideClient-sideClient-sideClient-sideClient-sideClient-sideServer andclient-side Interface GraphicalGraphicalGraphical and APIGraphicalGraphicalGraphicalAPI References  Article Sources and Contributors4 Article Sources and Contributors A/B testing   Source : http://en.wikipedia.org/w/index.php?oldid=621619769 Contributors : A0Carol, Allenkristina, Alston2, Ankneo, Apparition11, Banej, Bco321, Bennose, Bhny, Bigoptimizer,Bill william compton, Bn, Bonadea, Brossow, Coderzweb, David Vincenzo, David spector, David2006ABC, Dekart, Derek farn, Designermargaret, Discospinster, Dougluce, ENeville, Ebakshy,Ed g2s, Elliotykim, Elvanor, Epicgenius, Fabianjoerg, Fhorton, Herrold@owlriver.com, Icowboy, Israelrs, Isuttle, Jamesontai, JanisAnalytics, Jayeshsenjaliya, Jehochman, Jreffell, Jweiss11,K8lj, Karlblanks, Kayau, Kikichugirl, LaurenABtest, Lineslarge, LittleBenW, Logical Cowboy, MKar, MMN389, MPM10223, Mabdul, Magioladitis, Makhinko, Mangotron, MartinDower,Matma Rex, Mcoupal, Melcombe, Metricamob, Mint5auc3, Mizgulch, Mortense, MothProofLemming, Mpadavala, Mwexler, Necessary Evil, Nightscream, Nimrod Cohen, Oli Gardner,OwenBlacker, Phatom87, Poiuytrez, ReconditeRodent, Reedy, Researchninja, Ringbang, Rkettani, Ronnykoh, SFK2, Sander S‚de, Seraphim, Shoecream, Siddes, Skcpublic, Sougat818,StaticGull, Stj6, Stor, Sun Creator, SwisterTwister, Tajujoseph, Tawatson29, Tbhotch, Teachtosing, Thosjleep, Tomƒ„ Pil…k, Tony1, Trappist the monk, Trivialist, Usmanwardag, Vhann,Webtistic, Xanzzibar, 205 anonymous edits License Creative Commons Attribution-Share Alike 3.0 //creativecommons.org/licenses/by-sa/3.0/ 
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks