Presented by Evelyn J. Boettcher, Founder of DiDacTex, LLC
Developers & Data Analytics
A / B Testing
A/B testing is a randomized controlled experiment done in production.
There are two tests: A and B, in which a single variable adjusted (B Test).
This variation, might affect a user’s behavior.
Goal: Increase End User’s Objective
Business: Income increases >> Costs of change
Health Care: Health increases >> Side effect
Weekly Planner Choices
A
B
So why do A/B Testing
Why spend the resources to do tests
Why risk angering your customers with changes?
I got data miners, I do not need tests!
I am a developer, I don’t need to know the business side
Life is complicated.
Domain knowledge only gets you so far!
Subject-matter experts (SME) from Microsoft, Netflixs etc, find when they implement changes that only a small fraction of their ideas have the planned outcome (Sweet 2022).
Even though we are isolating a single variable, that variable interacts with a million other variables. You simply can not model everything or know everything.
Any change can hurt.
Even tested changes
Though less likely!
Change can help
“If you’re not growing, you’re die-ing”
Gedanken:
Trader Janes has a New Pizza
Say there is a grocer called Trader Janes and it wants to add a new pizza to its line up. However, they need to keep the same number of types it sells a constant. Freezer only holds N pizzas.
They will have to remove one pizza from their lineup to add the new item.
Typical Work flow
Marketing asks a Data Miner to rank popularity of pizza.
The Data Miner finds the pizza that sells the least.
Don’t know, because the store implemented many changes that week!
It’s the week after Thanksgiving and sales always go down that week.
Turns out there was a small group of heavy spenders that love this pizza.
A / B Testing help predict what changes will increase the bottom line.
A / B Testing Limitations
A/B Testing is used to clarify a vision, but does not create vision.
For example, an ophthalmologist quickly gives you a set of two choices; 1 or 2 (2 or 3) that lead to sharpen vision. Their test, like A/B, can not give you vision.
Though without a clarity, a vision has serious limitation.
Frequency range for the Voice Spectrum over copper: 300Hz to 3,400Hz
Men, Women and Children have different fundamental frequency
Shrill
Most consonants spoken are in the 400 to 4500Hz.
With Women having most of their consonants sounds showing up in the higher frequency’s.
Green Bar shows the cutoff for the voice spectrum
This caused women’s voices to sound shrill.
It also made it hard to understand what they said, since their voice was cut off.
Bias has caused women to change.
Margret Thatcher during her career changed her voice.
Dropped her main vocal frequency roughly 60Hz! (Tallon, n.d.)
Almost a 1/3!
Women’s voices have dropped on average over 23Hz from 1945 to 1993. (Cecilia Pemberton 1998)
Women’s voices have been becoming more manly.
Was the Voice Spectrum Biased: Yes
In 1927 a voice spectrum had to be defined.
J.C Steinberg (from AT&T) knew that the proposed voice frequency cut off women’s voices. He wrote a letter titled “Understanding Women”.
He states that men traditionally have an inability to understand women except when their tone is soft.
So, it is a “biological failing of women” (Tallon, n.d.) that we can’t understand them. \(\therefore\)The technology as is, is good.
Market prospective
In A/B Testing, we focus on the question does doing A or B make the company more successful.
Women make up 1/2 the market.
When another company/technology comes along that cover’s women’s voices better, it is reasonable to assume that they will get that market share.
Who has a landline?
My hero: HD-VOIP
Note: Narrow Band (free to use) VOIP is 300 to 3,400Hz.
With more diverse workforce, a research(er) bias will go down.
True
But, there is still unseen bias in voice in the 21st century!
Google and Apple had a hard time getting voice recognition to work for kids (Scanlon, n.d.)
Not only do kids speak at higher frequencies than women, they have different speaking patterns.
One can not simply take an adult’s voice and shift the frequency. So ML/AI have a hard time figuring what kids are saying.
Why kids are important
The market for voice recognition for kids looks to have a strong market growth.
Conclusion
A/B testing is a randomized controlled experiment done in production.
There are two tests: A and B, in which a single variable adjusted (B Test).
This variation, might affect a user’s behavior.
Your A/B Testing should:
Make the company more successful.
Follow some ethical guidelines, like the \(F_{upped}\) potential
If \(F_{upped}\) > 0, test should be a no-go.
If \(F_{upped}\) < 0, \(\Delta\) should be halved.
A/B Testing has Risk
No free lunch.
Even after testing, test results might not make the company more successful.
Thank you
Please check out Gem City Tech.
Gem City Tech
Gem City TECH’s mission is to grow the local industry and the community by providing a centralized destination for technical training, workshops and providing a forum for collaborating.
Cecilia Pemberton, Alison Russel, Paul McCormack. 1998. “Have Women’s Voices Lowered Across Time? A Cross Sectional Study of Australian Women’s Voices.”Journal of Voice, 208–13.
Kim, Gene, Kevin Behr, and George Spafford. 2013. The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win. 1st ed. IT Revolution Press.
