Bayesian Bandits - Just the Beginning of the Story

Earlier this year, I put together an online Bayesian Bandit explorer as part of learning how these things work. These are used in lots of different areas, but one in particular is for selecting ads for display on a web page. Really, this would be applicable in any scenario where there will be lots of potential chances for putting something in front of someone, and you can measure the success or not of this placement.

What I have not done yet is look more deeply into exactly how this would actually be applied. I will being to scratch the surface on that here (as I understand it).

The online tool is embedded below.

Click "Start Simulation" to watch Bayesian Bandits at Work
The underlying probabilities can be modified while the simulation is running
by changing the numbers in the boxes, or by dragging the line in the current pdfs

Why is it just the beginning?

Er.... or just the end. By that, I mean that the process of updating the underlying probabilities based on the observed success rates, as is demonstrated in the visualization above, is arguably the easiest part.

The more challenging aspect is incorporating additional information - incorporating some kind of context that is relevant - so that for a given ad, you would be storing an object of the form (ad, context, α,β) where α and β are the parameters for the Beta distribution for that ad for a given context.

What is the context? The sky is the limit, it seems. It could be anything, or nothing. It could incorporate user location, demographics, what else is on the page, colors, what the user has clicked on before, whatever.

For example, a certain ad may have different click rates for certain locations in the world, or at different times of the day. You'd like to take advantage of any historical information to this effect.

It is an intimidating blank slate.

I hope to sketch out more detailed information on specific applications - as soup-to-nuts as possible - in subsequent posts on this topic.

Things to Read

• Ted Dunning's blog post on Bayesian Bandits
• Chris Stucchio's post on Bayesian Bandits
• Cameron Davidson-Pilon's post on Multi-Armed Bandits (and he has a nice reading list as well at the end of his post)
• (on my list to read) John Myles White's Bandit Algorithms for Website Optimization (from 2012) - the reviews indicate it's mostly about A/B testing, though, but I'll see.