More than a Meming? The Color of a Dress, an Interactive Tool, and Our Interpretation of Reality

Last Thursday was when I learned of the "what color is the dress" meme. The wave of interest has given way to irritated weariness as the next memes show up. However, this one is peculiar, and it feels like there is much more going on with it and how it reveals how differently and how strongly we can interpret reality.

I read how several people would have it flip back and forth between blue/black and white/gold, while others did not. This lead me to wonder about how to quantify the "degree" of your blue/black-ness or white/gold-ness. I put together a simple tool to try to do that - it is embedded below, and also available at https://learnforeverlearn.com/dresslevel.

A Few Things I Learned in my Latest Visualization Effort

I recently incorporated a decent amount of data on US religion demographics into a US Births/Deaths visualization. Here are a few notes from the road.

Reflection after the Latest Visualization Effort

I am always surprised at the things I learn or appreciate when working little side projects.

My most recent project consisted of more than thirty interactive visualizations to explore various preattentive attributes, all created with d3.js.

Slow to Launch - Perseverance

It took me a while to get any momentum on actually starting to implement this thing. I hadn't seen that a site existed with a bunch of interactive visualizations that demonstrated preattentive attributes, I figured one was needed, and I didn't want to abandon the idea. I had a vague notion of what I wanted, but for some reason I wasn't hitting on anything that got me really moving. I had to keep circling around for a week or so, reading the literature, before I hit a point that I realized that I needed to start moving up the little mountain. Things then starting clicking as momentum built up, and the form started to reveal itself. Whew.

Robert Floyd's Algorithm Pops Up Again

All of the visualizations required picking a random set of points on a grid. It's not a hard problem, but I had to think about it for a second after seeing a stackoverflow post about it. The basic idea is to treat the grid of points as a single list of points, and then randomly pick without replacement using Robert Floyd's beautiful (tiny!) algorithm, which I have written about before. I had first come across this when implementing the "When Will I Win the PowerBall" interactive visualization.

svg's animateTransform

Prior to this visualization, I had not used much of the animation capabilities "built-in" to the svg spec. The magic word here is animateTransform, and it turned out to be a simple way to get the animation I wanted: the animation itself is nicely and powerfully customizable, too.

Workflow: WebStorm, yeoman, bower, grunt...

I was already a WebStorm devotee, but had only recently been making use of the maturing javascript toolchain of yeoman and friends. This time I included the deployment itself as a grunt task, and became more familiar with the workflow in general. Despite new little bottlenecks to deal with at some point (such as the change/build/see-result-of-change delay), it feels pretty good.

Bootstrap Rocks

Gawd, the beautifully designed twitter bootstrap makes page layout/arrangement so easy and nice. Love it.

Ain't This Fun?

I have done a good bit of programming in my life, both server-side and front-end. Every now and then I would find myself doing visualization-ish stuff, but it hasn't been until the last couple of years that I have been slowly finding a comfortable "home" pursuing various visualization efforts, situated at the confluence of awesome browser features and the staggering capabilities of javascript and javascript libraries.

This stuff is great fun.

Home Pages of the United States Judicial System

I have been playing with a visualization of the U.S. judicial system as part of learning about it. As I was tracking down web sites that were not in CourtListener.com's site, I couldn't help but notice a variety of styles for the home pages for the court. Not unexpected, of course.

Anyway, I was curious what it would look like to use the home pages in the visualization, and so I used phantomjs to grab down screenshots of the sites, and then the handy "sips" command on the mac to reduce to the file size. A screenshot is below.

Designing a Viz for Multiple Screens - A Pragmatic Perspective

I was preparing to tinker some more with a d3.js visualization from last year, with the idea to improve the experience on mobile devices. In looking at it, I had forgotten I had already done this in a first pass, so that it is roughly usable on iPhone and iPad.

A Summary of Some of My Visualizations

The purpose of this page is to summarize in one place some of the interactive visualizations I have worked on. Most of these were built with d3.js, and all of them were javascript side projects I was simply interested in exploring.

"... a user interface..."

Recently, I have been studying sort algorithms, and thinking about how to create effective interactive visualizations for them. So far, I've done a visualization for Quicksort that kind of shows the power of the divide-and-conquer approach of the algorithm, but does not deal with pivot selection strategy, or deal with the in-place nature of production versions. It seems that a lot of the time, the code for these things is overly concise, static (of course), and stalls deeper appreciation. In the case of Quicksort and pivot selection strategy, this attitude seems supported by this email from (eminent) J Bentley as he was coming up to speed on how a (fairly recent) dual pivot strategy worked; in particular,

World Births/Deaths Simulation - Adding World Cities

I've been tweaking on the World Birth/Deaths simulation a little bit more.  The (still beta) version is on googledrive at any of these sites:

https://googledrive.com/host/0B2GQktu-wcTicEI5VUZaYnM1emM/
https://googledrive.com/host/0B_n1OLMaOursX2hsRUhZNElwZmc/
https://googledrive.com/host/0B4G_4-zdiD1gU2NoX285dmxpTXM/

I have also wrapped this in a self-contained "Chrome Packaged App" that can run in Google's Chrome browser.  You can get the app on the chrome app store here.

I was able to round up world city names and locations, so that this could be included as well.  Specifically, I came across  Josh Smith's worldwide-city-database on github, which itself is based on the Maxmind WorldCities Database.  This is a tab-delimited ~130MB file of a more than two million world cities/places, their population, and latitude/longitude.  Since I already had US cities and their location, I parsed out just the non-us cities with a population of at least 10,000 via a sequence of awk commands (included for my future reference):

awk -F $'\t' '{if ($3!="us") {print $0}}' cities.sql > non_us_cities.tab
awk -F $'\t' '{if ($2>10000) {print $0}}' non_us_cities.tab > non_us_cities_with_population_10000.tab

This resulted in a file of about 1MB containing information on about 21,000 cities.  I then needed to convert it from ISO-8859-1 to UTF-8 via this command

iconv -f ISO-8859-1 -t UTF-8 non_us_cities_with_population_10000.tab > non_us_cities_with_population_10000_utf8.tab 

which resulted in the data file used in the visualization (I only realized I needed to do this conversion after some of the characters were not rendering correctly).

Here's an example record from the data file (with the header row included)

combined	population	country_code	region	latitude	longitude
Andorra la Vella, Andorra	20430	ad	07	42.5	1.5166667

For a given non-US country, a random city is chosen by picking randomly based on the city's population as fraction of the sum of all of the cities for the country in this data file - this leaves out the smaller cities that might need to be included in some cases, and so I might return to this.

A recent screenshot is below.  Clicking on one of the cities in the lists will perform a google search on that city in a new window.  I still need to deal with the challenge of displaying the events on the map as they occur in a way that makes it easier to discern the specific places.

There are so many cities I just had no idea existed.  And seeing the simulated events in such specific geographic context as the simulation progresses adds a sobering poignancy.


Note: The simulation described here includes data created by MaxMind, available from http://www.maxmind.com/.

Visualization of World Births and Deaths in Real-Time (A D3 Simulation)

Note (Nov 16, 2013): I am gradually tweaking this to look a little better on some mobile devices.  This visualization also requires significant processing power that can be a challenge for mobile devices. Viewing it on the desktop is still the best option.

Last year I put together (with Bill Snebold) a D3 visualization for a statistical simulation of US births and deaths in real-time.  When I had started that project, I had meant to do one for the entire world. However, I found a nice county svg map for the US, and decided to start with that.

Recently I came across Michael Bostock's work with world maps, and decided I would look at incorporating world birth/death rates to do something similar with that.  I also wanted to build on the earlier work, in that the births/deaths in specific locations in the US would still be included.

A beta(!) version of such a world birth/death simulation is available via:

• One of these web sites
• http://worldbirthsanddeaths.com
• https://googledrive.com/host/0B2GQktu-wcTicEI5VUZaYnM1emM/
• https://googledrive.com/host/0B5Pq-FWu886acXU3bmVmTmdLbDQ/ 

• A  Google Chrome app from the Chrome store for this visualization
• This is an app that contains this visualization that runs in your Google Chrome browser, and thus is unaffected by the load on the googledrive sites below. You can get the Google Chrome browser here

Simulation of Real-Time Births/Deaths around the World

(https://googledrive.com/host/0B2GQktu-wcTicEI5VUZaYnM1emM/)

This has not been tweaked yet to run well on smaller screens.

There are different challenges with this thing.  First, the events happen at a fairly higher pace - about four births per second, a little more than one death per second, on average (roughly).  It's still amazing to think about that, btw.  Anyway, this means that it did not seem to be possible to bring each event to the forefront as they occur as was done in the previous US-only visualization, since too many would get "backed up".  The summary information is prepended to the lists, although there is an option to have it appended.  This makes the list move fast, but also helps to convey the rates involved.  Putting your mouse on the list will pause the list, although events will still be "backing up".  Moving your mouse off of the list will result in the backed up events flooding back in.

A few (slightly out of date) screenshots are below.  There are still tweaks to be done on different browsers and mobile, as getting the svg map area to line up properly for more screen sizes is yet to be implemented.  Nevertheless, the change in perspective from US-only to the whole world is interesting.

The lists can be scrolled, and as each list element is moused over, the corresponding location on the map is shown, accompanied by a small popup with summary info.

Similarly, on mouseover of a country itself on the map, a similar summary popup is displayed.

As for the mathematical approach, each country (there are about 230 included) is modeled separately for each one second interval, since different birth/death rates are used for each country.  Most of the annual average birth and death rates are mostly from the CIA factbook (Births, Deaths) - these are so-called "crude" rates, representing the average number of events per thousand population at midyear, with some values from wikipedia.  These are of course only approximations.  For births/deaths in the United States, an additional step is performed in which a specific location is chosen at random based on the population of counties and places (as was done in the US-only visualization).  Things have been implemented in such a way that this can also be done for other countries as the needed data are rounded up.  Note that I finally got around to getting a list of world cities and populations so that that could be incorporated as well, based on Josh Smith's worldwide-city-database on github, which itself is based on the Maxmind WorldCities Database.  A particular city is chosen based on relative population size in its country - of course, there are probably fancier ways to do this using additional data that may be available.

The flag images are from www.icondrawer.com

Seeing the magnitude of the events at other locations around the world, 
while of course technically expected based on the birth/death rates, 
is still surprising
(https://googledrive.com/host/0B2GQktu-wcTicEI5VUZaYnM1emM/)

Moving your mouse over a country will
bring up summary and current information for the simulation
for that country
(https://googledrive.com/host/0B2GQktu-wcTicEI5VUZaYnM1emM/)

As mentioned, there are some things to tidy up on the display for tablets, etc., and while the simulated results seem to be qualitatively similar to other world birth meters out there, there may be some refinements still needed at some point with that as well with the underlying implementation here.

This is an evolving project. Please feel free to send me any info weirdness or problems you happen to see, as well as what additional summary information made available during the simulation might be interesting.

Learning and Javascript

I like to learn.

I like to help others learn. Having to teach mathematics in graduate school taught me that. I won the B.F. Bryant Teaching Award at Vanderbilt many years ago, and I have always been proud of that.

I like to build things that help me learn. It's nice when I can tell that something I built has helped someone else learn as well.

Javascript is a wonderful way to accomplish all of these things. I have used it to build a number of interactive visualizations within the last year that have allowed me to understand things in a way I don't think I could have otherwise:

• A statistical simulation of births and deaths real-time in the U.S.
• A real-time searchable tweet map of the U.S.
• A real-time searchable vine map of the U.S.
• A Box2D visualization of money in real time piling up based on a specified rate
• An interactive visualization to help understand how the Pagerank Algorithm works
• An interactive bigram analyzer for arbitrary documents
• A Bayesian Bandit interactive visualization
• An svg marker explorer (it's tricky to understand how the various parameters affect these things)
• A cooccurrence matrix explorer for understanding how a simple recommendation system works
• An online powerball simulation to give you a feel for how long it will probably take to win the powerball, and how much you will spend.
  

Even for what are really "simple" things, like the "Idea Muscle Exerciser" I did last Sunday afternoon, I learn something new about javascript, or css, or a specific browser, or a deeper appreciation of the subject of the visualization itself. It has always been satisfying.

This is just scratching the surface of what is possible for education with this medium, and of course I'm not the only one doing it. The visualizations that I've done so far are in somewhat specialized and advanced niches based on my own interests, but these same techniques can be used for elementary and secondary educational tools. For example, in linear algebra, interactive visualizations that can help folks learn for once and for all how matrix multiplication works. How eigenvalues and eigenvectors work. Solving systems of equations.

There are certain things that a lot of folks get stuck on when it comes to certain aspects of mathematics. Having a simple interactive visualization that they can bang on, can watch as the interacting pieces of the process are highlighted in the proper context at the right time, can be incredibly effective.


These are exciting and satisfying times.

Time & Money - a Box2DWeb Visualization

We read about money rates all the time, and see numbers thrown around for how much something is costing per hour or per year. What would it look like to "see" that money as it was being spent?  This work-in-process project is one way to visualize it.  You can see the visualization here on googledrive.

Using the Box2DWeb physics javascript library, along with the public domain images of coins and (specimen) bills from the public web sites of the United States Mint and the Federal Engraving Bureau, this visualization provides some perspective on the relationship between time and money.

Currently, it is best viewed on a desktop with Safari, Google Chrome, or Mozilla Firefox.  On Internet Explorer 9,  it may act a bit flaky and this has not been completely tracked down yet.

Time & Money (click to open website in new window)

The visualization shows money being "dropped" corresponding to a specified or calculated rate. It can be used for visualizing real-time dollars per hour for a person or group, or annual expenditures by institutions.

I believe that just seeing those coin and/or bills piling up or rushing by for a few moments can stick in one's head: it adds an extra level of "realness" to the connection between time and money.

Clicking on a coin or bill will suspend it in the flow, and bring up a popup that provides summary information on the particular currency (e.g., the designer).

There are a few preconfigured scenarios as well, using estimated rates: the combined cost of Congress and the Executive branch, the US budget, the US budget deficit, and the total income of the entire US working population.

Note that while I was able to estimate costs for the Senate and House of Representatives,  the costs associated with the Executive Branch are extremely difficult to simply find, let alone estimate.  For the purpose of this visualization, the 2008(!) vaue of 1.5 billion dollars from Bradley Patterson's To Serve the President is used.  I could not find any criticism or corrections of these estimates, but will update the default values should better public lower bounds become available.  However, exact and precise values are not considered critical here: the point is that whatever the amount, it's big.

I think it would be interesting to see the impact of some kind of tool like this if they showed it on a large screen on the House floor, the Senate floor, at White House press conferences, or at the State of the Union speeches.  It just seems like it would heighten focus and attention to productivity.

Special Thanks

The coin and "specimen" bill images are from the public web sites of United States Mint and Federal Engraving Bureau, respectively.

Angie Hicks of the United States Mint provided assistance in confirming which coin images could be used in this visualization (the "covered coins").

Glen "Tommy" Smith of the United States Secret Service provided assistance when confirming that the "specimen" bill images could be used in this visualization.

A Few Technical Notes

The basis for this implementation is the Box2dWeb javascript library, which is a javascript port of the Box2D physics engine by Eric Catto.  The images for coins and bills are overlaid onto Box2D bodies that the library tracks physics for.  The coins are allowed to interact with each other, but the bills interact only with the boundaries.

For performance reasons, the bar at the bottom opens up periodically to let the currency flow down.   This currently occurs when there are about 100 coins/bills being displayed.  This visualization may be pushing the limits of what Box2D can do in some cases.  On an iPad, performance is very sluggish - I've seen this same problem with svg on an iPad.  Supporting Joel Webber's observations from last year, it seems to perform better in Chrome, although I have not yet gathered detailed statistics to confirm this, and I still need to make a serious optimization pass through the implementation.

The clock in the upper left is a customization of the "March" HTML5 Canvas Clock.  The pocketwatch image is from the the clockskin gallery of clockworlds.zxq.net, a cool site that provides free clock apps, screensavers, and skins.

The "old paper" background images are from photographer Liz West's flickr site (Creative Commons license).

And finally, there might be slight apparent discrepancies between the amount that has "fallen" and the exact amount expected.  For example, in looking at the screenshot above, the exact amount that should have fallen by exactly 14 seconds is $3.89.  However, the actual time elapsed could be as small as about 13.5 seconds, which would correspond to about $3.75, or as large as about 14.5 seconds, which would correspond to about $4.03.  Actually, in this case of 10 people at $100/hr, the implementation when running on my iMac in Safari seems to consistently hit at about 13.64 seconds (which corresponds to about the $3.79) as it marches along performing the updates, setting up a 500ms timer each time for the next update.

Simulating US Births/Deaths in Real-Time - a D3 Visualization

Note: I have extended this to a visualization for the entire world, which is on googledrive here.

Recently, I was wondering how what it would look like to "watch" as births and deaths were occurring across the globe, and I thought it might be interesting to put together a little app to assist with this.  I began exploring doing a 3D visualization with OpenGL/Shaders, but as part of the initial exploration into this I realized that the geographic data file for US counties was readily available from wikipedia, and so I decided to start with this first, using this as a good way to continue to learn how to use the excellent D3 javascript library as the core visualization engine.  Since the implementation is just html/css/javascript, this actually could be incorporated into a mobile app a la something like PhoneGap, but that is still down the road.

You can check out the visualization here.  The interface design is by Bill Snebold of Bill Snebold Design.

Simulating Births and Deaths in the United States
Link