Last Thursday at the hot Chicago co-working space 1871, Tech In Motion: Chicago held their 15th event with the largest crowd to date - over 300 techies were in attendance to learn about Big Data and Analytics from a few of the lead developers from the Obama for America campaign. Gabriel Burt, Dan Wagner, Chris Coté and Scott VanDenPlas were the guest panelists for the evening and each served an integral role on the tech teams behind Obama for America during the 2012 campaign.
The evening kicked off with Dan Wagner who served as the Chief Analytics Officer on the 2012 Obama campaign, overseeing a 54-person team of analysts, engineers and organizers that provided analytics and technologies for voter contact, digital, paid media, fundraising and communication. His department’s work was credited with “reinventing how national campaigns are done” and has been highlighted in Time Magazine, MIT Technology Review, the Wall Street Journal, Bloomberg, the Los Angeles Times and Harper’s. It was really interesting to hear him talk about his struggles to actually convince many of the companies and organizations he's worked for that big data actually works, and that an analytics department is one worth putting funding into.
Gabriel, who now works with Dan at Civis Analytics, spoke next. Gabriel co-led the Analytics Technology team on the campaign that created its 50 TB integrated database and created tools that empowered hundreds of analysts and organizers, and oversaw some of the campaign’s most creative and impactful projects, including media optimization, mapping and analysis tools, record matching, and targeted sharing. He blogged more about his campaign experience.
After the Civis Analytics team spoke, Chris Coté, who was lead engineer for the DigiTech group discussed more about his experience with OfA. The DigiTech group was responsible for making the massive technology backend (Narwhal) accessible to the digital team, who handled majority of the front-end work. With his prior experience in social media integration he was tasked with creating "Blaster", software that "blasted" twitter and facebook messages to the President’s millions of followers.
Wrapping up the evening was Scott VanDenPlas who was the Director of DevOps for the campaign. Scott explained how the infrastructure they built has been credited as a major differentiator in the President's successful reelection bid. In total, these systems pushed a peak of 4 gigabits per second and 10,000 requests per second over 2,000 simultaneous nodes in 3 geographically separate data centers, serving 180 terabytes of data and over 8.5 billion requests. Below is the sample image that he presented showing the details of the infrastructure used by the team. To see the larger scale version of the image, click here.
We're currently putting a recap video together of this event as it really was a fascinating presentation about how to best utilize big data. Check back later if you're interested in viewing it!