Nomadic Teams

A groundbreaking, revolutionary experiment in … seating arrangements

Yesterday one of our project teams started sitting with each other. They picked up their things — computer, monitors, accessories, USB-powered Nerf dart launchers — and took over an unoccupied set of contiguous desks. The idea is to sit together for the project’s duration, pounding out great software monitor to monitor. We call it “nomadic teams”. The draw for us is a function of our size, org structure and ample spare desks. It might be interesting to your team too.

We’re structured, and normally seated, in a way that inhibits efficient project conversation. Our team is organized functionally — to simplify, front-end, back-end and QA — and that’s how we’re permanently seated. Projects, however, are carried out by temporary combinations of engineers and testers, who normally work apart at their respective, physically separated, desks. How lonely. How inefficient.

Despite the past century and a half’s technical advances, and our technologist/geek bias towards asynchronous communication where no human to human contact occurs, we hypothesize that this thing we call speech still hasn’t been trumped by anything that’s come since. Ignoring mathematical and scientific notations, nothing’s more information-dense. Yeah, you can read more written words per unit of time than you can speak them, but first-hand, in-person speech takes the cake for problem solving and relationship building. Think about the back and forth of a great conversation. Or how about the info carried in non-verbal cues? I digress…

So we hope this will enable us to build a better product even quicker. And we’ll have a better time along the way. We’ll report back on how it works out.

Our first "nomadic" team

Our first “nomadic” team

Bitten by MySQL 5.5′s metadata locking

This will be interesting if you run MySQL 5.5.3+ and have been blocked by the dreaded metadata lock. If you’re unfamiliar, the situation goes like this. You want to run a DDL query such as a schema change, say to expand an ENUM or add a column. If the table(s) in question are heavy queried, your attempt to change the schema will contend for the metadata lock. The acute effects are your DDL query hangs, along with every other query hitting the table. Everything grinds to a halt until you kill your query. In all accuracy, this “feature” might behave differently in 5.6.

In Sprout’s case, we have some extremely high-volume tables. Several are internally notorious for being unchangeable. We will have to shard them eventually.

Today we got too clever in an attempt to alter one. We wanted to change an ENUM, which is usually a quick operation not needing a table rebuild; a quick aside, never use ENUMs and always use small integer types instead to avoid ever needing to change them. To reduce the metadata lock likelihood, we paused some services and redirected a bunch of important, high-volume SELECTs to slave databases. Note that part about the slaves; that’s what bit us.

The DDL ran and presto! It …. actually …. worked. The before-unchangeable table just changed. It was one of those incredibly exciting yet anti-climactic moments. Then a minute later all of our phones lit up. Pingdom was alerting that Sprout was not responding. Our app was, from the user’s perspective, down. Three minutes later Pingdom suddenly said it had reached Sprout — it was “up” again. We could login too.

So what happened? I’ll skip to the chase. The DDL worked on the master and then replicated to the slaves. Those “important, high-volume SELECTs” were hitting said slaves. As the DDL executed on them, it caused contention while getting the metadata lock. Remember they were under a different read load pattern than the master. We were lucky it resolved itself quickly.

So don’t forget about replication. Metadata locking will impact slaves too (duh!) and, pay attention here: independent of how it impacts the master. In hindsight that is completely obvious. The metadata lock has been such a foe that the chance at one small victory completely clouded our judgement.

Android: Using Volley and Loopj with self-signed certificates

A new Android networking library named Volley was introduced this year at Google I/O. According to Ficus Kirkpatrick, one of the creators, it should provide a performance boost, give customization options, offer request tracing and more. We recently converted most of our networking stack to Volley, though we still use Loopj for making POST requests as we found some limitations with Volley.

We use self-signed certificates during development and thus needed to work that out with each library. Here’s how to make that work in both libraries.

Loopj

The first step is to grab the self-signed certificate from the server which you’d like to make requests against and copy it to your local development environment. Easy enough, just use something like scp to copy it down.

Once you have the server’s self-signed certificate locally, convert it to BKS format. Assuming your certificate is in CRT format, you can use this command:

keytool -importcert -v -trustcacerts -file "server.crt" -alias imeto_alias 
-keystore "server.bks" -provider org.bouncycastle.jce.provider.BouncyCastleProvider 
-providerpath "bcprov-jdk16-146.jar" -storetype BKS

When you run this command you will be prompted for a password. Remember whatever password you enter as you will need to modify strings.xml to include it.

The next step is to copy your certificate to your resources directory in your Android project: /res/raw. Once the certificate is in your project locally, modify your strings.xml file to include the password you were prompted for.

<string name="store_pass">your password here</string>

Next, create a custom socket factory class which uses your self-signed certificate in /res/raw (or wherever you placed it). Below is an example.

The last step to get Loopj requests to work is to instantiate an Async client and use your custom SSL factory.

Volley

This will cover any Loopj requests, but what about Volley? There are several stack overflow threads describing how to effectively ignore certificates which seems to work with Volley. Ignoring SSL certificates is very dangerous so be careful about making sure this code does not run in production. Below is an example of class that will create an X509TrustManager that will accept all certificates.

Our dear friend, MySQL

We use MySQL a lot at Sprout Social. NoSQL is throughout our stack too; Cassandra, HBase and Redis are very important to us for niche problems. But MySQL is our workhorse and we power many parts of our platform with it.

With the database world finding an equilibrium between relational and NoSQL, I want to share some thoughts on why we still use MySQL and why we always will. I don’t mean to argue against other databases, relational or otherwise. Technical choices should be grounded in business need, and so I will intentionally keep this high-level. We can get into the weeds in another post or on twitter.

To start, it’s economical; that’s to say it offers a nice balance between scalability and performance. Get yourself some modest hardware. Shove in 100s of GBs of diverse data: many tables of all shapes and sizes that are queried together, separately and in all sorts of ways. Now, hammer that system with 100s to 1000s of reads and writes per second. Under said conditions, MySQL can perform brilliantly, with response times in the 10s of milliseconds. I say “can” because it always depends on the circumstances.

And that leads me into my second point: awareness. It’s the devil we, at Sprout Social, know and trust, which I always prefer over the devil I don’t know. We’ve been burned jumping on something shiny and promising only to discover what makes it buckle. And that’s not to say MySQL is flawless. I think of technologies like a musical instrument. You can make any sound bad. When you know how to play the guitar well, why attempt a concert with a banjo? With technology growing out of the pure sciences, we often forget that much of engineering is an art or, at least, heuristics. We humans always perform better in well known circumstances than in newish ones, no matter how well prepared. In the words of Ron Burgundy, “… it’s science.”

My point is, our team knows MySQL a little like an experienced musician knows their instrument. Our developers can concoct powerful, scalable solutions on the spot. Our ops team can churn out operationally sound clusters like it’s their job. As a team, it’s a hammer we swing with finesse. And that lets us get to the problem at hand of building great software. And, sleeping at night.

One of the hardest problems in software engineering is making the engineers themselves efficient. Try not to over-engineer that one.

Building a Better Grunt Plugin

Here at Sprout we’ve been switching over our build process from Phing to Grunt. It was an obvious choice since most of us are mainly JavaScript devs on the front-end side and we’re trying to minimize the amount of PHP we all have to touch.

Grunt Background

Grunt is a task runner framework built on top of Node.js. A Grunt-based build typically involves a Gruntfile.js in the root of your project and a couple Grunt plugins installed via NPM.

Default Plugin Template

The default plugin template is pretty straightforward. It involves a package.json describing your module (like all other NPM modules) and at least a peerDependency on grunt. A typical contrib plugin is structured with a tasks directory containing a single JavaScript file that exports an init function for Grunt.

A Better Plugin Template

I’m not a fan of the single init function. It leaves your code largely untestable and it doesn’t scale well as your task becomes more complicated. I favor a class-based approach that lets us test the code outside of Grunt in an easier fashion.

Here is an example of a structure that I use when creating plugins:

You’ll notice from the structure that we’ve separated out the task registration from the actual work being done. The JSHintTask will now hold the logic for our task, and we can require it independent of the task being registered with Grunt.

Now let’s take a look at the shell of our task class.

We pass the original task in to the constructor, which allows us to grab the files and options. In addition, we’ve added some static fields on the class that should help with registration and testing.

Now all we need in our task registration file is this:

Okay, that’s probably over the top as far as keeping our code separated but I really like keeping that registration file small.

Untested Code is Broken Code

Here is a quick example of how to setup your task for testing.

Wrapping Up

Now we’ve got ourselves setup on some solid footing and ready to grow as our code needs. I’ll be working on getting this template into a grunt-init version soon, so be on the lookout.

Now Playing100 Black Coffins – Rick Ross

We’re hiring Back End Engineers

Sprout Social’s platform team is responsible for processing, storing, analyzing, and enriching millions of messages per day using a myriad of database technologies and languages. We use the latest techniques and tricks, from hadoop to tornado, to build software that is used 24/7/365 globally by names like Yammer, Twilio, McDonald’s and Fender. A typical day has a healthy mix of building new product features, improving existing ones and fixing problems identified by our global user-base. If you’re excited to work with tech like Cassandra, MySQL, Redis, Kafka, Solr, hbase, Java, and Python and are ready to immerse yourself in one of Chicago’s hottest startups, apply at backenddev@sproutsocial.com with your best code samples.

Skills & Requirements

  • You enjoy building software in a variety of languages.
  • You’re on top of the latest in scaling techniques.
  • You build your side projects to be distributed.
  • You want to work in a small, agile team with fast projects and tight deadlines.
  • Unit tests keep you sane.
  • You work hard and don’t need much oversight.
  • You like variety in your projects.
  • You deploy bulletproof code with time to spare.


Pluses

  • Scala, clojure
  • Open Source software contributor
  • Heavy SQL chops

About Sprout Social Inc.

Sprout Social is social media management platform that integrates with Twitter, Facebook Fan Pages, LinkedIn and other networks where consumers are engaging with businesses and brands. In addition to communication tools, Sprout Social offers contact management, competitive insight, lead generation, reporting, analytics and more – all in a package that’s intuitive and easy to use.

Without Sprout Social, an organization using social media is like a whole company sharing an email address. Our product processes 1000s of social media events per second to allow large and small organizations to efficiently interact with their customers. Have you ever tweeted about a product or service and received a prompt reply? That’s the kind of service that our customers strive for and that we enable.

Why else you might want to work at Sprout:

  • Your choice of Mac or PC setup with multiple monitors and whatever accessories you need.
  • Catered lunches, excellent coffee and loads of snacks and drinks
  • Comfortable office with many working environments. Stay heads-down or battle others on our 300 game arcade.
  • Great peers who share your passion for building great products.

We’re hiring Web Engineers

Sprout Social’s web team lives and breathes JavaScript. We use the latest techniques and tricks, from WebSockets to Backbone.js, to build software that is used 24/7/365 globally by names like Yammer, Twilio, McDonald’s and Fender. A typical day has a healthy mix of building new product features, improving existing ones and fixing problems identified by our global user-base. If you walk the JavaScript, HTML, and CSS stack with ease and are ready to immerse yourself in one of Chicago’s hottest startups, apply at frontenddev@sproutsocial.com with your best code samples.

Skills & Requirements

  • You really enjoy building software in Javascript.
  • You’re on top of the latest HTML5/CSS3 and want to use every Modernizr test you can.
  • You have great software design skills.
  • You want to work in a small, agile team with fast projects and tight deadlines.
  • Unit tests keep you sane.
  • You always test for cross-browser compatibility.
  • You work hard and don’t need much oversight.
  • You like variety in your projects.
  • You deploy bulletproof code with time to spare.

Pluses

  • Backbone.js experience
  • Open Source software contributor
  • Samples of websites or code you’ve built

About Sprout Social Inc.

Sprout Social is social media management platform that integrates with Twitter, Facebook Fan Pages, LinkedIn and other networks where consumers are engaging with businesses and brands. In addition to communication tools, Sprout Social offers contact management, competitive insight, lead generation, reporting, analytics and more – all in a package that’s intuitive and easy to use.

Without Sprout Social, an organization using social media is like a whole company sharing an email address. Our product processes 1000s of social media events per second to allow large and small organizations to efficiently interact with their customers. Have you ever tweeted about a product or service and received a prompt reply? That’s the kind of service that our customers strive for and that we enable.

Why else you might want to work at Sprout:

  • Your choice of Mac or PC setup with multiple monitors and whatever accessories you need.
  • Catered lunches, excellent coffee and loads of snacks and drinks
  • Comfortable office with many working environments. Stay heads-down or battle others on our 300 game arcade.
  • Great peers who share your passion for building great products.

JavaScript Time Zone Conversion with Walltime

Time is important.  Keeping accurate time has been one of the major challenges of technology throughout all of history.  Here’s a little history lesson from the US Navy:

Naval Observatory

In 1845, at the request of the Secretary of the Navy, the Observatory installed a time ball atop the 9.6-inch telescope dome. The time ball was dropped every day precisely at Noon, enabling the inhabitants of Washington to set their timepieces. Ships in the Potomac River could also set their clocks before putting to sea. The Observatory’s Time Service was initiated in 1865. A time signal was transmitted via telegraph lines to the Navy Department, and also activated the Washington fire bells at 0700, 1200, and 1800.

For the Navy, time is important for navigating all those ships around the globe.  For Sprout Social, time is important because we have some pretty cool technology that determines the optimal time to send a tweet or Facebook post to get the most reach for our customers.

Time in the Browser

Right now, browsers can give you pretty accurate time information for the time zone currently configured on your computer.  The problem comes in when you need to know what time it is in another time zone than your own.  For instance, if I’m a user in Australia, and I want to post something at 3:00 AM Chicago time, where the majority of my followers are, how do I calculate that accurately with JavaScript and send it up to the server?

Daylight Savings and Olson Files

It turns out, the hard part of these equations is determining the daylight savings offset for a given time zone.  We already have a really easy way to get UTC for any time in JavaScript; the problem is determining if another time zone than our own currently has a daylight savings rule applied.  Otherwise, we could just keep a map of all the time zone offsets and apply the different offsets to our UTC – back and forth.

It turns out, there is a group of files that keep a record of all the daylight savings rules across the world just for this kind of thing:

The tz database, also called the zoneinfo database or IANA Time Zone Database, is a collaborative compilation of information about the world’s time zones, primarily intended for use with computer programs and operating systems. It is sometimes referred to as the Olson database after the founding contributor Arthur David Olson.

The problem is that no one has really implemented using them in JavaScript yet (there is another library called timezone-js, but it turns out they are not applying the rules properly).

Introducing Walltime-js

Walltime makes it easy to convert from one UTC date to the “wall time” of just about any time zone:

In addition, what if we need to convert from a local time zone time in another time zone than your own to a UTC time?  We’ve got you covered there too:

Make sure to check out the source on Github and file any issues you may find.

My Change Control Workflow

My goal as a developer is to deliver a great product in the least amount of time possible without sacrificing quality.  In recent years I’ve refined my workflow to facilitate my needs.  Having a strong workflow is an important factor when trying to produce a high quality deliverable.  I’d like to highlight some of the things I’ve learned along the way that have helped me deliver good code.  That being said, I believe that you should own your workflow.  What may work for you may not seem natural to another developer.  There is no “right” way.  The “right” way to me is the methodology that allows you to create something quickly with high quality and does not create problems for other developers. <   Continued  >

Building a Custom Icon Font for the Web

With mobile devices accounting for over 10% of all Web traffic and hardware manufacturers producing higher pixel density screens, it’s clear that the days of one-size-fits-all icon sprite images are coming to an end. Icon fonts (aka symbol fonts) have grown in popularity as one such replacement for traditional sprite and icon sets because they produce high quality vector shapes without the hefty download and are compatible with most legacy browsers. <   Continued  >