“Rails is slow, but Rails tests are slower.” Rails may be slow, but I’m here tell you that it’s likely you have only yourself to blame for your slow test suite.

I’ve seen some bad test suites in my day. I was once pulled onto a rescue project that had a total build time of over 24 hours. The health and future of your production code depends on a fast, reliable test suite. On my current project, we believe in fast feedback and we’ve focused a lot of detail and attention to our test suite. It’s paid off in dividends.

We’re six months into a project, and we have a one-minute test suite. That includes both rspec and cucumber (and the time to boot up rails for both). Here’s how we’ve accomplished it:

1. Pyramid

Your test suite is a pyramid. Skinny at the top (journey acceptance tests), fat at the bottom (unit tests). Conceptually, this sounds right, and it’s easy to think that it should just naturally fall out of the BDD outside-in process.

But in reality, it’s not a given. Tools like cucumber make it so easy to spin up new high-level acceptance tests for every edge case you can think of. Taken to an extreme, this can lead to unit-testing at the browser level. That doesn’t mean you were wrong to drive out exceptional paths at the acceptance level. But you should ask yourself this question before you check in: “Will I have any less confidance in my test suite if I don’t check in this acceptance test?” If you have already covered a happy path of a feature, and at least one exceptional path of a feature at the acceptance level, do you then need to cover every other exceptional case at the acceptance level? If you’ve added tests down at lower levels in your stack, then you might already have all the verification value you need.

2. Grooming

We groom our test suite. Not every day. But we watch it and keep it in order. We listen to pain in our test suite (“Why did that change cause so many other tests to break? And why did I have to go to every single test and fix each one of them individually?”). When we revisit old tests, we reconsider it in light of what we know about our application. We keep an eye out for duplicate tests.

3. Refactoring

Our fast test suite practically begs us to refactor. And it turns out, refactoring often leads to simpler designs, objects with fewer methods (and therefore fewer tests), as well as objects that are easier and more natural to test in isolation. Great tests will lead you to refactor your production code, which will lead to even better tests.

4. ActiveRecord Containment

How many tests require you to create data in the database? Ideally, that’s limited to just your acceptance tests and your activerecord model tests. But the reality is never that simple. Although it’s technically possible to accomplish this with any Rails application, it’s not always feasible, or even desired. Libraries you use may make this difficult in some circumstances (e.g., devise).

You might also find yourself in a situation where testing an object in isolation would lead to a very brittle test that knows the entire implementation via stubs and mocks. Worse, you may not see a way to untangle these dependencies. That’s OK. Test the object’s behavior, even if it means integrating with other objects. As your understanding of your application’s problem domain crystallizes, and as more patterns begin to emerge, you will eventually find a way to simplify it, to untangle the dependencies.

5. GC.disable

At one point on our project, our build time crept up to nearly two minutes. We saw ourselves slipping down a very slippery slope. We started running it less, refactoring less, and even finding less motivation to keep our test suite clean. So we threw a chore at the top of our backlog to bring the test suite time back down to a minute. Our PM was naturally wary, asking us to timebox the chore to an hour or two, but we made the case that this was of critical importance to the future of the project, and that the pair working on it would check with the rest of the team after a day.

Near the end of the day, the pair stopped and told the team that they thought they had done all they could, but it only got the test suite down to 1:20. Shaving forty seconds off a test suite in a day is a fantastic feat, and when we looked at what they’d done, we were really impressed. I jokingly suggested disabling GC. And then they actually did it. The test suite time dropped to 48 seconds. I’m not recommending you do this. This is a bit of a nuclear option. If this is the first thing you try to improve your test suite time, then you are missing the point of bringing your test suite time down. But if you feel like you’ve done everything you can to legitimately bring down the time of your test suite, then consider disabling garbage collection. Ruby GC is a beast, with the potential of turning a linear algorithm that creates N objects into a quadratic algorithm. Weigh the pros and cons, and decide for yourself if you can live with the dirty dirty feeling this will give you.

Pairing is great for knowledge transfer.  With a reasonably sized 3 pair project, you can completely avoid code reviews, training sessions, tutorials, or any other means of teaching a developer how something works.  I had a good amount of time on a 6 pair project with a ~15 developer-year code base and felt a lot of pain with poor knowledge transfer during that time.  Pairing just wasn’t enough.  We experimented with a few additional ideas during that time and found four that I would certainly use again on any future large project.  The parking lot, a pairing whiteboard, frequent technical retros, and developer documentation.  If your project is big enough, you might want to give these a shot.

Parking Lot

This has been used on a number of other Pivotal projects, but worth reinforcing.  As you are working through the day, if you ever create something other developers should use in the future (a helper method, or useful mixin, or new jasmine matcher, etc), write it on a whiteboard.  This goes double if there is something old that developers should have been using but didn’t know about.  I’ve found I typically write one thing per day.  At the beginning of standup (or perhaps before standup, with only the devs), go over all of the items on the whiteboard.  The same info could be contained in the standup itself, but there is a lot more useful info transferred this way in practice.

Cost:  A few minutes a day, less than the cost of standup.  The ask ticket for the whiteboard.

Pairing Whiteboard

Also taken from other projects.  We found that if pair rotation is basically random, you have very little control over what you work on and therefore very little control over what parts of the code you know.  We have every person’s name on a post-it note and write down every story that is carried over from the day before on a whiteboard with the owner’s post-it next to the story.  This goes for stories that weren’t started the day before but clearly need to be started today.  You then assign pairs based on who needs to learn what.  You can also use the same mechanism to create power pairs if the situation calls for it, but that doesn’t help knowledge transfer.

Cost:  Same as the parking lot, plus some post it notes.

Technical Retros

I used to think technical retros were only when the whole code base is about to fall apart or you need make a major infrastructure decision and want the whole team involved.  Developers know how to solve technical problems and we don’t need meetings to do this. It turns out these retros are great for knowledge transfer.  We only did this for a month or so before our team size was reduced for other reasons, but I’d recommend having technical retros on big teams once a week until you feel they aren’t helping and then dial back.

I have observed three mechanisms for this knowledge transfer.  If someone has been working on a component they don’t understand, this will end up in the unhappy column and now everyone knows that component needs more explanation.  If someone knows a component and thinks it should be changed, everyone else gets to listen in when the two people who actually understand it discuss it.  Lastly, when people discuss changes to the infrastructure, it ends up including good explanation of how the current infrastructure works.

Cost:   An hour-long meeting every week with the whole team is a lot.  Only really worth it on large teams.

Developer Documentation

I’ve noticed we treat documentation at Pivotal the same way we treat code comments, lies waiting to happen.  What is the point of writing down how some API is going to work when you already have tests describing the behavior and the API is just going to change next month?  This may be true for low-level documentation, but we have drawn a lot of value from higher-level documentation, showing general code flow and including examples of how pieces fit together.  We originally wrote the documentation because we are open source and needed to help out some third party devs, but it turned invaluable as a teaching resource.  Our documentation is split into two parts, one with info on how the code works (here) and a second “DevOps Manual” with info on dealing with our development and testing setup.  The DevOps manual is in a private repo.

Documentation writing was prioritized in tracker.  There were many stories “Document X” that built things up over time.  The stories were usually the result of action items from retros (technical or process) whenever a knowledge silo is identified.  Many pages were written by a pair with one experienced person and one new person.  This has the benefit of teaching the new person and forcing the experienced person to write something a new person can understand.

None of this addressed the ‘lies waiting to happen’ problem of documentation.  This is why we go over it all with each new developer on the project.  After a new developer has had some time to get their bearings, we put a chore in tracker to go over documentation with them (only if you are pairing with them of course).  The other part of that chore is to fix anything that is wrong.  This way every developer gets a baseline starting point and the documentation stays up to date.

Independent of knowledge transfer, the DevOps manual is pure gold.  It pays for itself as soon as something like “The VM running my third party database for development ran out of hard drive space and corrupted the database” happens more than once.  Whenever you fix something like that, even if you think it can’t possibly happen again, write it down.  When you set up that VM in the first place, write that down too.

One last note on documentation, even if you don’t use actual documents, it helps a lot to document in the code itself.  Detailed commit messages, detailed descriptions in tests, even code comments themselves occasionally.

Cost:  We have probably spent at a pair week or two writing our docs.  Maybe not worth it on a small project, but not too painful.  The DevOps manual itself would still be valuable on most projects.

In Part 2, we finished defining all the formal terms and symbols you see in the StackOverflow question on the Hindley-Milner algorithm, so now we’re ready to translate what that question was asking about, namely the rules for deducing statements about type inference. Let’s get down to it!

The rules for deducing statements about type inference

Read on at my blog (since these blogs don’t support MathJax) →

In Part 1, we said what the building blocks of the Hindley-Milner formalization would be, and in this post we’ll thoroughly define them, and actually formulate the formalization:

Formalizing the concept of an expression

We’ll give a recursive definition of what an expression is; in other words, we’ll state what the most basic kind of expression is, we’ll say how to create new, more complex expressions out of existing expressions, and we’ll say that only things made in this way are valid expressions.

Read on at my blog (since these blogs don’t support MathJax) →

Mike Sierchio wrote a cool post on password strength, and the concept of entropy. As he points out, entropy isn’t always entropy. That confusion is apparently not uncommon, as it’s been asked about on IT Security Stack Exchange as well. So what’s really going on?

Let’s step back for a sec and fill in some context. What are we trying to do? We’d like some way to measure how hard it is to guess our passwords, a number that serves as a heuristic standard of password strength. But there are two fundamentally different things we might want to measure:

1. How hard would it be for someone to guess your password with essentially no knowledge of how you created your password?

2. How hard would it be for someone to guess your password if they knew the process used to generate it? This is of course assuming that there is a process, for example some script that does some Math.rand-ing and produces a password string.

The term “entropy” has been used to refer to both kinds of calculations, but they’re clearly entirely different things: the former essentially takes a string as input, the latter takes a random process as input. Hence, “entropy is not entropy.”

Alright, well if entropy isn’t entropy, let’s see what entropies are. We’ll look at the standard mathematical formulation of the random-process-entropy which comes from information theory. And we’ll look at the function used to calculate particular-string-entropy in one of the most popular password strength testers. And that’s all we’re going to do, we’ll look at how the calculations are done, without dwelling too much on the differences between the two approaches or what their use cases are.

Read on at my blog (since these blogs don’t support MathJax) →