How testing exposes this problem

Testing has many forms in the world of software. There are automated acceptance tests, unit tests, functional tests, enemy tests and so on. There are also tests carried out by humans. Sometimes referred to as ‘click testing’ or Quality Assurance, this kind of testing is an essential part of the process of delivering working software. At Pivotal Labs, it’s usually the Product Manager who has the final say about whether a feature is complete, or a bug is fixed. In order to evaluate whether a feature is ready, the PM exercises the area of the application in question, using the interface that a customer or stakeholder has been provided with.

When testing software with a mouse and keyboard becomes difficult, however, it often doesn’t get done. When it doesn’t get done, bugs introduced by diligent programmers, who test-drive their code, are missed and end up making their way into production.

After a few cycles of missing bugs like this, a team will look for ways to ease the pain of click testing their app. The programmers on the team might come up with clever, easy-to-implement solutions to this problem:

• “We can manipulate the Time library to give us the time we want”
• “We can change all of the data for a given account to pretend that it was created in the past”

When these sorts of techniques dig in, new problems arise. In the former case, you can end up with code that works when the fake Time library is used, but not when the real one is used. In the latter, you are committed to a maintenance chore: when a timestamp field is added, it needs to change along with the rest, and when the code that changes those fields gets out of sync with how things really change over time, more bugs arise. It becomes difficult to tell which bugs are genuine and which are a consequence of artificially shifting time data.

It’s usually programmers who propose the above solutions. We often think in low-level terms like libraries and direct data manipulation. From a Product Manager’s perspective, however, what’s really needed is a design change. One could imagine the user story presented to the team like this:

As a Product Manager
I want to travel in time
So that I can test, for example, that an account gets billed each month

Time travel sounds like science fiction. How could a user of your system possibly travel in time? It turns out that there are low-level solutions to this. For example, Timecop, which many of us at Pivotal have contributed to. If depending on the passing of time is your addiction, then Timecop is enabling you. It lets you easily manipulate time, usually for automated testing purposes. For example:

Timecop.freeze(1.month.from_now) do
  future_time = Time.now
  sleep 10
  future_time == Time.now # this is true
end

Here we’ve frozen time to pretend that it’s one month in the future. I can imagine some cases where this would be useful (not least in existing systems that are infected with code coupled to the current time), but in a lot of cases this is just wrong. Under what circumstance do you actually expect your code to be frozen in time? What are the consequences of testing code under these conditions?

Perhaps most importantly to the topic of this post, Timecop lets you forget about managing time at the unit level, and doesn’t encourage you to build time controls into your application.

I think there are better ways to get a grip on time that we should all consider before reaching for the magic wand. Let’s look at some real-world problems that can occur and then look at ways of building time control into an app’s design.

Examining the moment

Let’s think about the properties of the current time:

• There is only one current time, unless you’re modelling multiple realities e.g. storylines about time travel.
• Its value is always changing.
• Everything is potentially affected by it.

So, the current time is a global, constantly mutating singleton. We know that the presence of global singletons is undesirable, because they are polluting. We also know that mutation ought to be contained, because mutable state makes our programs less predictable and harder to reason about. If a function deals with mutable state, then it might have different results each time it is called, even when it apparently has the same inputs.

Let’s look at a timing bug that can result from the fact that the current time has these undesirable properties:

policy = Policy.find(1)

if policy.current_state == :active
  notify_customer("You are still insured!")
end
# more code goes here
if policy.current_state == :inactive
  notify_customer("You are not insured. Hope you weren't planning on driving anywhere today.")
end

Imagine that the above code is within a web request. The request comes in at a certain time, and the customer wants to know whether their insurance policy is current. The code above is deep in the guts of a model somewhere, and gets called after the customer has been authenticated, their request has been authorized, and their account record has been pulled out of the database. Now it’s time to see what the state of the policy is, so we use a method someone wrote (current_state) that fetches the current time and returns a state based on whether the policy’s end date was before or after that time.

The customer sees this on their screen:

You are still insured!
You are not insured. Hope you weren't planning on driving anywhere today.

The policy could potentially be active on one line and inactive the next. This kind of bug gets worse when one line makes an external call if the object were in one state, and the next makes a conflicting call if it were in another.

I recently ran into a real bug similar to this on my current project, which was caught when an acceptance test I was writing would fail on one run and pass on the next. The temporary workaround was to memoize the method that checked the current state (current_state above). Unfortunately, this introduced even more mutable state, because memoization requires changing the state of an instance variable. The next programmer might wonder why fetching the current state works the first time he asks, but stays the same with consecutive calls.

current_nothing

The current_state method is guilty here. But what of? It has a hidden input, which is the current time. It’s not explicit, and that’s where the confusion lies. It wouldn’t make sense to have a method called current_something and have it take the time as an argument, because the prefix “current_” implies that it’s supposed to know what the current state is.

The internal functions of a program shouldn’t know this stuff. In most web apps, a request is made at a certain point in time, but it’s not important that the request takes some time. With most scheduled jobs, the job is run at a certain time, but it’s not important that the job takes some time (or if it is, it’s stored as metadata).

A name less prone to attracting this kind of bug might be state_as_of(time). If we force ourselves to pass the time as a parameter to all of our low-level functions, then we can:

1. More easily unit test the basic correctness of the method without resorting to stubbing out the time with Timecop.
2. Force out a decision to be made about what moment the lower-level methods should be operating on. Ideally the control would move as high as possible: to the controller level, or in the case of jobs, to the job itself, or to an environment variable.

A word about scheduling

Scheduled jobs are often concerned with when they think they’re being run, but a PM doesn’t want to wait for a month to see if, for example, the billing system is working. It’s important to give control over when a job thinks it’s being run to the PM or other person evaluating whether a system works. This might mean dropping your out-of-the-box scheduling interface for the purposes of feature acceptance.

Resque-scheduler has become very popular amongst developers as it’s easy to install and provides a cron-like syntax for declaring when jobs are run. It also provides a GUI for triggering scheduled jobs immediately. Unfortunately, there’s no way to set parameters for the jobs, so the time can’t be set. If you choose to heed the advice in this post and parameterize time, you’ll need to provide your own interface for passing the current time into the system. This is a good idea anyway. See GOOS for a good treatment of externalizing event sources, which goes even further than the suggestions in this post in many ways.

Making time a parameter to your jobs can often make the jobs more reusable. For example, I might want to invalidate all sessions in a particular time range because there was a system fault at those times.

A word about external dependencies

What happens if your external dependencies are dependent on the current time? Well, you’re going to have problems with that no matter how much control you build into your system. I would argue that the services should be wrapped, and the wrappers should allow the time to be passed in to fake responses if necessary. The acceptance (as in story acceptance in Pivotal Tracker) of your system doesn’t have to depend on the state of the other systems.

A word about customer-facing status pages

What if your customer needs to be shown what’s happening right now?
I think the trick here is to depend on the order of time, and not the passing of time. To move into the future, we should be reacting to events by adding data, not mutating it. A database can easily figure out what the latest order is, or what the latest billing cycle is. It doesn’t have to be a function of the current time.

If, however, it’s just too difficult to implement the functionality without checking the current time, it could be argued that the current time be a parameter passed in through the browser, available only in certain testing environments. I would resist this as far as possible, but think that it would be a preferable solution to mutating the state of the database.

The post The passing of time, and all of its crimes. appeared first on Pivotal Labs.

This wasn’t a wild jump into the unknown: I had run various flavours of GNU/Linux since I was a teenager, eventually settling on Ubuntu. I bought a dedicated Ubuntu laptop from Dell when they came out, but caved and bought this MacBook Pro when I could afford it, around 2008. Persuading myself that I needed to run Photoshop for my fledgling freelance career, it struck a good balance between the benefits of the open-source community and the plug-n-play reliability of a popular commercial OS. Development on OSX is pretty easy these days, but there are still some weird issues. Installing Ruby on OSX was non-trivial last time I tried, homebrew eases some dependency installation pain but introduces its own set of maintenance chores, and Xcode is a bit of a dog to keep up to date.

There’s some conventional thinking in the Linux desktop world that says old hardware has a better chance of working. The MacBook Pro 5,5 that I have is a recognisable set of venerable components (“sudo dmidecode -s system-product-name” gives you the version of the Mac hardware you are running). However, the latest Ubuntu 13.04 sets things up so that I:

1. Am greeted with a corrupted boot screen each time I start the computer
2. Can turn my machine into a brick by simply closing the lid
3. See a long horizontal shadow at the top of full-screen videos
4. Am regularly thrown into a different app when I use two-finger scrolling

Granted, this is a far cry from, “my wifi and sound took three weeks to set up and still didn’t work well”, but it’s hardly the smooth experience that Mac users are used to. Here are the fixes.

Corrupted boot screen

Use proprietary nvidia drivers. These work better, but you’re then stuck with a permabright, battery-sucking screen because the keyboard brightness controls don’t work. This can be fixed by getting your hands dirty in a config file (note the older version of Ubuntu this was solved for). Not awesome.

Brick state

This happens because a lid close defaults to suspend, and suspend breaks the open source nvidia drivers. A reboot doesn’t seem to fix it. Kudos for getting so far with reverse engineering, but I wasn’t prompted with an option to choose the proprietary drivers. I’d rather check a box that says “non-free software that works” over “free software that ruins your day”. This prompt used to be present on previous versions of Ubuntu.

You can sometimes recover from the brick state by moving to a regular non-X console (ctrl+alt F1) and killing lightdm with sudo. Otherwise, you’re in boot-from-CD land. How many non-technical types do you know who can / would want to work that out?

Horizontal shadow of panel when running videos full-screen

This one is interesting. The Ubuntu team knew about the problem, but shipped anyway. I can sympathise with the difficulties of maintaining a dual server and desktop OS, and with sticking to a deadline, but if the 80% case could have been solved by loading Workspace Switcher by default, why not do that?

Two-finger scrolling occasionally throws you into another app

This happens because Unity has hardcoded gesture support for touchpads. Three finger double-tap means ‘take me to the previous window’, and three finger tap then tap-and-hold means ‘open a super-sensitive window switcher that I control with the mouse’. I never want to change windows with the mouse.

There are several solutions to this problem floating around, but they all involve modifying the Unity source code and recompiling. This is crazy. Please make gestures optional!

In summary

I had previous experience with dealing with this category of problems, but someone who isn’t a full-time computer nerd might not have so much fun dealing with these bugs. Many of them seem to be reflective of Ubuntu’s confused attitude towards free software. The OS seems to push free drivers onto the user, but they aren’t always ready for prime time. Are end-users being used as testers?

Perhaps I’ll give it a few weeks before I cave and reinstall OSX. Most of the creases are ironed out now, but if I have to spend another few hours googling for solutions to mindless bugs, I may just give in to temptation again. YMMV.

The post I’ll come running back to GNU appeared first on Pivotal Labs.

Act I, Scene I

YOLOPO: The gods have decided that we will send out emails when people haven’t interacted with our service for a while. We’d better make it happen.

ROFLPM: OK we can totally do that. I’ll write a story for it.

Act I, Scene II

ROFLPM: So, we’ve been discussing this story for a while now. We probably need some control over the various factors involved. What do you loldevs think?

LOLDEV: An automatic admin interface is easy to install. It Just Works™. Add a line to your Gemfile and you’re done! Who has the time to write all those repetitive screens anyway?

ROFLPM: Perfect! Now anyone with access to the admin interface has direct access to those models you’ve been writing that do all the work! Rails developers bring so much free stuff.

LOLDEV: Never mind all that. How should we break down this enormous story?

ROFLPM: Well, it’s a biggie, and you loldevs are going to spend too much time away from master before we see any progress, so we’d better break it into three:

1. the first story is about an admin user being able to change a time limit in the automatic admin interface
2. the second story has a customer using the system, and then not using it for the configured amount of time from the first story, and an administrator expecting a value to appear in the automatic admin interface when that time is up
3. the third story has an administrator changing the aforementioned value in the automatic admin interface and a user receiving an email

LOLDEV: Great! That means we can do as YOLOPO wants and parallelize our team to build stuff faster. One pair of loldevs can add the time out setting, another pair can work on getting the value to appear in the automatic admin interface, and yet another pair can work on dispatching the email.

Act II, Scene II

LOLDEV1: Phew! We finally finished that set of stories. It’s a shame we had so many integration problems, but at least it’s done now. Plus, the admin interface is so easy to use that YOLOPO can mess around with it all she likes.

LOLDEV2: Yeah I heard she really likes being able to see all the data and play around with different things.

LOLDEV3: Hmm, there’s something I don’t like about that, but I can’t put my finger on it.

LOLDEV1: What?

Act III, Scene IV

ROFLPM: Hey, remember that feature about sending emails to users when their accounts haven’t been used for an arbitrary amount of time? Well, YOLOPO says it doesn’t work anymore.

LOLDEV: Oh, I think YOLOPO will find that it does.

ROFLPM: Nah-ah! I changed the send_email_soon attribute to true and nothing happened!

LOLDEV: You’re not still using that old thing? We moved email sending to the controllers because we were randomly sending emails to people with active accounts and our test suite was slow and…

ROFLPM: Oh, OK then what do I do now?

LOLDEV: You have to sign up and then pretend it’s the future.

ROFLPM: How do I do that?

LOLDEV: Well, there’s an attribute on every user that you can change in the admin interface.

ROFLPM: Perfect!

The post Admin for nothin’, design for free. appeared first on Pivotal Labs.

I’d heard a bit about Clojure, so decided to explore the ecosystem and community. I’m attracted to the idea of learning a functional language, avoiding mutable state and re-evaluating the object oriented programming paradigm. Functional programmers are often accused of being “idealistic”, “academic” or a variety of other thinly-veiled primitive insults. If nothing else, I could learn to sympathise with these crackpot underdogs who claim to have the cure for most of the technical pain I suffer in my chosen occupation.

Hopefully this assortment of links and lessons learned will encourage you to try out the language, or inspire you to try something else that’s refreshingly new.

Clojure is easy to get started with

There are various tutorials online. However, they can be a little out of date. The quickest route to Clojure hacking is via leiningen, a sort of rake-meets-bundler utility. Assuming you’re on a Mac, have homebrew installed, a recent JDK and the usual code compilation dependencies:

brew install leiningen
lein repl

That’ll pull in Clojure for you. Once you’ve had a play in the REPL, use leiningen to generate an app:

lein new my-test-project

The project.clj that the above command generates is the equivalent of a gemspec, and declares the dependencies of your app. Leiningen helpfully fetches dependencies lazily when you start a repl, server or run tests. No bundle install step!

Homebrew can also install Clojure for you, but this version will be independent of leiningen apps. When you install it, you’ll be recommended to use a REPL wrapper for rlwrap. I’ve found that the one from the wikibooks site works better (i.e. at all).

Rich Hickey is an entertaining figurehead

Rich Hickey, creator of Clojure and designer of Datomic, the FP-friendly database system everyone’s talking about, has some inspiring talks from the end of last year about Clojure and Datomic, and notably why everybody is doing programming wrong. There are some pretty strong views in the talks, but they’re a refreshing take on the sociological patterns in our industry, and how project pathologies can be treated.

There’s an active community

Many clojure developers are ex- or current-Rubyists. For example, Jay Fields maintains the expectations testing library (more on this below).

Clojure has artsy cred, too: there are several videos on the web showing Clojure being used for live music.

Syntax change requires healthy brain re-wiring and thought simplification

After spending so much time wrapping my head around object-oriented constructs and trying to work out how best to structure OO programs, it’s a refreshing change to work with a functional language. Not having a CS degree, I’d previously tinkered with Scheme following a workbook. I wasn’t put off by parentheses, but did have some fears that I’d be transported back to the bad old days of the top-level PHP namespace. I soon realised, however, that my beef with PHP was mainly the aforementioned comingling of state and logic, as well as the inconsistencies of method signatures and names.

Clojure philosophy is all about consistency and reuse. I’ve trained myself to believe that reuse is all about identifying concepts, naming them and adding an abstraction, often a named value object. I’m pretty attached to my explicitly named classes of values, like SubscriptionEvent or PersonName. This is largely in reaction to the recognisable OO smells like Primitive Obsession. The problems inherent in such smells generally melt away in the FP world, since you’re encouraged to use the built-in types to provide equality and the like, and to use the common functions like map, reduce and so forth to achieve what you would do in OO with a value object’s methods.

Web development is potentially full-stack

This is worth knowing if you don’t already: Clojure already compiles to JavaScript in the browser, as well as to Java on the server. So we potentially could be doing single-language full-stack development like node.js but without the oft-cited pitfalls of JavaScript. An excellent resource for newcomers to Clojure is this translation between JS and Clojure.

I’ve begun a project for playing around with the Compojure web routing library. Compojure forms the basis of many Clojure web apps, and I’m using that repository as a place to stick my learnings. I’m test-driving each new feature, so that it might serve as a reference for other newcomers on how to do TDD for Clojure web apps.

TDD is easy, fast and totally not banned.

Watching the Hickey talks, you might come away with the impression that the Clojure community is anti-TDD. However, there are some excellent developments going on in this area, such as the aforementioned Jay Fields’ expectations library, which provides an achingly simple way to express intent in tests. See my first attempts at testing a web app for an example of this. Structural whitespace optional.

Monads still a mystery

One week just wasn’t enough. See if you can figure it out and send me a postcard with a succinct summary.

Until next time!

The post Everything I know about Clojure after one week appeared first on Pivotal Labs.

You can turn on parallelization for a particular test case:

describe Server do
  parallelize_me!
end

or for all tests:

require 'minitest/hell'

As the name implies, the latter approach turns up the test pain level to 11, but it’s the kind of pain that can have positive effects. For ‘fun’, I started to use minitest’s parallelization on a side project, which has a stateful API backed by a relational database. Here are some of the decisions that were forced out by using parallel test examples.

Commitment to fast tests

I thought that running tests in parallel from the start of a project would make me lazy, causing me to neglect slow tests because they’d be running at the same time as others. Surprisingly, the opposite happened: the need to constantly rerun the whole suite to iron out nondeterministic conflicts encouraged me to fix slow tests early. I ended up being able to run the entire suite several times within a matter of seconds in order to check the tests’ ability to run in parallel.

Side note: this is an early-stage project, with a very low quantity of tests! It will be interesting to see how test speed increases as the volume of tests increases.

Avoiding test duplication

Since the unique constraints of my database could be hit by tests with the same fixture data running at the same time, I was encouraged to use more intention revealing test data for each example, avoiding foo and bar, which commonly litter a suite and make tests harder to read.

For IDs, I used Ruby’s SecureRandom library, which provides GUID and hex generation. I sometimes used hex generation when the user-supplied unique display name of something didn’t matter to the test.

Client-side ID generation

Although not strictly forced out from parallel tests, parallel testing got me thinking about how best to interact with the backend, which has a single database being served by multiple concurrent requests (just like any web server).

Using GUIDs instead of autoincrementing IDs can be a smart decision to make if you can (i.e. you don’t need human-friendly URLs), because it means your database server doesn’t need to worry as much about ensuring uniqueness, since the GUID algorithm effectively guarantees it.

TDDing my API from scratch, without external requirements, encouraged me to use GUIDs to simplify the design and to avoid bottlenecks at the database layer. POST requests canonically return the new URL of the resource you’re creating in the Location header of the response. So to test that a thing really got persisted I’d need to:

1. POST to /items with a representation of the resource
2. Grab the Location header of the response to get the new URL
3. GET /items/:newid and ensure the response body matched the representation I sent

This seemed a very laborious process for storing some data. Much less work is:

1. PUT to /items/:newid
2. GET /items/:newid and ensure the response body matched the representation I sent

Since GUIDs can be treated as unique, it didn’t make much sense for the server to generate them.

Positive effect: the app would now cope with a distributed database system on the back-end, despite starting out on a technology that’s thought of as difficult to scale horizontally (SQLite).

Avoiding database resets

It’s common practice to wipe the whole database when starting a new example, or to run each example in a transaction and roll it back when each example finishes. I wanted real black-box tests, so I didn’t want to use transactions. Yet, deleting the whole database at the start of an example didn’t play nice with minitest’s parallelization, since data that one example required would be deleted by another.

The usual approach when using parallel_tests is to create a database for each process. However, since minitest doesn’t manage databases (nor should it) I chose to keep a single database and find a different solution.

I chose to sandbox all of the tests by creating new entities each time, and only checking for output that indicated that particular entity had been worked on. The product I’m working on is a Continuous Integration server, so I’d be creating CI projects (you might also know them as jobs) and expecting them to appear in an XML feed. The tests had to be OK with other data being present, since the other tests could be working too.

This approach precluded tests that checked that the number of records had increased by one, because in an otherwise acceptable “green” test situation they’d occasionally increase by more than one (another test added a record too), stay the same (another test deleted a record) or decrease (more than one had been deleted).

Constraints are fun

While I wouldn’t recommend going rogue like this on a client project, playing with constraints like truly parallel tests can get you thinking about your normal testing procedure. Some of the above decisions allowed for a much faster test execution time, and always having the assumption that other processes could be working on the database forced out some interesting techniques. Some of the techniques I had to avoid due to parallelization would normally necessitate different workarounds with their own drawbacks. For example, if you always assume the count of an ActiveRecord class will go up, you require exclusive use of the database. If instead you scope your queries to a parent entity, this restriction would be removed.

Hell isn’t so bad after all.

The post Designing an API in Hell appeared first on Pivotal Labs.

• “How will you ensure that those bugs don’t creep back in?”
• “Wouldn’t it be nice to know that one change doesn’t break another?”

In conversations between practiced test drivers, though, design topics tend to pop up:

• “What is this test telling us about the design of our code?”
• “Why is this test boring to write?”
• “Why is this test so slow?”

Then there are really exciting questions, when getting close to a design breakthrough, like:

• “Is this test telling us we’re lacking polymorphism in our design?”
• “I’m tired of constructing this thing. How can we group this set of arguments into an object with a name?”

One distinguishing factor between these types of questions is the level of trust in TDD. Someone with little trust might be predisposed to abandon testing before implementation, instead choosing to test afterwards, or not at all. To such a person yet to be sold on the benefits of TDD, the safety questions make more immediate sense, while design questions are often met with blank stares. However, the safety concerns are easily brushed off: it’s a prototype. My team is so smart we don’t need tests. We need to move fast, so we’ll worry about tests later.

Explaining the basic advantages of TDD doesn’t always work as a sales pitch, because those explanations don’t reveal why testing can be difficult, much less why testing sometimes ought to be difficult. Take someone who has never let the design of their code be influenced by tests: they dislike testing for being difficult or boring. Encountering resistance in the TDD process, they choose to forgo the safety advantages of testing, and the design advantages haven’t been made clear.

As you may have gathered, I’m more excited by the design aspect of TDD and related tools than by the safety aspect. I’d like to think that if we sold how TDD can improve the design of code that’s yet to be written, we’d have an easier time tricking our friends into writing code with regression protection.

Learning to listen

There is much talk about “listening to the tests” among TDD practitioners. The listening analogy is apt. Like listening with our ears, the ability to understand what a test tells us about code quality can improve with practice. It’s a subtle concept to grasp, and one I frequently find is not well understood by otherwise experienced developers. This is unfortunate, because it’s a crucial part of getting rapid feedback on the quality of production code. By quality, I’m referring primarily to the ability to cope with changing requirements, as opposed to good coverage of features and edge-cases.

If you can’t hear what your tests are trying to say, there are tools for cranking up the volume. Below are a couple of my favorites. They’re not intended as hard-and-fast rules, but as exercises to try out when you’re frustrated with a test or wondering why it’s getting difficult to test something.

If you haven’t already, you should read about known test smells and their solutions, because we can apparently smell with our TDD ears.

Use your testing framework’s convenience helpers sparingly

In the RSpec world, this often comes down to writing readable examples without using ‘subject’, ‘let’ or ‘before’. It turns out that straightforward assignment is usually OK.

As this Thoughtbot post argues, the let helper effectively introduces Mystery Guests (implicit, hidden fixtures), and overuse results in slow and fragile tests.

I like to avoid lets, subjects and other test helpers for another reason: if I can’t stand to repeat myself in examples, I think about how the code that uses my code will feel. A boring, repetitive test setup might be telling me that my code has too many dependencies. If I’m frantically stuffing things into the database and stubbing out web service requests just to allow myself to construct an object, perhaps the object’s scope is too broad.

If you come across a test that is apparently repetitive, consider tidying the implementation of the system under test before the test itself. You may find that the noise in the test can be dramatically reduced with some production code tweaks.

Avoid stubbing methods to return values

I owe this one to Greg Moeck, who introduced something like it at the San Francisco eXtreme Tuesday Club.

First, a reminder of the definition of stubs versus mocks (to paraphrase Gerard Meszaros):

1. A stub is a test double that allows you to control the indirect inputs of the system under test.
2. A mock is a test double that allows you to test the indirect outputs of the system under test.

If you return a value from a stubbed method, you force your production code to depend on a blocking, synchronous call. If you could otherwise send a message and not expect an immediate response, you permit your design to (now or in the future) be asynchronous.

Further to that, if you instead use a mock to expect an output to the collaborator you were previously stubbing, you can more cleanly divide your testing into inputs and outputs of the system under test. It’s the difference between:

it "ensures user is authentic before performing the action" do
  user = stub('user')
  authenticator = stub('authenticator')
  authenticator.stub(:authentic_user?).with(user) { true }
  action = Action.new(user)
  action.perform
  expect(action).to be_complete
end

and:

it "ensures the user is authentic when action is requested" do
  user = stub('user')
  authenticator = mock('authenticator') # assume the player of this role knows who to tell when authentication succeeds or fails
  authenticator.should_receive(:authenticate_user).with(user)
  action = Action.new(user)
  action.perform
end

it "performs an action once a user has been authenticated" do
  action = Action.new(stub('unauthenticated user'))
  authenticated_user = stub('user')
  action.user_successfully_authenticated(user)
  expect(action).to be_complete
end

The code that passes the second set of examples is in better shape for when you need to queue requests to the authenticator and complete the action asynchronously. It uses a “tell, don’t ask” style. The fact that an explicit message is sent to the system under test (‘user_successfully_authenticated’) makes it clear to the reader that the request for authentication and the triggering of the action are separate bits of work. It’s someone else’s business whether I get told about the successful authentication, and how many steps are taken before I’m told.

There are several more techniques I’d like to tell you about, but this post is getting a bit long in the tooth. Maybe next time. Happy listening!

The post Smelling with your ears: TDD techniques to influence your design appeared first on Pivotal Labs.

I was going to blog this week about the latest improvements I’d made to the code, and how it was going to improve the end-user’s experience despite being a mere refactor and a minor user interface tweak. I got bored after writing the first few paragraphs, and instead got distracted by the code, refactoring and tweaking it more than was necessary. A couple of hours in, I’d totally gold-plated the code in a way that I might have deemed unacceptable whilst on client time. This made me a little angry with myself: why was I procrastinating from blogging, and was I a bad developer who’d lost the ability to work alone?

Something we’re encouraged to do at Pivotal Labs is reflect on our behavior. Thinking about my boredom and procrastination a little, I realized there was something more interesting going on. The process went a bit like this:

1. Start blogging about the new features. Paste the visual output of the program into the blog post. Realize that the output didn’t meet the requirement of improving the user experience.
2. Start to implement the missing parts of the feature. Spend hours enjoying the freedom to refactor, delete and re-implement with no time limit, far more than when pairing on client time.

When we pair, we have a safety net to catch us when we get distracted by neat language features or elegant implementation tricks. Our partner will often remind us that we’ve spent, say, two hours making no discernible feature changes and no improvement to maintainability of the code. When soloing, however, especially on pet projects, we are free to choose a goal for the activity. Sometimes we decide to perform code katas, but other times we don’t consciously choose a goal: the goal could become apparent during the activity itself.

So it seemed that I’d accidentally found validation in what I was doing, and a potentially more interesting blog post at the same time. I’d managed to get some programming exercise in instead of banging out a not-quite-earth-shattering new feature. Specifically, I:

1. Thought like a user until it became obvious that changes were still needed. Switched to developer mode by accident.
2. Allowed myself to try out several approaches to the Presenter pattern, applying them to a trivial example that would not need a presenter in ‘real world’ programming.
3. Used the Introduce Null Object refactor in a situation that didn’t call for it. Yet, it felt good and might even prove useful for the project later.
4. Practiced several coding techniques that might become useful in the work environment.
5. Temporarily inverted my work-based insecurities about performance and timeliness, providing stress relief as an unexpected benefit of my brain switching itself off from the task at hand.

Reflection is a useful technique for improving well-being. Your mileage may vary. If you’d have preferred to read about the changes made to Scrubber this week, you can read the commit history!

The post Procrastination, considered. appeared first on Pivotal Labs.

When test pollution builds up, it can mean that a project’s build fails unpredictably, which can stop a whole team from shipping code regularly. This is an expensive way to not build software.

Here’s an example of test pollution. You can save and run it with Ruby if you like. You shouldn’t need anything but a recent version of Ruby. If you run it several times, it will sometimes fail and sometimes pass:

Why is it so unreliable? MiniTest orders tests randomly by default. When the first test runs before the second test, the test case fails, because the first test has the effect of setting the @logged_in instance variable to true, and the second test is effectively expecting the value of @logged_in to be false. The code under test has global state: the class instance variable, @logged_in.

The problem with the first test is that it’s a bad citizen: it sets global state and doesn’t clean itself up. The problem with the second test is that it’s presumptuous: it relies on global state being something in particular. As an aside: the code is terrible, and the flakiness of these tests should prompt you to change it, but I used a contrived example for the purpose of demonstration.

Fighting pollution with existing tools

I mentioned that MiniTest orders tests randomly by default. This is a Good Thing: it’s a deliberate ploy to flush out test pollution. If you ran the above code and it failed, it would give you a ‘seed’ number to pass in to the test, so that you could consistently run the test in the failing order. From there, you could hopefully work out why the particular order of tests failed. Both RSpec and MiniTest allow you to run tests in a random order, and both allow you to re-use the order from a previous failed run.

These aspects of MiniTest and RSpec are useful. They allow you to fix the order, so that you can find the dirty polluters of your test suite. This is often a quest to find two items: a polluter and a polluted test. Often you’ll find more than two, or a strange combination of tests that, when run in a particular order, cause a failed build.

On larger projects, these tools aren’t enough. Large codebases tend to have correspondingly large, slow test suites. Finding a source of test pollution in such beasts can involve looking at a lot of code, and can take a very long time.

Why can’t we automate this?

We’ve seen above that tests are reorderable chunks of code. So, you’d think that digging through reams of code just to reduce a pass/fail situation down to two examples would be an automatable process. You’d be right, but it’s not as straightforward as it ought to be.

When my pair and I first set out to write a tool to reduce tests down to their polluting / polluted components we thought we could get RSpec to output a list of files. We could then just feed the list of files back into RSpec in the order in which they failed, and use a binary search to find the offending files.

Unfortunately, it’s not that easy. RSpec isn’t file-centric, but groups-and-examples-centric. For the uninitiated, an RSpec test is composed of groups and examples:

Once test files are processed, they’re loaded into memory and randomized regardless of file (but keeping group hierarchy intact). RSpec’s randomization algorithm is pretty simplistic:

1. Programmer: Hey RSpec, run the suite with this seed: 123
2. RSpec: I found a set of sibling groups. What should I do?
3. Randomizer: ‘Randomize’ them according to the number of items in the set, with this seed: 123
4. RSpec: OK, I’ve got a set of sibling examples inside one of the groups. What should I do?
5. Randomizer: ‘Randomize’ them according to the number of items in the set, with this seed: 123

And so on. This approach is great so long as you don’t want to reduce the problem set. If you do reduce the set of examples that RSpec is running, the order is lost, because the number of items in the list changes.

Enter The Scrubber

I’d ideally like to tell a computer to go and find my test pollution. I’m some of the way there. So far, I’ve managed to create a semi-automatic solution: get RSpec to output the order of a test run to a human-readable file, so that it can be edited and fed back into RSpec to order the next run.

Scrubber is a project I started last week that allows you to persist RSpec run orders, edit them, and replay them. It relies on a relatively new feature of RSpec that allows you to define custom ordering strategies. These strategies are just blocks of code that take as an input the list of groups or examples in a particular section of your suite, and return the groups or examples you want, in the order you want them.

The main stumbling block when writing the utility has been deriving a unique ID from each example or example group. The ID needs to be human readable and also reproducible across runs. So far I have a simplistic solution that just dumps the group or example description, and file location. This isn’t very unique, especially considering that RSpec has no restriction on having groups or examples with duplicate descriptions.

In future, I hope to use a more robust ID, perhaps a checksum of the example/group’s metadata. Again, this isn’t straightforward as some of the metadata are Proc objects. Perhaps the RSpec team would be interested in persistable suite runs as a core feature?

Anyway, here’s an example of Scrubber in use. If you’re interested, I suggest you clone the repo and have a play with the example to see how editing a file might work. It’s very rough around the edges right now, but serves as a proof of concept. Hopefully I’ll get enough time, or contributors, to make this into something more automated and user-friendly.

The post Fighting test pollution with an RSpec custom ordering strategy appeared first on Pivotal Labs.

• New Mail Rack

There’s a new mail rack near the lockers.

• Capybara inconsistent about what page it’s on

We have a spec that does something along the lines of

visit '/#foo'
visit '/#bar'
save_and_open_page
page.should have_content("bar")

The save_and_open_page gives the bar page, but the assertion fails because it looks for "bar" in the contents of the foo page rather than the bar page.

One suggestion was to tell capybara to resync.

• Webviews for iOS app – slow?

One team was worried about the speed of Webviews versus the speed of e.g. mobile Safari. It was suggested to try it and see, benchmark even.

Interestings

• Mad at your Rails logs? Try Lograge

https://github.com/roidrage/lograge uses the amazing notification system in Rails 3 – which itself is worth a read – to make more useful logs for staging & production environments. Looks like it came out of the TravisCI project. And if you’re logging to Splunk, this is more like what you want.

• Jasmine busted after Firefox upgrade?

Upgrade the selenium-webdriver gem to 2.25.0 and happiness will prevail.

The post Rager Party (Standup 7/31/12) appeared first on Pivotal Labs.

“How do I avoid compiling gems on production servers?”

Apparently there’s a project that Chef uses that packages dependencies into an installer.

Also, FPM is potentially a good solution.

Interesting

Survey: who’s actually deployed Carrierwave?

2 people said they had.

The post SF Standup – March 1st, 2012: so who’s deployed Carrierwave? appeared first on Pivotal Labs.