There was a feature to generate some documents and email a zip file on a weekly basis. After a lively discussion, it was decided this was the MVP to solve the need which only impacted one end-user performing business administration tasks. My MVP would have been minimizing maintenance with an index page that the zips gets dumped to, but it was decided we didn’t want to store the files.

It was actually relatively simple to implement and we were leveraging code to do the document generation for us. We tested zipping up 500 documents, sending it and all ran within specifications.

First week in production, the user was reporting that it didn’t arrive. Instantly, ideas will rush in to your head about the common suspects for a scheduled email task failure. It was none of them.

After a good deal of digging around and rerunning the job on a staging server with sporadic failures and annoyingly no backtrace, we ran some queries and discovered that there were over 1000 documents that needed to be generated. Our email attachment limit budgeted for 400. However this was not the problem, but it did lead to the next discovery.

Deep within the document generation code, Ruby Tempfile was being used and the files were never being closed. The OS borked at 1024 open file handles and worse, part of the document generation Kernel.system’ed out which made tracing the proper point of failure difficult. The code actually silently failed until later in the process. Half a day on a refactor.

Running the job on development through Resque, it worked a charm. We cherry picked the fix up on to staging and reran the job. Failed. Head in my hands, I couldn’t help but be mildly amused. Our workers apparently didn’t load the zip library by default in any environment except development. I’m sure there’s a good reason for this, but it was unexpected given that it worked on development. With a simple require directive, the job was on the way and worked.

Reflecting on this, again what failed me were not the task being complex, but my assumptions. The key one being that there weren’t going to be more than 400 documents to generate – however on the first run there was going to be many more than subsequent weeks as one of the criteria for document generation was that for some large set of objects there were documents relating to it which hadn’t yet been generated.

Assumptions are a good thing. They’re useful heuristics that allow you to keep just enough in your head to make reasonable calculations. Just be prepared to peel them back methodically when you’re debugging.

I woke up in the mid morning to a heavily blinded room and crept outside hoping not to wake anyone. I fumbled along the walls in a corridor of a foreign house grasping for a lightswitch which always tends to be lower in these old Sydney houses, but instead my toe found a corner. Minutes of agony later I was sitting in front of a blinding CRT monitor, eyes now the subject of abuse and agony. Emails – eyes scanning the page and my head fluttered. An email from Pivotal Labs. Singular left click. I got the job. My brain worked a while to massage the gravity of the news in. I had a few hours before my unsober companions returned to hazy consciousness and so I sat and thought about it for a while.

I had a pair of shoes to fill. Remarkably, there was already a James Somers working for Pivotal that left shortly before I touched down. I had never met another Somers beyond my family, let alone one that shares my first name and interest in programming. Amidst the amused remarks and tales regaled upon me of his ping pong prowess, I felt a certain odd uncertainty and expectation placed on me (news quickly spread of my subnormal ping pong ability).

I was new to TDD, vim and had essentially worked independently of career programmers for the past few years. My previous job was ambitious. I was embedded in a marketing company automating much of their reporting practices – opening 100MB in Excel was ‘beginning’ to impact the productivity of their analysts. It was perfect while I finished my degree. I learned a vast amount about interacting with people, managing and marketing my product and it was very successful, but always crawling in my mind was the programmer isolation and the great sense that I was not at all honing my development craft or pushing myself to the limits.

My first experiences pairing were something to behold. I was introduced to a whole new style and had to force my mind to synthesise concepts on the fly faster than I ever had to before, attempting to minimise the impact to the velocity of my pair. Concentration is a fundamentally difficult thing to maintain 8 hours a day – but thankfully for the first time since graduating a few months earlier, I wanted to learn again. I wanted to impress and not just witness the impossibly fast minds of my peers. I have all the building blocks to be an excellent programmer and this was the opportunity to prove it.

My first realisations revolved entirely around the nature of pairing. I had by then worked with several Pivots and client developers and seen a wide variety of skills and interests. I worked with programmers that utterly humbled my ego but this was key. I have not met one developer at Pivotal who programmed like me. The ones I admired had me exhausting myself trying to see and predict their next steps and I would adopt and attempt to apply their patterns. Things from both the macro level of testing principles, design patterns right down to the micro of their spacing and brace structure which despite being defined by the project, were still open to stylistic flourishes which mattered to the individual. Each day I would change pair and carefully observe and ask questions.

My most baffling realisation was in how differently each person thought, coded and tested. All of a sudden I was being exposed to bugs, design challenges and concepts that threatened to be just over my head just because in my style of coding I would never have experienced them. That’s not to say my style is perfect at all – rather that I was more familiar with the types of problems my style would expose. Here though, I was struggling to come to grips with why someone had coded things in a certain way. It’s easy to throw up your hands and say it was ridiculous, but the skill comes in tracing why someone developed something in a certain way. I also had to shut off the desire to know every aspect that I was touching for a given problem. I hadn’t developed it all and it was futile to throw myself in at the micro level of every problem. I instead had to learn the peculiar abstractions of another’s mind and use it without caring as much for implementation details unless it was directly related. Undoubtedly, I could be picked out for just as many problems but this is my point. Each of my peers were intelligent beings and proven developers. I even began to see patterns between my pairs where I knew one would disapprove of the others decisions. I became a better developer and am still continuing to be by synthesizing their styles and consciously shaping my own code which leads me to my next point.

Don’t be precious about your style, but use everything as an opportunity to develop it and become more confident in what you believe as a developer. I felt as though I was contributing most to my pair not when I was coding but when I was able to have a logical discussion about an issue. Obviously, this required a certain level of ramping up on the vast domain knowledge of my project.

If you don’t understand something and are anxious about it, remember that everything in programming is logical and can be broken down. If someone is doing a poor job of explaining it to you, it’s likely because they don’t fully understand it themselves or they are having a hard time remembering what it was like. If you’re the senior in a pair, I like to think of times when I had struggled for hours with an assignment or problem thinking about how intangible it all seemed until suddenly something snaps and everything makes complete sense and you wondered how you could have ever thought differently. It’s this tacit knowledge which needs to be shared most. There is a skill in asking the correct questions and taking the initiative to lead and figure things out together too. I’d even say much of programming is a series of slapping the forehead “D’oh” moments as I’ve noticed that even excellent developers can spend a lot of time on problems only to realise it was a subtle assumption that was their downfall.

Finally, learn vim as fast as you can. It’s the closest thing I’ve seen to map programmer brain fragments to a computer. I’ve still got a long way to go but the natural rhythm of the movements in vim closely mimic the movements my brain wants to make which I find very exciting. I can’t imagine going back to another editor.

In my 4th month at Pivotal, I’ve fallen back in love with developing for the sake of developing. My next blog post is half written and contains information for new international Pivots moving to NYC.

Special thanks to these champs for making my first few months settling in at Pivotal a pleasure: Evan Goodberry, JT Archie, Jeff Saracco, Alex Kramer, Dirk Kelly & Trace Wax