The Pivotal News Network has been going strong for six months (Pivots: talk to me if you’d like to share into the feed). Here are some highlights from May:
When starting any software project, there’s an age old argument: should we build something simple that solves our current problem or should we use an existing product that’s more complex, but more feature rich, since we know that’s where we’re going to end up in the future?
an oft neglected repercussion of building too much too quickly is that the extra functionality can calcify your product and make it very rigid. Releases become more complex, new features take longer to implement and bugs take longer to fix. You can find yourself a prisoner of your product, maintaining functionality and features that no one ( or very few ) people use. It can demoralize a engineering team, making them more and more susceptible to the nuclear option: the big rewrite.
I think the tendency to lean towards a more exhaustive solution upfront comes from a time when the effort require to change software was much higher than it is today. When systems were written in C, C++, Perl or even Java, making changes was a large undertaking. The thought of possibly throwing away chunks of code was nerve racking. It represented a huge investment in time and money. However, with todays rapid development languages and frameworks like Ruby/Rails & Python/Django, the investment required to create something, both in time and money, is rapidly shrinking.
Jeff [Patton]’s reply shocked me:
“The Ruby community cares about building high-quality apps, but doesn’t necessarily care about shipping high-value apps.”
Jeff went on to say that the Ruby community is obsessive about craftsmanship. This is a good thing, of course. We test. We write clean code. We take the time and care to build applications that are beautiful and do what our customers ask for.
Therein lies the rub: what customers ask for is rarely what they want, and almost never what they need. As Henry Ford put it, “If I had asked what people wanted, they would have said faster horses.” Or as I put it, your customer may pay you $1000 to deliver him a knuckle sandwich, but no amount of precision or strength training is going to leave you with a happy customer.
It turns out that constructing a high-quality application is not enough – you have to conceptualize and design an application that users will actually find useful. Doing this is every bit as difficult as constructing the software, if not harder. It requires a combination of research – generating new ideas from asking questions & identifying problems – and feedback – testing out ideas you’ve created. The Ruby & Agile worlds have been primarily focused on getting user feedback, without doing the all-important research.
Weeks ago, some people in the Ubuntu community got a bit disappointed with the distribution’s core team:
We are supposed to be a community, we all use Ubuntu and contribute
to it, and we deserve some respect regarding these kind of decisions.
We all make Ubuntu together, or is it a big lie?
We all make Ubuntu, but we do not all make all of it. In other words, we delegate well. We have a kernel team, and they make kernel decisions. You don’t get to make kernel decisions unless you’re in that kernel team. You can file bugs and comment, and engage, but you don’t get to second-guess their decisions. We have a security team. They get to make decisions about security. You don’t get to see a lot of what they see unless you’re on that team. We have processes to help make sure we’re doing a good job of delegation, but being an open community is not the same as saying everybody has a say in everything.
- from Velocity as a Goal
From my experience having velocity as a goal doesn’t make any difference to the motivation of the team which is often cited as the reason for referring to it as a target.
In all the teams I’ve worked on people are giving their best effort anyway so they can only really have an impact on the velocity by doing one of the following:
- Working longer hours
- Cutting corners on quality (by less testing perhaps)
- Finding a smarter way of working
In reality I haven’t noticed that people on the teams I’ve worked on pay that much attention to whether velocity is considered a target or not. People just do their job and we pretty much always have the same velocity each week regardless.
More popular shared items:
belongs_to associations can now automatically create back references each other, thanks to a Backport of :inverse_of from Rails 3 to rails 2.3.6. This allows us to keep our object graphs more correct and avoid situations where we have 2 copies of the same object because the object graph is walked in reverse. Here’s how to use it:
class Parent < ActiveRecord::Base
has_one :child, :inverse_of => :parent
class Child < ActiveRecord::Base
Ask for Help
A pivot asked “what is the current state of the art in scheduling recurring processes?”
The first-order answer was simply “cron”, but then the conversation got interesting.
Cron has a few downsides-
- Each task execution has to re-load the entire ruby/rails runtime, so, you pay a significant penalty in terms of startup time
- Crontabs often don’t get checked into source control, so there ends up being little visibility into which jobs are running when
One suggestion to solve the visibility problem was to use the whatever gem, which allows you to express your cron schedules in a ruby DSL that can easily be kept in source control.
A suggested alternative that eliminates cron altogether is resque-scheduler with resque.
The upsides with resque-based scheduling are that all your schedule logic is expressed in ruby, and you don’t pay the ruby/rails startup penalty for each worker.
The downside is that it adds additional operational infrastructure for you to manage (the resque workers and the redis server(s)).
We’ll be making some DNS changes that affect Pivotal Tracker this Saturday, May 29, at 10:00am PDT. We expect a fairly short outage, but for some users, Tracker may be unavailable for up to 6 hours.
Ask for Help
Identify and uploaded files
Running identify on uploaded files sometimes fails, because by the time identify gets hold of the file, the file extension has been lost (it has been written to
/tmp or the like).
The suggested workaround was to see if the filename specified in the
Content-Disposition header, if present, could be carried through to the temporary file.
Amazon has recently introduced a reduced-durabilty pricing tier for S3 storage, with significant discounts available to those who can live with 4 9’s versus 11 9’s of reliability. This seems like a win for things like thumbnails, which can easily be re-created from the originals on the off chance that they were actually lost by S3.
Rails 2.3.7 is out, but it doesn’t play nice with HAML just yet. Projects using HAML should avoid upgrading until those incompatibilities are ironed out.
- One of our clients had a large production issue due to a long-standing bug in Rails with case sensitivity.
Here’s the situation: Rails validates_uniqueness_of has a flag called :case_sensitive. This flag defaults to ‘true’, but can be flipped.
MySQL’s default collation is case-insensitive. As a result, queries will, in general, ignore case unless specifically overridden.
So one might imagine that setting :case_sensitive to false would be completely harmless in a standard MySQL application.
One would be wrong. Setting case_sensitive to false changes the query to lowercase the field in question, causing the MySQL database to ignore any indices it may have and turning the validates_uniqueness_of operation from something cheap and quick to something requiring a full table scan.
The open Lighthouse ticket on this issue is:
Ask for Help
“Any clever ways to catch out of bounds exceptions from Solr?”
This is a follow-up to yesterday’s Solr question. After some investigation, it looks like none of the major providers catch out of bound exceptions for very large numbers. Rather than instrumenting every Ruby call with validations to prevent these numbers from getting into Solr, are there any other brilliant ideas?
Goto File + line #: If you use ctrl-shift-N to go to a file, try typing in a line number after a colon, something like “my_file:30″. You’ll end up on that line.
Analyze stack trace: This tool lets you paste in an external stack trace, and gives you the ability to browse to all of the pieces of that stack trace.
At Pivotal Labs, one of the services we provide is helping clients interview and hire. Pivotal Labs and our clients place a strong emphasis on Agile development and its many aspects: test-driven development, rapid iterations, frequent refactoring, etc.
Here is a job posting from Revelation, a former Pivotal client looking for a lead Rails developer.
Rails Developer – Portland, OR
Revelation creates tools that help people understand other people. Our offerings are used for market research – helping companies better understand their customers to make better things. Our customers include some of the world’s leading research and consumer companies. Our culture is one that encourages the intellectually curious and intensely creative.
Are you entrepreneurial in spirit? Do you have the desire to not only improve product, but improve process and tools? Are you passionate about test-driven and agile development? And most important of all, are you ready to make the leap from skilled developer to leading a rails development team?
Come add your experience and thought-leadership to an incredible team. We are looking for someone with great code and people skills, who wants to take on a leading role in the development of our product and our practices. Reporting to the CTO, this position will be responsible not only for developing a great product, but also helping to create a solid foundation for an expanding team.
Candidates need to come to the table with the following:
- 3-5 years of experience building web applications
- Strong background with ruby and rails
- Experience with relational and non-relational data modeling
- Experience with virtualized environments (Xen/EC2)
- Experience with test-driven development and agile software practices
- Strong interpersonal communication skills
Even better if you have:
- An active github account with contributions to open-source projects
- Experience with Objective-C/iPhone and mobile development
We offer generous stock options, health insurance with vision coverage and a great work environment. If being a key part of the core team of a young, growing company is appealing to you, then let’s talk.
Please email resume and cover letter to firstname.lastname@example.org
Ask for Help
“Our site requires crafting URLs in a very particular SEO-friendly way. Rails doesn’t seem to give us a good solution for our URLs. Any suggestions?”
One of our clients needs to make their app accept and generate compound URLs that look something like the following:
where author, series, and book are all different domain concepts. Rails RESTful resources don’t really support this format. There wasn’t an immediate solution, but among the peanut gallery of ideas:
Hyphens are better than slashes in URL crafting, but Rails doesn’t separate on slashes at all
to_param solutions – Overriding to_param to something that starts with an integer ID generates URLs that look very slug-like, but can use standard Rails Domain.find mechanisms. For example, a book.to_param might be overridden to be “1-bookname”, which works for all purposes. The problem with this solution is that it doesn’t quite fit the requirements here, and doesn’t cover the compound needs.
Custom routes are always a possibility. You can hook up a special (non-resource) controller that understands flexible browse-y routes like the one above, parses them, and delegates to the more standard resource controllers. The problem here is that you have to figure out a decent delegate pattern and route generation pattern.
In general, URL crafting is a separate art from domain model crafting, and Rails doesn’t really cater to this. You will have to design URL-centric code to suit your URL crafting.
“Any ideas on ways to performance test IE7?”
No immediate ideas, but potentially more later.
“When users enter very large search parameters for numbers we get the following exception out of RSolr:”
RSolr::RequestError: Solr Response: For_input_string_11111111111111111111__javalangNumberFormatException
Is there an elegant solution to this aside from validating that all input parameters aren’t larger than max int?
When using named scope methods that refer to other named scope methods, you may discover that your SQL has some redundant condition clauses. This is a bug in Rails 2, and has been true for several versions. However, it’s a harmless bug – MySQL will understand the extraneous condition clauses just fine, without performance implications.
Mocking Paperclip for tests is a careful art. See our other blog post: Stubbing out Paperclip ImageMagick in Tests
Many people use the ultra popular Paperclip library to handle file attachments in Rails. Unfortunately the Paperclip documentation does not cover how to stub out calls to ImageMagick in your test suite. Without the proper stubs in place a test suite that uses Paperclip will take much, much longer to run.
In the grease your suite presentation by Nick Gauthier it has a slide titled Quickerclip that describes what needs to be done to spend up Paperclip in tests, basically you need to keep it from shelling out to ImageMagick. Alas, the presentation does include code for how to achieve Quickerclip.
As the presentation shows Paperclip.run is the method that needs to be changed. The first parameter passed to Paperclip.run is the ImageMagick command be executed. Paperclip uses the identify and convert commands. The
identify command is used to determine the dimensions of an image. The
convert command is the really heavy one that does image manipulation and thumbnail generation. Here is a redefinition of Paperclip.run with sensible behavior for tests.
def self.run cmd, params = "", expected_outcodes = 0
post_process in Paperclip::Attachment is an optional additional optimization. In Paperclip,
post_process eventually calls
Paperclip.run("convert") and by short-circuiting the method earlier in the chain we save a few cycles.
- Keep your OAuth Nonce values simple.
The Twitter Api, which is requiring all clients to move to OAuth for authentication by June 30, 2010, like all OAuth systems requires a nonce value for every call. This value is supposed to be random and unique for each request you make.
While there are many ways to generate a random ASCII value, our recent experience with Twitter’s OAuth system shows that a nonce value should not include a ‘%’ character – which would happen if your value has any non-URL-safe character. Twitter will return you a 401 error and tell you that your signature and token cannot be verified.
We’ve filed a bug with Twitter. But until then, keep your nonce value to ASCII letters & numbers and the calls will work just fine.