This also makes the generated sprite really small which is great if you care about page load times. A Ruby+ImageMagick sprite generator might also be a good thing to build.
“Is there an onReady() for AJAX events?”
We ran into an odd bug using FasterCSV to import some data. We were requiring the CSV files to be UTF-8 encoded, but some users tried to upload files in other encodings. FasterCSV ended up choking on characters that weren’t valid UTF-8 and truncating the data to the end of the line and leaving fields blank. We didn’t want to ask the user to select an encoding, because they’d probably get it wrong anyway, so we decided to reject any files with characters that would cause problems. The trick then, is how to detect that.
First, the tests. We want to detect if an input string contains valid characters in UTF-8 encoding. And we need to deal with both strings and IO (File or StringIO) objects (more on that in a bit).
describe "::encoding_is_utf8? checks strings and IOs" do
before do
@utf8 = "This is a test with รง international characters"
end
it "returns true when all characters are valid" do
Importer.encoding_is_utf8?(@utf8).should be_true
Importer.encoding_is_utf8?(StringIO.new(@utf8)).should be_true
end
it "returns false when any characters are invalid" do
bogus = Iconv.conv('ISO-8859-1', 'UTF-8', @utf8)
Importer.encoding_is_utf8?(bogus).should be_false
Importer.encoding_is_utf8?(StringIO.new(bogus)).should be_false
end
end
Here’s the implementation:
class Importer
def self.encoding_is_utf8?(file_or_string)
file_or_string = [file_or_string] if file_or_string.is_a?(String)
is_utf8 = file_or_string.all? { |line| Iconv.conv('UTF-8//IGNORE', 'UTF-8', line) == line }
file_or_string.rewind if file_or_string.respond_to?(:rewind)
is_utf8
end
#...
So the meat of the check is that we are using the Iconv library to detect bad characters. We convert from an assumed UTF-8 to UTF-8, ignoring any characters that can’t be represented in UTF-8. If the output and input aren’t identical, that means there were bogus characters and the uploaded file should be rejected.
The #rewind is needed to reset the read position in the file so FasterCSV can start over from the beginning. Specs for that aren’t included here.
Then in our controller, we ensure the CSV doesn’t have any bad characters before we give it to FasterCSV. We extracted that check into its own method, shown here:
def require_utf8!(csv_content)
unless Importer.encoding_is_utf8?(csv_content)
raise "Import file must be UTF-8 only. You can paste non-UTF-8 CSV directly into the CSV Text field for automatic conversion."
end
end
As you can read in the exception message (which ends up in the flash), the user can work around the encoding issue by pasting the CSV into a textarea input in the browswer, which automatically transcodes the data into UTF-8. Aren’t browsers awesome? The other option would be to transcode the CSV file, but the textarea is easier if the files aren’t gigundous. Anyway, since we can input CSV as either a file or a textarea string, that’s why #encoding_is_utf8? needs to check both files and strings.
This approach and implementation seem fine to me. I get the feeling there might be a much simpler way, though. Anyone got a better idea?
Michael Buffington has posted the first of a what will hopefully be a series of tours on Pivotal Tracker, this one about how Michael uses Tracker as a personal GTD-style task manager.
http://blog.grockit.com/2009/01/14/how-i-use-pivotal-tracker-for-gtd/
If you haven’t heard of GTD, it stands for “Getting Things Done”, a work-life “action management” method and book by David Allen. Read more about it here.
A lot of us here at Pivotal Labs have been using Pivotal Tracker as a personal productivity tool, and we’re definitely getting more things done!
Thanks, Michael, we’re looking forward to more tours.
Today I spent the day on Capitol Hill, presenting the Open Source Digital Voting Foundation to the 12th Annual Tech Exhibition of the Congressional Internet Caucus. OSDV is a non-profit foundation working to build open source solutions for voter and election management, as a way to increase the transparency of elections, restore confidence in the vote, and reduce the cost to states and counties to implement such systems.
The application we were showing was one we’ve been developing with them for the past 8 weeks, an online voter registration tracking system, called Reggie. It’s designed to help eligible voters to register to vote, and to track their registration process; and to give registrars greater oversight and auditability.
By applying agile methods and leveraging Ruby on Rails, we were able to deliver a fully test-driven alpha product in just 8 weeks from the initial scoping meeting, during the holiday season, with a single pair of developers.
OSDV has had serious interest from a number of State Secretaries and registrars, and hopes to pilot with at least one state in the coming months.
Pivotal-developed projects like this one, Peer to Patent, and Casebook apply modern web technologies to improve the efficiency and transparency of government. We’re proud to work on projects like this one, and to be an agent for change.
“Daemon best practices in Ruby?”
“cut doesn’t handle strange characters in large (5GB) text file, are there other unix commands for text file manipulation that are utf-8 compliant?”
awk/sed maybeod/hexdump to figure out what the weird characters areUPDATE 01/14/2009: Chad’s corrections
Ruby does not throw any exceptions or warnings if an object defines two methods with the same name. The second definition always wins. While this provides great flexibility for many Ruby tasks it can be problematic when writing tests. In particular, if you define two tests with same name in the same test class, one will get run and the other will not. If you have been writing tests for a while you understand that there is nothing worse than writing a test that never gets used!
Recently Matthew O’Connor and I set out to fix this problem for Test::Unit by alias method chaining :method_added for TestCase classes and their subclasses. We use the inherited hook on Ruby classes to dynamically define a method_added hook for every test case. This is required because method_added does not get inherited between classes, so it can’t be defined only in Test::Unit::TestCase.
The following patch raises an exception if it detects a duplicated test name.
class Test::Unit::TestCase
class << self
def known_test_methods
@known_test_methods ||= Array.new
end
def record_test_method(method)
if method.to_s.starts_with?("test")
if known_test_methods.include? method
raise "Duplicate test #{self}##{method}"
else
known_test_methods << method
end
end
end
def inherited(subclass)
class << subclass
def method_added_with_duplicate_check(method)
record_test_method(method)
method_added_without_duplicate_check(method)
end
alias_method_chain :method_added, :duplicate_check unless method_defined?(:method_added_without_duplicate_check)
end
end
end
end
When we first tried this against a legacy code base we found almost a dozen duplicated tests that were not running.
Frederick Cheung discusses this in more detail here. This might be related to the Selenium + class reloading issues some pivots have experienced in recent weeks. The alternative is to rely on Rails automagic loading or ‘require_dependency’.
Props to pivot Pat Nakajima for creating acts_as_fu.
Matthais Marschall at AgileWebOperations.com posted an interesting article the other day comparing Pivotal Tracker to Thoughtworks Mingle. He recognizes that the two tools have a very different philosophy when it comes to configurability: Mingle supports a huge number of different configuration options, whereas Tracker is intentionally very simple and more prescriptive about process. His take is that the simplicity of Tracker is fact a good thing. He likes Tracker’s backlog management, its simple UI, and the way that the tool fits naturally into the agile process:
Where Mingle is more like an assistant, whom you can tell to put a card on the wall or give you this or that report, Tracker is more like an agile coach, making sure all the administrative tasks like iteration tracking are “simply there”.
Chris‘s recent post called The Tracker Story points out that simplicity and ease-of-use has always been one of Tracker’s core design principles:
Tracker embraces simplicity. It should make managing projects easy, rather than make its users slaves to maintaining the plan. It should give every user of the system more information back than they put in.
Tracker doesn’t have a huge list of features, because it tries to stay true to its core purpose.
Thanks for the post, Matthais, and we’re glad you’re finding Tracker to be a good fit for your development process.
We had an odd bug last week where we ended up with different results after we had eager loaded an association vs loaded directly.
There are apparently two issues with :has_one :through, one of which also applies to :has_many :through.
So given:
class Person
has_many :friendships
has_one :best_friend, :through => :friendships, :conditions => "friendships.best = 1"
end
If you do a Person.find(:all, :include => :best_friend), the best_friend that gets preloaded is not necessarily one that has a “friendship.best = 1″
This is due to a bug in the association preloading code that doesn’t pass down the finder options, so any :conditions or :order are completely ignored. This problem is easy to fix, just a one line change, but it then exposes another problem.
This problem applies to both :has_many :through and :has_one :through associations. The problem is that the :through association is loaded separately from the :has_one or :has_many association. So it first loads :friendships, and then when it tries to load :best_friend, it doesn’t have the table it needs for the :conditions and explodes.
Our current work around is basically putting the conditions on the :through association, although sometimes you need to create a new association just for that which is certainly not idea, especially if you plan on accessing the :through model after it has been loaded.
The way to fix it in Rails is unfortunately a rewrite of how the :through associations are eager loaded.
Pivotal Labs is a consulting shop and not a product company, so it might seem odd for us to release a product like Tracker. We didn’t build Tracker for the sake of building a product; we needed it.
We’ve been doing agile software development before terms like “Agile” and “XP” existed. Over the years we’ve made numerous attempts to use the variety of software project management applications available, from Microsoft Project to the more current agile-specific products.
We kept returning to index cards, sometimes augmented with a patchwork of wikis and spreadsheets. We followed the agile planning tools out there, but each attempt to adopt an one resulted in frustration. Configuration and data entry were a constant expense. User interfaces were clunky and had too much back and forth navigation. The workflows were inefficient and the overhead was high. (We kept hearing circus music in our heads, with all the hoops we were jumping through.) The usage cost never seemed sustainable. It always felt like we were working for the product, instead of it working for us.
As our business grew, so did our frustration; index cards were far from ideal but the alternatives were worse. And at the same time, we were just starting to look at a promising new technology, Ruby on Rails. So we decided to build our own tool.
We started working on Tracker in late 2006. Its beginnings were very humble. All we needed was to have a backlog and be able to easily prioritize. We created simple story editing and built drag-and-drop prioritization and we were ready to start the switch from index cards. (These simple features gave us parity with index cards, with the bonus of having the backlog online.)
We started using Tracker on all of our client projects. We continually improved and tweaked the application, integrating our experience and feedback. And we started to find that it really transformed both the transparency and the flow of development.
Tracker quickly developed a following among client developers who had used it when working with us, too. We would routinely received requests for accounts: “I used Tracker at my last job. Can I have an account to use in my new one?”
We would always accommodate these word-of-mouth, friends and family type requests. Entire client organizations, beyond our projects, started using Tracker. The interest continued to grow and with much greater enthusiasm. We kept getting emails saying things like, “Please let me pay for Tracker; I can’t live without it.”
So we decided to make Tracker publicly available. We launched the public beta at RailsConf 2008. Over 10% of the attendees signed up that week.
We had three key insights at the outset that made Tracker work:
The first point is simply about good UI design, but it’s amazing how many tools out there get this wrong.
The last point is more subtle. In agile processes, particularly XP, there is a concept called velocity. The basic idea is that you assign a point cost to stories and the sum of the point costs of completed stories in a given iteration is the velocity. You then use the velocity to project how much you will be able to get done in each future iteration. As teams start to gel, they exhibit a strong central tendency to get a consistent amount done each week, and this velocity becomes extremely predictive and reliable.
Traditionally, after completing an iteration, you would re-plan each subsequent iteration, adjusting for the actual rate of progress. This can also be onerous, but it’s critical, as it is allows you to project when each milestone will be reached. The feedback needs to be taken into account so better-informed decisions can be made.
The solution is the emergent iteration. Pivotal Tracker automatically (and in real-time) tracks story completions, iterations and velocity and will dynamically adjust iteration backlogs based on actual progress; Tracker does the grunt work and your project schedule is always up-to-date.
Tracker embraces simplicity. It should make managing projects easy, rather than make its users slaves to maintaining the plan. It should give every user of the system more information back than they put in.
Tracker doesn’t have a huge list of features, because it tries to stay true to its core purpose. (Remember, it exists partly because of our frustration with the bloated alternatives.)
At times, Tracker appears opinionated as it can place strict limits on how you manage your project. But this is a consequence of its focus and the interdependence of features; the sum of Tracker is far greater than its parts.
While we don’t always get it right, we try to make sure each additional feature or product change adds a lot of value and makes users’ work easier.