Interesting Things

• A few projects are using parallel_test to speed up their tests/specs. One project has had the best results by running one less parallel spec than the number of cores on the machine, unless of course you only have one core.

• Another team discovered an odd bug in RubyMine where they could not click-through "require spec" or run specs if their project had code in vendor/plugins that extended ActiveRecord::Base. Here is the RubyMine ticket they filed.

• Finally, it looks like a Linux machine running Ruby 1.8.6 patch 287 had rounding errors when using the ActiveSupport helpers for 2.days.ago. They fixed the issue by upgrading to Ruby 1.8.7

Interesting Things

• One project had trouble installing the Rails 2.3.3 with the most recent version of Rubygems (1.3.5) on a Gentoo box (EY Solo instance). Installing the gem on an OSX box worked fine with the same versions. No obvious solution presented itself, other than to roll back to Rubygems version 1.3.1.

• If you do update your Rubygems version to 1.3.5 and you use Geminstaller you must update your Geminstaller gem to 0.5.2; Rubygems made some changes that break Geminstaller's implementation. If you use an older version of Rubygems then keep using Geminstaller 0.5.1.

• One project wanted to find all places where they needed to escape user input on their site, so they injected <script>alert('foo');</script> into every text field in the database, ran through their site, and looked for every place that a pop up box appeared.

Interesting Things

• RailsBridge is organizing a global BugMash to help knock down some of the bugs on the Rails lighthouse this weekend. If your interested in helping out there is more information on their wiki

• We started using JRuby for a project and wanted to use JRuby to run our specs. If you install RSpec 1.2.7 or higher you can specify the ruby command used to run your SpecTask in rspec.rake like this:

  desc "Run all specs in spec directory (excluding plugin specs)"
  Spec::Rake::SpecTask.new(:spec) do |t|
    t.spec_opts = ['--options', "\"#{File.dirname(__FILE__)}/../../spec/spec.opts\""]
    t.spec_files = FileList['spec/**/*/*_spec.rb']
    t.ruby_cmd = 'jruby'
  end

Interesting Things

• We discovered a bug in the load order of CGI extensions in ActionController in Rails 2.3.3. The body of the first request to a Mongrel server after loading the Rails environment will appear empty, even if it's not. The problem is the default CGI::QueryExtension#initialize_query in the Ruby standard library will read the stdinput stream and not rewind it. As a result, future reads of stdinput look empty. In Rails, this behavior is overridden in action_controller/cgi_ext/query_extension.rb. Unfortunately this file isn't loaded until after the first request to a new server, at which point subsequent requests work normally. The Rails ticket is here.

• The easiest fix is to require "action_controller/cgi_ext" in your environment file or an initializer file.

Interesting Things

• If you're on Rails 2.3+ you can use NestedScenarios instead of FixtureScenarios and FixtureScenarioBuilder and your life will be better.

Ask for Help

• The Pre Dev Camp in San Francisco is looking for a home this weekend. If you have a location you can donate that can hold about 150 people with laptops, has power and wifi, please contact the organizer, Luke Kilpatri at (650) 745-5302 or luke@lukek.ca

• "Does anyone have experience, good or bad, with RJB (the Ruby to Java bridge)?" General consensus was murmur murmur murmur no.

Ruby does not throw any exceptions or warnings if an object defines two methods with the same name. The second definition always wins. While this provides great flexibility for many Ruby tasks it can be problematic when writing tests. In particular, if you define two tests with same name in the same test class, one will get run and the other will not. If you have been writing tests for a while you understand that there is nothing worse than writing a test that never gets used!

Recently Matthew O'Connor and I set out to fix this problem for Test::Unit by alias method chaining :method_added for TestCase classes and their subclasses. We use the inherited hook on Ruby classes to dynamically define a method_added hook for every test case. This is required because method_added does not get inherited between classes, so it can't be defined only in Test::Unit::TestCase.

The following patch raises an exception if it detects a duplicated test name.

class Test::Unit::TestCase
  class << self
    def known_test_methods
      @known_test_methods ||= Array.new
    end

    def record_test_method(method)
      if method.to_s.starts_with?("test")
        if known_test_methods.include? method
          raise "Duplicate test #{self}##{method}"
        else
          known_test_methods << method
        end
      end
    end

    def inherited(subclass)
      class << subclass
        def method_added_with_duplicate_check(method)
          record_test_method(method)
          method_added_without_duplicate_check(method)
        end
        alias_method_chain :method_added, :duplicate_check unless method_defined?(:method_added_without_duplicate_check)
      end
    end
  end
end

When we first tried this against a legacy code base we found almost a dozen duplicated tests that were not running.

Matthew O'Connor and I recently worked on a project that sent SMS messages to mobile customers. Unfortunately the SMS aggregator we used on the project rejected messages with non-ascii characters.

One approach we considered was to strip our messages of any characters that were not ascii and send them as is. After looking through some of the rejected messages we realized most of the problems occurred with unicode punctuation. Instead of simple deleting the characters we tried transliterating them to their ascii equivalent.

Our first approach used IConv:

require 'iconv'

module SmsEncoder
  def self.convert(utf8_text)
    text = Iconv.iconv("US-ASCII//TRANSLIT", "UTF-8", utf8_text).first
    text.gsub(/`/, "'")
  rescue Iconv::Failure
    ""
  end
end

For some reason the backtick ` also caused problems so we converted that after using Iconv.

This approach worked perfectly on OS X but as soon as we moved to the Linux servers the libiconv characteristics changed and most untranslatable characters became question marks instead of empty strings.

Instead of wrestling with libiconv we looked for a solution entirely in ruby. We found unidecode which got us most of the way there. Unidecode did a little more than we wanted though and translated Chinese and Japanese characters to their approximate sounds. e.g. 今年1月 gets transliterated to Jin Nian 1Yue

We decided to only transliterate extended latin charaters, punctuation and money symbols.

Here is the final code with the unidecode monkey patch:

require 'set'
require 'unidecode'

module SmsEncoder
  def self.convert(utf8_text)
    Unidecoder.decode(utf8_text.to_s).gsub("[?]", "").gsub(/`/, "'").strip
  end
end

module Unidecoder
  class << self
    def decode(string)
      string.gsub(/[^\x20-\x7e]/u) do |character|
        codepoint = character.unpack("U").first
        if should_transliterate?(codepoint)
          CODEPOINTS[code_group(character)][grouped_point(character)] rescue ""
        else
          ""
        end
      end
    end

    private

    # c.f. http://unicode.org/roadmaps/bmp/
    CODE_POINT_RANGES = {
      :basic_latin => Set.new(32 .. 126),
      :latin1_supplement => Set.new(160 .. 255),
      :latin1_extended_a => Set.new(256 .. 383),
      :latin1_extended_b => Set.new(384 .. 591),
      :general_punctuation => Set.new(8192 .. 8303),
      :currency_symbols => Set.new(8352 .. 8399),
    }

    def should_transliterate?(codepoint)
      @all_ranges ||= CODE_POINT_RANGES.values.sum
      @all_ranges.include? codepoint
    end
  end
end

and tests:

class SmsEncoderTest < Test::Unit::TestCase
  def test_transliteration_of_blank
    assert_equal "", SmsEncoder.convert(nil)
    assert_equal "", SmsEncoder.convert("")
  end

  def test_transliteration_of_whitespace
    assert_equal "", SmsEncoder.convert(" \t\n")
  end

  def test_transliteration_of_text_surrounded_by_space
    assert_equal "abc", SmsEncoder.convert("  abc  ")
  end

  def test_transliteration_of_ascii
    orig_text = "!\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~"
    conv_text = SmsEncoder.convert(orig_text)
    assert_equal orig_text.gsub(/`/, "'"), conv_text
  end

  def test_transliteration_of_unicode_punctuation
    utf8_text = "“foo” ‹foo› ‘foo’ ,foo, –foo— {foo} (foo) `foo`"
    ascii_text = SmsEncoder.convert(utf8_text)
    assert_equal "\"foo\" <foo> 'foo' ,foo, foo-- {foo} (foo) 'foo'", ascii_text
  end

  def test_transliteration_of_common_latin1_characters
    utf8_text = "ñ ò ^ ¡ ¿ Æ æ ß Ç §"
    ascii_text = SmsEncoder.convert(utf8_text)
    assert_equal "n o ^ ! ? AE ae ss C SS", ascii_text
  end

  def test_transliteration_of_money_characters
    utf8_text = "€ £ $ ¥"
    ascii_text = SmsEncoder.convert(utf8_text)
    assert_equal "EU PS $ Y=", ascii_text
  end

  def test_untransliterable_characters
    utf8_text = "ɏ \x1f \x01 \x00 Ʌ \x7f"
    ascii_text = SmsEncoder.convert(utf8_text)
    assert_equal "", ascii_text

  end

  def test_transliteration_of_chinese_characters
    utf8_text = "ウェブ全体から検索"
    ascii_text = SmsEncoder.convert(utf8_text)
    assert_equal "", ascii_text
  end
end