Removing Old Ruby Source Installation After a Leopard Upgrade

Friday, February 22, 2008

I just upgraded to Leopard on my Mac. Previously, on Tiger, I had installed Ruby from source, in the default /usr/local/lib prefix. After reading the discussion on the Apple-provided Ruby installation, I decided to try it – mainly to ensure that my apps, such as GemInstaller, play well with it (on Pivotal’s Mac pair workstations, we still install Ruby from source, so everything matches our demo/production environments as closely as possible, and things are in consistent locations).

So, I wanted to uninstall the old Ruby source installation, and only have the Apple-provided Ruby on disk. Googling for a few minutes did not provide exact instructions for this, so I’m writing up what I did, in hopes that it will help you!

I didn’t use the “–prefix” option when I originally installed Ruby from source, so it was in the default location of /usr/local/lib/ruby, with binaries in /usr/local/bin.

WARNING: Use ‘rm -rf’ at your own risk – a sleep-deprived encounter with ‘rm -rf’ and a stray file named ‘~’ is what “motivated” my Leopard upgrade in the first place…

First, I deleted the old ruby libraries/gems, which was easy enough, because they all lived under /usr/local/bin/ruby:

sudo rm -rf /usr/local/lib/ruby

However, this left all the old ruby/gems executables in /usr/local/bin. This resulted in errors when trying to run executable gems that I had not yet installed under the Apple Ruby installation:

$ cheat
/usr/local/bin/cheat:9:in `require': no such file to load -- rubygems (LoadError)
from /usr/local/bin/cheat:9

Instead of a cryptic rubygems error, I should get a ‘file not found error’:

$ sudo rm /usr/local/bin/cheat
$ cheat
-bash: /usr/local/bin/cheat: No such file or directory

So, I want to purge everything ruby-releated from my /usr/local/bin folder. I whipped up a quick ruby one-liner which just prints out (almost) all ruby-related files in /usr/local/bin:

ruby -e "old_ruby_execs = `egrep 'rubygems|bin/ruby|env ruby' /usr/local/bin/*`; require 'pp'; pp old_ruby_execs.split("n").collect{|line| line.split(':').first}.uniq"

Yeah, I know, ugly and obtuse, but one-liners are kind of fun, and help me remember that Ruby is great tool for sysadmin scripts. Feel free to put it in a class and test it if you are so inclined.

Even though I tried to make a fairly specific regexp for egrep, when inspecting that list, I did find a ‘jgem’ file, which was part of JRuby. I’m planning on reinstalling JRuby anyway, so I didn’t care if that got deleted along with the other ruby stuff.

Anyway, if the output of that looks like everything you want to delete, then run this one-liner to do the actual deed (the ‘sudo echo’ is to ‘prime’ the sudo auth, so you don’t get a noninteractive password prompt):

sudo echo; ruby -e "old_ruby_execs = `egrep 'rubygems|bin/ruby|env ruby' /usr/local/bin/*`; old_ruby_execs.split("n").collect{|line| line.split(':').first}.uniq.each { |exec| p 'removing ' + exec; `sudo rm #{exec}`}"

After that, the only thing that I saw left was the ‘ruby’ executable itself, which I whacked as well:

$ sudo rm /usr/local/bin/ruby

That seems to be about it, as least good enough to get all the old invalid executables off my path. I’m sure this could have been done cleaner if I had taken more care with the original source install. However, a good brute-force approach never hurt anyone. Much. Feel free to post links to relevant and helpful stuff.

ruby-debug in 30 seconds (we don't need no stinkin' GUI!)

Chad Woolley

Tuesday, January 8, 2008

Many people (including me) have complained about the lack of a good GUI debugger for Ruby. Now that some are finally getting usable, I’ve found I actually prefer IRB-style ruby-debug to a GUI.

There’s good tutorial links on the ruby-debug homepage, and a very good Cheat sheet, but I wanted to give a bare-bones HOWTO to help you get immediately productive with ruby-debug.

Install the latest gem

$ gem install ruby-debug

Install the cheatsheet

$ gem install cheat
$ cheat rdebug

Set autolist, autoeval, and autoreload as defaults

$ vi ~/.rdebugrc
set autolist
set autoeval
set autoreload

Run Rails (or other app) via rdebug

$ rdebug script/server

Breakpoint from rdebug

(rdb:1) b app/controllers/my_controller.rb:10

Breakpoint in source

require 'ruby-debug'
debugger
my_buggy_method('foo')

Catchpoint

(rdb:1) cat RuntimeError

Continue to breakpoint

(rdb:1) c

Next Line (Step Over)

(rdb:1) n

Step Into

(rdb:1) s

Continue

(rdb:1) c

Where (Display Frame / Call Stack)

(rdb:1) where

List current line

(rdb:1) l=

Evaluate any var or expression

(rdb:1) myvar.class

Modify a var

(rdb:1) @myvar = 'foo'

Help

(rdb:1) h

There are many other commands, but these are the basics you need to poke around. Check the Cheat sheet for details.

This can also be used directly from any IDE that supports input into a running console (such as Intellij Idea).

That should get you started. So, before you stick in another ‘p’ to debug, try out ruby-debug instead!

The Power of Versions (Monkey Patches Targeted with Friggin Laser Beams!)

Chad Woolley

Friday, January 4, 2008

We all love to Monkey Patch Rails and other Ruby apps. However, we sometimes want to target these patches to the specific versions where they are needed.

Here’s the easiest way to do this, via RubyGem’s built-in version requirement support. The version 0.11.0 should indeed be greater than version 0.9.0:

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> Gem::Version::Requirement.new(['> 0.9.0']).satisfied_by?(Gem::Version.new('0.11.0'))
=> true

Notice that you can’t do this with string comparison, because with a per-character comparison,1 is not greater than 9:

irb(main):001:0> '0.11.0' > '0.9.0'
=> false

Here’s a little class which puts some helper and example methods around this approach (these methods are all in real use for some of our multipart mailer hacks):

module Pivotal
  class VersionChecker
    def self.current_rails_version_matches?(version_requirement)
      version_matches?(Rails::VERSION::STRING, version_requirement)
    end

    def self.version_matches?(version, version_requirement)
      Gem::Version::Requirement.new([version_requirement]).satisfied_by?(Gem::Version.new(version))
    end

    def self.rails_version_is_below_2?
      result = Pivotal::VersionChecker.current_rails_version_matches?('&lt;1.99.0')
      result
    end

    def self.rails_version_is_below_rc2?
      Pivotal::VersionChecker.current_rails_version_matches?('&lt;1.99.1')
    end

    def self.rails_version_is_1991?
      Pivotal::VersionChecker.current_rails_version_matches?('=1.99.1')
    end
  end
end

(note: some angle brackets changed due to code formatting bug)

Here’s an example of how you’d use this:

if Pivotal::VersionChecker.rails_version_is_below_2?
  # do some backward compatibility stuff
  # or handle bugs that have been fixed in Rails > 2
end

Note that this is only possible now that Rails has started using a more sensible strategy for versioning edge gems and improved support for using advanced versioning with RAILS_GEM_VERSION.

For many projects, this may be overkill. It is useful at Pivotal, though, where many various projects may be on different rails versions, but still want to use the latest common core libraries (and monkey patches) without having to upgrade Rails for their app.

This isn’t only useful for monkey patching. It can be handy for any library that wants to be backward- or forward-compatible with its dependencies. I’ve used this approach at Pivotal and on my personal projects to have Continuous Integration automatically run my tests against multiple dependency versions, without having to change anything other than the CI project name:

There are numerous other related topics for discussion in this area, such as the power of versions or the wisdom of freezing, but I’ll save those for future posts. Even if you do freeze the trunk of Rails/plugins/gems, since the version is included in the source, this approach should work barring any conflicts with trunk changes since the last release.

Happy Versioning!

Helpful Named-Route Error Messages

Pivotal Labs

Friday, January 4, 2008

Sometimes I call a named route incorrectly: edit_user_project_path(project). And I get an illegible error message:

user_project_url failed to generate from {:action=>"show", :user_id=>#<Project id: 1, name: "Andy falls off a cliff", created_at: "2007-12-03 15:15:08", creator_id: 2, completed_at: nil, description: "", deleted_at: nil>, :controller=>"projects"}, expected: {:action=>"show", :controller=>"projects"}, diff: {:user_id=>#<Project id: 1, name: "Andy falls off a cliff", created_at: "2007-12-03 15:15:08", creator_id: 2, completed_at: nil, description: "", deleted_at: nil>}

I can’t read that. This, however, is much clearer:

user_project_url failed to generate from {:action=>"show", :user_id=>"1", :controller=>"projects"}, expected: {:action=>"show", :controller=>"projects"}, diff: {:user_id=>"1"}

The error message really ought to call #to_param on the path parts, don’t you think?

class ActionController::Routing::RouteSet
  # try to give a helpful error message when named route generation fails
  def raise_named_route_error(options, named_route, named_route_name)
    helpful_options = options.inject({}) {|hash, (key, value)| hash.merge(key => value.to_param) }
    diff = named_route.requirements.diff(options)
    unless diff.empty?
      raise RoutingError, "#{named_route_name}_url failed to generate  from #{helpful_options.inspect}, expected:  #{named_route.requirements.inspect}, diff:  #{named_route.requirements.diff(helpful_options).inspect}"
    else
      required_segments = named_route.segments.select {|seg| (!seg.optional?) && (!seg.is_a?(DividerSegment)) }
      required_keys_or_values = required_segments.map { |seg| seg.key rescue seg.value } # we want either the key or the value from the segment
      raise RoutingError, "#{named_route_name}_url failed to generate from #{helpful_options.inspect} - you may have ambiguous routes, or you may need to supply additional parameters for this route.  content_url has the following required parameters: #{required_keys_or_values.inspect} - are they all satisfied?"
    end
  end
end

Make it so!

alias_method_chain :validates_associated, :informative_error_message

Pivotal Labs

Friday, January 4, 2008

I dislike the vague error message produced by validates_associated.

class User
  validates_associated :profile
  delegate ..., :to => :profile
end

I see the following error message: profile is invalid. But WHY was the profile invalid? The validation errors from the profile should bubble up to the user. So,

module ActiveRecord::Validations::ClassMethods
  def validates_associated(association, options = {})
    class_eval do
      validates_each(association) do |record, associate_name, value|
        associate = record.send(associate_name)
        if associate && !associate.valid?
          associate.errors.each do |key, value|
            record.errors.add(key, value)
          end
        end
      end
    end
  end
end

Now we see:

Music tastes can't be blank

Eh, voila!

bookmark_fu: drop-in Iconistan

Joe Moore

Saturday, December 22, 2007

We just implemented bookmark_fu on Pivots and the experience was very smooth, taking only a few minutes. We how have an “Iconistan” of social bookmarking chiclets for either remembering or promoting content on Digg, reddit, del.icio.us — almost 20 sites in all.

Install via the normal plugin install process (the -x installs it as an SVN:EXTERNAL):

#> ruby script/plugin install -x svn://rubyforge.org/var/svn/pivotalrb/bookmark_fu/trunk/bookmark_fu

I did have one issue — the script/plugin install script pulled all the code down but ultimately failed because we have multiple versions of Rails on our development machine (about 5); this seemed to confuse the install script. No problem, though: I ran the install.rb script manually:

#> script/runner vendor/plugins/bookmark_fu/install.rb

iPhone-Optimized Webapp

Alex Chaffee

Thursday, December 13, 2007

iPhone on Rails – Creating an iPhone optimised version of your Rails site using iUI and Rails 22

Rails 2.0 Released

Alex Chaffee

Thursday, December 13, 2007

DHH’s official announcement of Rails 2.0
Ben’s Rails 2 Upgrade Notes
Ryan’s Feature Summary

Collapsing Migrations

Alex Chaffee

Wednesday, December 12, 2007

(6:30 pm: updated to use mysqldump)
(12/14/07: updated to remove db:reset since the Rails 2.0 version now does something different.)
(12/15/07: updated to not set ENV['RAILS_ENV'] since that gets passed down to child processes)

There was an old hacker who lived in a shoe; she had so many migrations she didn’t know what to do. Every time her build ran clean, she spent a whole minute staring at the screen.

Fortunately, she read this blog post and now her db:setup task is so fast she’s started building multiple test environments so she can run tests in parallel!

Figure out what migration to collapse to. This number should be less than or equal to the oldest deployed version of your app. E.g. if most of your deployments are on version 348 but there’s one client running a branch that’s only up to version 298, then pick 298 (or 297 if you’re afraid of off-by-one errors). For this example we will use 100.
Install lib/tasks/db.rake and lib/db_tasks.rb (source below)
Clear the development database by running
rake db:clear
Dump the development structure by running
rake db:dump
Delete all the migrations up to and including your target version. Here’s a sneaky awk script that deletes everything up to and including 100. (Go ahead and run it, it won’t bite, and you can always revert.)
ls db/migrate/ | awk ‘{split($0, a, “_”); if(a[1]<=100) print $0}’ | xargs svn rm
Create a new migration called “100_collapsed_migrations.rb” using the following template.

100_collapsed_migrations.rb:

class CollapsedMigrations < ActiveRecord::Migration
  def self.up
    sql = <<-SQL
  # development_structure.sql goes here
    SQL

    execute("SET FOREIGN_KEY_CHECKS=0")
    sql.split(";").each do |statement|
      execute(statement)
    end
  ensure
    execute("SET FOREIGN_KEY_CHECKS=1")
  end

  def self.down
    raise IrreversibleMigration
  end
end

Open up db/development_dump.sql and copy its entire contents into your clipboard, then paste it above the “SQL” line in your new migration 100.
Search for the statement that creates the schema_info table and remove it.

Mine looks like this:

CREATE TABLE `schema_info` (
  `version` int(11) default NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

Set up your databases and run your tests.
rake db:setup test
Congratulations! Your migrations are now blazingly fast, just like back in the (scaff)old days. You can run “rake db:setup” any time you get a svn update that looks like it may have done something funky to your schema, rather than shying away from that minute-long migration and just hoping your tests still pass.

Why do we need to use db:dump rather than db:schema:dump? Well, unfortunately, db:schema:dump doesn’t dump everything. It misses CONSTRAINT statements and also seems to get the charset wrong (although that may have been a function of how I constructed the db in my test). And db:structure:dump misses any data that may have been added by your migrations.

Here’s my current db.rake. Unfortunately, it only works with MySQL, but if you want to make it support your favorite DB (or even your least favorite) then please go right ahead.

Oh, and that part about multiple test environments and parallellized tests? Stay tuned… :-)

db.rake:

require "db_tasks"

namespace :db do
  def tasks
    (@db_tasks ||= DbTasks.new(self))
  end

  desc "Drop and recreate database"
  task :clear => :environment do
    tasks.clear
  end

  desc "Clear and migrate dev and test databases, and load fixtures into development db"
  task :setup => :environment do
    tasks.setup
  end

  desc "Dump the current environment's database schema and data to, e.g., db/development_dump.sql (optional param: FILE=foo.sql)"
  task :dump => :environment do
    if ENV['FILE']
      tasks.dump ENV['FILE']
    else
      tasks.dump
    end
  end

  desc "Load an sql file (by default db/development_dump.sql). (Optional param: FILE=foo.sql)"
  task :load => :environment do
    if ENV['FILE']
      tasks.load ENV['FILE']
    else
      tasks.load
    end
  end
end

db_tasks.rb:

# This creates a duplicate of the database config for a db config as defined in database.yml.
# For example, if the "test" database is named "myapp_test",
# for clone number 0, the new environment is named "test0", and the database is "myapp_test0".
# All other settings are preserved (esp. username and password).
module ActiveRecord
  class Base
    def self.clone_config(original_config, worker_number)
      original = configurations[original_config.to_s]
      raise "Could not find conguration '#{original_config}' to clone" if original.nil?
      worker_config = original.dup
      worker_config["database"] += worker_number.to_s
      configurations["#{original_config}#{worker_number}"] = worker_config
    end
  end
end

class DbTasks
  def initialize(rake)
    @rake = rake
  end

  def init
    connect_to('development')
    clear_database
    migrate_database
    dump
    test_environments.each do |test_db|
      if test_db =~ /([0-9]+)$/
        clone_test_config($1.to_i)
      end
      connect_to(test_db)
      clear_database
      load
    end
  end

  # db:clear -> drop and create db for RAILS_ENV
  def clear
    clear_database
  end

  # db:setup -> drop, create, and migrate dbs for test and development environments, and import fixtures into development
  def setup
    init
    connect_to 'development'
    load_fixtures
  end

  def dump(file = "#{RAILS_ROOT}/db/#{environment}_dump.sql")
    puts "Dumping #{database} into #{file}"
    system "mysqldump #{database} -u#{username} #{password_parameter} --default-character-set=utf8 > #{file}"
  end

  def load(sql_file = "#{RAILS_ROOT}/db/development_dump.sql")
    puts "Loading #{sql_file} into #{database}"
    query('SET foreign_key_checks = 0')
    sql_file = File.expand_path(sql_file)
    IO.readlines(sql_file).join.split(";").each do |statement|
      query(statement.strip) unless statement.strip == ""
    end
    query('SET foreign_key_checks = 1')
  end

  protected

  def clone_test_config(worker_num)
    ActiveRecord::Base.clone_config("test", worker_num)
  end

  def connect_to(environment)
    ActiveRecord::Base.establish_connection(environment)
    @environment = environment
    Object.const_set(:RAILS_ENV, environment)
    # Note: don't set ENV['RAILS_ENV'] since that gets passed down to invoked tasks (including 'rake test')
  end

  def environment
    (@environment ||= RAILS_ENV)
  end

  def test_environments
    environments = ['test']
    if Object.const_defined?(:TEST_WORKERS)
      TEST_WORKERS.times do |worker_num|
        environments << "test#{worker_num}"
      end
    end
    environments
  end

  def load_fixtures
    puts "Loading fixtures into #{environment}"
    Rake::Task["db:fixtures:load"].invoke
  end

  def clear_database
    puts "Clearing #{environment} database"
    sql = "drop database if exists #{database}; create database #{database} character set utf8;"
    cmd = %Q|mysql -u#{username} #{password_parameter} -e "#{sql}"|
    # puts "executing #{cmd.inspect}"
    system(cmd)
  end

  def migrate_database
    puts "Migrating #{environment} database"
    ActiveRecord::Migration.verbose = false
    Rake::Task["db:migrate"].invoke
  end

  def config(env = environment)
    ActiveRecord::Base.configurations[env]
  end

  def query(sql)
    ActiveRecord::Base.connection.execute(sql)
  end

  def database
    config["database"]
  end

  def username
    config["username"]
  end

  def password
    config["password"]
  end

  def password_parameter
    if password.nil? || password.empty?
      ""
    else
      "-p#{password}"
    end
  end

  def execute(cmd)
    puts "t#{cmd}"
    unless system(cmd)
      puts "tFailed with status #{$?.exitstatus}"
    end
  end

  def system(cmd)
    @rake.send(:system, cmd)
  end
end

rake query_trace

Alex Chaffee

Saturday, November 17, 2007

QueryTrace is a great Rails plugin (which I learned about from ErrTheBlog) for pinpointing where in your Rails application that slow query is running. Once you have it installed, your logs won’t just tell you that you have a problem, they will pinpoint the exact location of that problem for you. This is invaluable when doing load & performance testing or just trying to understand what the hell ActiveRecord is doing.

Unfortunately, even though it only logs query traces in DEBUG log mode, it still clutters up your test and development logs, and actually slows things down a bit too. So you don’t want to leave it in your project’s vendor/plugins directory after you’re done using it. So I wrote a pair of rake tasks to enable and disable it.

rake query_trace:on                     # Enables the query_trace plugin. Must restart server to take effect.
rake query_trace:off                    # Disables the query_trace plugin. Must restart server to take effect.

The “on” task actually checks out the plugin from query_trace’s subversion repository, makes a tarball in vendor/query_cache.tar.gz, then leaves the tarball around so it doesn’t have to keep going to the network all the time. You can even check the tarball in to your project and it’ll never go to the query_trace repository again. The “off” task just removes the whole vendor/plugins/query_cache directory.

namespace :query_trace do
  desc "Enables the query_trace plugin. Must restart server to take effect."
  task :on => :environment do
    unless File.exist?("#{RAILS_ROOT}/vendor/query_trace.tar.gz")
      Dir.chdir("#{RAILS_ROOT}/vendor") do
        url = "https://terralien.devguard.com/svn/projects/plugins/query_trace"
        puts "Loading query_trace from #{url}..."
        system "svn co #{url} query_trace"
        system "tar zcf query_trace.tar.gz --exclude=.svn query_trace"
        FileUtils.rm_rf("query_trace")
      end
    end
    Dir.chdir("#{RAILS_ROOT}/vendor/plugins") do
      system "tar zxf ../query_trace.tar.gz query_trace"
    end
    puts "QueryTrace plugin enabled. Must restart server to take effect."
  end

  desc "Disables the query_trace plugin. Must restart server to take effect."
  task :off => :environment do
    FileUtils.rm_rf("#{RAILS_ROOT}/vendor/plugins/query_trace")
    puts "QueryTrace plugin disabled. Must restart server to take effect."
  end
end