A few days ago I finally discovered why rake db:migrate:redo consistently angers me nearly as much as watching Paula Dean deep fry the vegetable kingdom. As any devoted connoisseur of the db rake tasks in Rails knows, db:migrate:redo always leaves your schema.rb file in the wrong state. The reason, as mentioned in our standup blog, is that rake will only invoke a given task once in a particular run.
To trivially test this try running a single task twice:
rake db:rollback db:rollback
You’ll find that your database only rolls back one migration. Now, you can set the STEP environment variable when calling db:rollback, but this is, as I said, a trivial example. It gets worse.
Take a look at the implementation of the db:migrate:redo task. The part we’re interested in looks like this:
namespace :migrate do
task :redo => :environment do
...
Rake::Task["db:rollback"].invoke
Rake::Task["db:migrate"].invoke
end
end
That looks fine; db:migrate:redo just verifies that your new migration will properly run down and up without blowing up. Sweet.
But, here’s what db:migrate looks like:
task :migrate => :environment do
# Do migratey stuff
Rake::Task["db:schema:dump"].invoke if ActiveRecord::Base.schema_format == :ruby
end
And rollback:
task :rollback => :environment do
# Do rollbacky stuff
Rake::Task["db:schema:dump"].invoke if ActiveRecord::Base.schema_format == :ruby
end
Both db:migrate and db:rollback dump the schema after they run, as they should. If you were to migrate or rollback your database and not dump the schema, then your schema would be in an invalid state. So, of course you can see where this is going, when you run db:migrate:redo the task performs the rollback, dumps the schema, performs the migrate, and then doesn’t dump the schema, because that task has already run. Boom, your schema is one migration behind, db:test:prepare loads the invalid schema into your test database, and all your tests fail (or, worse, pass inappropriately)
Now, I assumed this was a bug in Rake, and so I went on a little investigatory safari through the jungles of the Rake code to find it and kill it. I found the culprit, but invoking each task at most one time is, somewhat surprisingly, the expected behavior; it’s tested and everything. Now I can only wonder why. Why prevent invocation of a task more than once in a given rake run? The code contains unrelated guards against circular task dependencies, so that’s not it. Is this an example of overly-speculative defensive coding, or is there an actual use case for which this behavior is desirable? I’d like to hear from anyone who has written tasks that depend on this behavior, as well as anyone who (like me) considers this behavior unexpected and has run into problems because of it.
Assuming no one steps forward with a compelling reason that Rake should behave this way, I’d suggest that this be changed. I could see the value of it (perhaps as a performance optimization?) if rake tasks were guaranteed to not change the state of anything they operate on, or even were guaranteed to be idempotent; but neither is the case. This behavior severely limits the composability of tasks, since a task writer has to know which atomic tasks have run, and avoid any task that might try to run them again.
In the meantime, Rake provides a way to explicitly re-enable tasks that have run once, but it doesn’t seem to work. The db:schema:dump definition looks like this:
namespace :schema do
task :dump => :environment do
# Do dumpy stuff
Rake::Task["db:schema:dump"].reenable
end
end
That #reenable call is meant to tell the task “hey, task, you can run again.” I tried calling #reenable on the db:schema:dump task inside the db:migrate and db:rollback tasks as well, but without any luck.
Fellow Pivot David Stevenson would likely put it this way: Khaaaaaaaaaaaaaaann!
Hey Adam,
Thanks for the post!
I’ve been using Rake for a few years now to compile, run and debug complex ActionScript and Flex projects. The compiler is unusually slow, and some tasks involve preprocessing hundreds of files. For me, it has proven to be a benefit that Rake only executes a particular task once in an execution cycle.
With that said, I do agree that (in spite of the fact that it doesn’t seem to be a word) the ‘reenable’ method (or some such feature) should be available and it should work as expected.
August 23, 2009 at 9:46 pm
Rake works this way because it was originally designed to be a replacement for Make. So, when you are are saying
rake :my_task => :another_task
you are actually saying”‘before `my_task` runs, ensure that `another_task` has been run”, which is subtly different from “before `my_task` runs, run `another_task`”. It’s the definition of a prerequisite.
Typically, in a `make` situation, this would be something like “before you link the binary, ensure that any changed source files have been compiled”. If all the compiled files are already up-to-date, you don’t need to run the prerequisite task, saving a bunch of time.
The problem here is really that Rake is being used in a way that it wasn’t designed for, but for the most part this behaviour suits us well (we only ever want to load the environment once, for example). To change this characteristic of Rake would be to completely change it’s nature as a tool.
I completely sympathise with your frustration though – after months of minor annoyance, I finally got around to writing a patch for this, only to discover that it’s already fixed in the master branch of Rails.
August 24, 2009 at 3:13 am
FYI, here’s the [commit](http://github.com/rails/rails/commit/ba146a84d0ed8a886fdc6b6794ce99a9d37c0190), and the [ticket](https://rails.lighthouseapp.com/projects/8994/tickets/1412-dbmigrateredo-does-not-dump-the-schema-after-migrating-back-up), which looks to be along the same lines are your solution. If it definitely doesn’t work for you, that’s the place to add feedback.
August 24, 2009 at 3:29 am
Last comment, I promise. Here’s my test Rakefile:
task :redo => [:rollback, :migrate]
task :rollback do
puts “rollback”
Rake::Task["dump"].invoke
end
task :migrate do
puts “migrate”
Rake::Task["dump"].invoke
end
task :dump do
puts “dumping”
Rake::Task["dump"].reenable
end
and here’s the output of `rake –trace`
$ rake –trace
(in /Users/james/Code/experiments)
** Invoke default (first_time)
** Invoke rollback (first_time)
** Execute rollback
rollback
** Invoke dump (first_time)
** Execute dump
dumping
** Invoke migrate (first_time)
** Execute migrate
migrate
** Invoke dump (first_time)
** Execute dump
dumping
** Execute default
As you can see, the `dump` task is successfully invoked the second time (you can see that Rake thinks that it’s never been invoked). So, `reenable` seems to work, at least in this simple situation.
August 24, 2009 at 3:35 am
What about calling .execute instead of .invoke on the rake tasks? Then they get run every time.
August 24, 2009 at 7:26 am
James, you provided the little mental nudge I needed in order to make this fit in my brain. It’s been a long time since I used make with any frequency, and I’ll admit I always avoided it as much as possible.
Based on your example, and the fact that the Rails patch you referenced quite clearly doesn’t work, I wonder if the problem has to do with namespacing. I’ll have to play with that.
August 24, 2009 at 5:14 pm
What version of Rake are you using?
August 25, 2009 at 3:08 am
James is right. Check out Yehuda Katz’ project called Thor: “rake and sake needed to be replaced for scripts, not as a replacement for make”. Basically it’s an interface for scripts, and most modern builds are better modeled as scripts, not as dependency trees.
http://yehudakatz.com/2008/05/12/by-thors-hammer/
I haven’t used it much yet so I don’t know if it supports invoking the same task multiple times, but, you know, if it doesn’t, it probably should…
Steve C raves, “It’s like Erector for Rake!”
August 26, 2009 at 12:07 pm
Here’s a nice recent article about using Rake as a dependency manager:
< http://www.jbarnette.com/2009/08/27/on-rake.html>
I still think that the first time you find you need to call `.invoke` you should probably think about switching to Thor…
August 27, 2009 at 4:19 pm
I’m happy to think about switching to Thor, but I’m not, so far, able to think about it for the entire Rails community. For better or for worse the current state of affairs is that Rails uses Rake.
Thor also has to overcome its potentially insurmountable lack of alliteration with Rails.
August 27, 2009 at 4:27 pm