TL;DR
We moved a Rails app into an unbuilt engine of a new blank slate container app to allow new parts of our app to live next to it as engines as well. It has been working great for us!
I have a sample app rails_container_and_engines of the result’s structure on github.
Skip to the pitfalls and discoveries section to read about some of the speed bumps we during our transition. Interested in the why and how? Read on!
Rationale
As part of the project we had built a web app, a mobile app that talks to the web app’s API, two ETL tools to 1) do the initial data import from the app we were replacing and 2) get data into another of our client’s apps. At this point we knew that we would create one more big web app and several auxiliary apps.
Running up to the decision to using unbuilt engines we had several intense discussions on how to build loosely-coupled, highly-cohesive systems with Rails applications. We saw basically three choices:
- All-in-one app that could get more structure through namespacing of its interals.
- Services (REST or whatever, but) running as separate apps and communicating via APIs.
- Engines running within a mostly feature-less container app.
The two big web apps share several models within the data access layer and use the same data. Because of this we chose the third option and left all of the apps within the same rails code base.
Internal discussions highlighted the costs/benefits of having all of these apps live within their own Rails project versus an engines approach. We feel that by using engines we are getting many of the benefits of a component based architecture without breaking Rails patterns. In addition, we feel that the cost of maintaining individual applications that share a central database or one giant application with less defined components would have been very high.
In order to make day to day development easier, and to avoid the “where do migrations live…” conversation and top level Rails deployment patterns, there is one twist to the architecture: everything resides in one git repo and engines are referenced from a single container application (similar to old school enterprise archive files). Each application is exposed via a unique context or resource identifier (each engine/app could also be isolated per instance via Apache). Here is the directory structure we ended up with:
container_rails_app/
...
app
config
engines/
etl/
shared_modules/
web_app_1/
web_app_2/
...
Mike described this pattern in his recent blog posts Unbuilt Rails Dependencies: How to design for loosely-coupled, highly-cohesive components within a Rails application and Rails Contained: A Container for Web Application Development and Deployment. On how to make RubyMine work seamlessly with engines, read my post on IntelliJ Modules in Rubymine.
The steps we took
- Generate a new, empty Rails app
- Within the
engines folder, create a new, mountable Rails engine Copy all the tests from the original Rails app into the test directory and make them green.
Ok. This step is a bit more involved. Essentially that’s it thought. Find some of the pitfalls we ran into described below. Here are a couple of highlights of what needs to happen:
- Copy the files you need for a test to pass and namespace its class
- Start with the model tests and work your way up towards integration and acceptance tests
- Namespace everything with the name of the engine (either by fully qualifying every name – don’t do that) or with the lexical scope trick explained below. This includes tests and all classes.
- Load needed gems explicitly: engines don’t load as much automatically as a Rails app
- Namespace tables and assets
- Copy the old .git directory into the new root folder in order to not loose any history. For us, git was not able to detect the changes as moves in many cases. This is certainly an area where you can improve on our solution!
For us, the real work started after this, when we started pullling out common code into a common models engine and began work on the second app.
We got all the tests to pass before we had namespaced any of the assets or rake tasks. That was an additional search-and-replace heavy step after the actual transition to an engine. It is not necessary right away if you do not have multiple applications at first, but to achieve the full effect, you will need to namespace the things as well.
Pitfalls and discoveries
Namespaces
Modules in your enige are not automatically loaded: make sure you reference them yourself before they are needed in other files.
Asset pre-compilation
Rails creates a default app/assets/stylesheets/application.css file which contains these lines:
/* ...
*= require_self
*= require_tree .
*/
If you have this file in your main app, all css files will be compiled into one file. For us, this made almost everything look right. Almost. Little things broke here and there. Our app contained a couple of sections for which the stylesheets were meant to be loaded separately. Add files that you want to have precompiled as individual files to config.assets.precompile list to have them precompiled into separate files and solve this problem.
Are all your references and associations breaking?
Try this
module M
class X
def use_y
Y.do_it!
end
end
class Y
def self.do_it!
end
end
end
instead of
class M::X
def use_y
M::Y.do_it!
end
end
class M::Y
def self.do_it!
end
end
The first way sets the lexical scope to the module M as well as to the class Y and allows you to reference other classes in M without their full name. The second way sets the scope only to class M::Y and you have to fully qualify every class name to find it.
Don’t fight conventions
We were using fixture builder and while we were namespacing classes in modules we were trying to override the default table names to not be namespaced… Fixture builder didn’t like that at all. There may or may not be other dependencies that make leaving the conventions hard. So, save yourself the trouble and do the migrations (and stay consistent!) and namespace your tables as well!
HABTM
They seem to be falling out of favor, which is probably a good thing. With engines they don’t seem to work (well). We ended up getting rid of our last two habtm relationships instead of trying to make them work. Creating the join as a class and adding the necessary has many :through relationships is straight forward enough.
We ended up with one production performance bug due to this: Rails
Engines depending on engines
We used kaminari for our pagination requirements. When our app first became an engine, kaminari stopped picking up our custom views. Instead it used its standard views. The load order got screwed up to which there are basically two solutions:
- require => nil on both engines in the Gemfile and force the correct load order in your app, or…
- avoid name clashes by namespacing your views (which you were going to do anyways, right?)
