A response to “Rails 4 Engines” at TaskRabbit
Throughout this post I am using “engine”, “Rails engine”, and “component” interchangeably, using what fits best in a given context.
Earlier this year, Brian Leonard published his blog post Rails 4 Engines on TaskRabbit’s use of engines in their service. Since then, I have been meaning to write a reply and I want to thank Shu Uesugi for prompting me to finally do it.
Brian’s original article is very comprehensive and touches on many aspects of engine use, so I will focus and comment only on three important points of his argument:
- Shared Code: Brian argues for avoiding code sharing between engines as much as possible. Specifically, he discusses not sharing code, sharing code through the main application, and using a “shared” engine.
- Tests: Brian argues for a pragmatic creation of tests in the main application – essentially, to not change the tests in comparison to any other Rails application.
- Routes: Brian argues for mounting of all engines at the root path.
In all three cases, I will end up arguing for more separation between the main app and the engines and also for separation between the engines themselves. I believe that this strictness leads to looser coupled components, which in turn leads to an application that is able to grow efficiently, longer.
The article lists a couple of options on what to do about shared code:
- Don’t share code between engines
- Keep shared code in the main application
- Put shared code in a “shared” engine
Let’s address these solutions in turn:
Don’t share code
When possible, not sharing anything is the best form of separation. To understand why it is best, think back to the rationale for splitting applications into components: to prevent having to deal with the big, monolithic, ball-of-mud Rails application.
If you were able to (miraculously) split your app in half and no code is shared, the potential complexity of your app drops dramatically (if, for example, you take possible interactions as a measure; see Metcalfe’s law or Reed’s law). Should you be able to split a part off of your app without any dependencies: do it.
In Brian’s admin engine example, such a split can be achieved by creating
post.rb separately for the engine. It is important to stress, however, that despite this separation the admin engine is still dependent on the main application, namely through the data schema that the main app defines. When changing the main application, but not the admin engine, one cannot be sure that the latter still works properly without running all of its tests. Added migrations in the main app have the potential of breaking code in the admin engine.
In the end, instead of sharing code, we are sharing the data schema and the data itself. The dependency becomes a side effect. It becomes difficult to exploit programmatically, because it is nowhere to be found in code. We will return to how that affects testing in the section Tests.
Keep shared code in the main app
In his first example in the section on shared code, Brian leaves
post.rb in the main app and moves controllers and views into engines for
When analyzing the code provided in Brian’s post, it turns out that both gems are dependent on code outside of their own folder structure – code they do not control. That is certainly not common practice for gems or Rails applications and I want to call it “reverse monkey patching”: this code expects to be monkey patched. So, the gems do not explicitly state that they have a dependency, while in fact, they do. Usually, a Gemfile would be our mechanism to declare such a dependency, but that won’t work here because we can’t require a Rails app.
admin are engines and gems. In my view, they should make sense on their own. For example, “future you” or someone else might find that one of them provides exactly what they need and they would like to see whether they can use it for their purposes. Or maybe someone thinks they can add a test to just one of the gems. In those cases, they will find that the gems are non-functional on their own.
Just don’t do this.
Brian’s next example solves the above problem by introducing a
shared engine. The moment you have this, you can explicitly state the dependencies of
shared. To do this, simply add the required gem to the gemspec of the dependent!
Adding these dependencies to the gemspec will not make the engines individually usable right away. This is because bundler won’t be able to find the right
shared engine when looking for it just by its name. If you intend to do this, be sure to add a Gemfile to your engines. Find an example in my sample app: sample app gemspec and sample app gemfile and note the
path option used in the latter.
Note that when you follow this advice you can use my gem cobradeps to draw a dependency graph of all the components in your system.
The one issue I have with this approach is that the name of this engine is “shared.”
There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors. Two Hard Things
Naming an engine “shared” is giving up on the second hard problem. And it is a huge bummer to give up on that, because with components we finally have a vehicle in Rails to talk about these higher level parts in our application! The same goes for names like “common”, “misc”, “miscellaneous”, “general”, “base”, “<fill in your app/company/team name>”.
I whole heartedly agree with Brian when he says:
Again, architecture does not exist for fun or to get in the way. If something is super-simple and obvious and easy to maintain while doing the “right” way for the design is difficult and fragile, we just do it the easy way. That’s the way to ship things for customers. However, we’ve found that in most case the rules of the system kick off useful discussions and behaviors that tend to work out quite well.
Keep the useful discussions going and bake their results into good names for your app. They will be able to keep the conversation going when you have long left the code.
Let’s do this for the
shared engine from the blog post:
Users work as an engine name for this? It seems like
post.rb would no longer fit into it. We would have an analogous problem if trying to name this engine
Posts. However, one might get away with turning this into two engines
Posts. Or, if posts and users are totally intertwined and we don’t want to separate them, I would argue that
UsersAndPosts is still better than
Without good names components are prone to turn into ball-of-mud engine replacements for ball-of-mud Rails applications.
Brian’s introduction to testing component-based Rails applications is this:
Many of the issues noted here revolve around testing. One of the promises of engines is the existence of the subcomponents that you could (theoretically) use in some other app. This is not the goal here. We are using engines maximize local simplicity in our application, not create a reusable library. To that end, we don’t think the normal engine testing mechanism of creating a dummy app within the engine is helpful.
I agree with everything except with the conclusion. I am convinced that “maximize local simplicity” is not possible if we cannot effectively find the boundaries of this “locality.” I have pointed out above how implicit dependencies lead to monkey patching and the inability to analyze (and execute) parts in isolation.
If you don’t force isolation of parts in your app, you are going to end up with only one part. You are going to end up with an entangled system.
Probably one of the most prolific examples of this is the
lib folder in Rails apps. Ever tried to remove all the code in
app from your application and run the tests for your
lib folder? Most likely, they won’t pass. Why is that? They are libraries after all. It is because dependencies onto
app sneak in over time and go undetected. The end result is that
lib becomes indistinguishable from
app from a code perspective – the two become entangled.
If you want locality that will last, you have to force separation.
All the problems around testing engines in a component-based Rails app in isolation have been solved (if you run into a snag, please tweet or email me!) and there is no reason why you should not do it. I believe that the cost of learning to deal with the differences you will find between testing an engine and testing a Rails app are far outweighed by the benefits of a cleanly separated system.
And once you do, you get other benefits: you would be able to simply move the engine onto a gem server and load it from there into any app you desire (but we don’t really intend to do that). What is really important is that you can prove which components affect others. This knowledge brings true locality.
It would also allow you to push code after running only some of the tests in your app. Since we know exactly which parts of the system we can not have affected with a change, we can opt to not run the tests of those components. Check out the cobratest gem. While still in early stages, it looks at git changes to determine components that are being changed, calculates all affected components, and runs only their tests.
Brian suggests that all engines always be mounted on the root level of the main app. In contrast to this, I always suggest mounting every component at a unique subfolder under root. Read Brian’s post for all the aspects of this tradeoff.
“For example, mounting the account engine at anything but root would prevent it from handling both the
/signup paths” is obvious, but not true. You can use route redirection in the main app or have your web server handle these kinds of special routes.
It may seem like a matter of preference, but there is a rationale for why subfolder mounting (which leads to strict separation of routes) is the better alternative. How do we find out which routes we are potentially in conflict with when adding a new one? When using subfolder mounting we only have to look inside the affected engine. With root-level mounting we have to look into every mounted engine, as all of them could be introducing conflicts.
Routes that need to be added to the root are exceptional cases. Given that, subfolder mounting makes the common case easy to handle. Root-level mounting makes the common case hard to handle.
I believe that there are many reasons to seek strict separation for components in component-based Rails applications. Specifically, I argued for it in the areas of sharing code, writing tests, and creating routes.
The most important aspect that Rails engines bring to the table in component-based Rails applications is a new way of thinking and talking about our applications. Other communities, languages, and frameworks are much further along in harnessing this and we have a lot of catching up to do.
There are a lot more topics in Brian’s post and if you still haven’t done so, head over and read Rails 4 Engines. Ideas like the BootInquirer and the use of the message bus need their own responses.
I have a lot of other resources around component-based Rails applications. Checkout my talks, other blog posts, and my sample app on github.