n+1 Problems and Identity Maps
Forward: This post is mainly copypasta from an answer I wrote on StackOverflow. It also takes into account the version of Mongoid being used predates the includes
method. The includes
method being the suggested best practice for the problem now. includes
will be mentioned at the bottom of the post.
Take me to the current best practice.
How to deal with n+1 issues utilizing identity maps:
What is the n+1 issue?
The n plus 1 issue occurs when you reference an associated record while looping an array of records. For example (using sql): You load a list of Posts
and loop those posts displaying the User
who created each post. The initial posts would require a single query SELECT * FROM 'posts'
. Then when you loop the posts to display the user using post.user
. The db will be queried again. SELECT * FROM 'users' WHERE 'user_id' = post.user_id
. This query will happen once for every post you loop. If you have 10 posts you will get 10 extra queries for the users who wrote those posts. This is the definition of the n+1 issue.
Avoiding unnecessary queries
In my case it was an n+2 issue.
class Judge
include Mongoid::Document
belongs_to :user
belongs_to :photo
def as_json(options={})
{
id: _id,
photo: photo,
user: user
}
end
end
class User
include Mongoid::Document
has_one :judge
end
class Photo
include Mongoid::Document
has_one :judge
end
# judges_controller
def index
@judges = Judge.all
respond_with @judges
end
This as_json response results in an n+2 query issue from the Judge record. in my case giving the dev server a response time of:
Completed 200 OK in 816ms (Views: 785.2ms)
The key to solving this issue is to load the Users
and the Photos
in a single query instead of 1 by 1 per Judge
.
You can do this utilizing Mongoids IdentityMap. Mongoid 2 and Mongoid 3 support this feature.
First turn on the identity map in the mongoid.yml configuration file:
development:
host: localhost
database: awesome_app
identity_map_enabled: true
Now change the controller action to manually load the users and photos. Note: The Mongoid::Relation record will lazily evaluate the query so you must call to_a to actually query the records and have them stored in the IdentityMap.
def index
@judges = Judge.all
@users = User.where(:_id.in => @judges.map(&:user_id)).to_a
@photos = Photo.where(:_id.in => @judges.map(&:photo_id)).to_a
respond_with @judges
end
This results in only 3 queries total. 1 for the Judges
, 1 for the Users
and 1 for the Photos
.
Completed 200 OK in 559ms (Views: 87.7ms)
How does this work? What’s an IdentityMap?
An IdentityMap helps to keep track of what objects or records have already been loaded. So if you fetch the first User
record the IdentityMap will store it. Then if you attempt to fetch the same User
again Mongoid queries the IdentityMap for the User before it queries the Database again. This will save 1 query on the database.
By fetching all of the User
and Photo
records that the Judges
require in manual queries we load the data into the IdentityMap for later use. Then when the Judge requires it’s User
and Photo
it checks the IdentityMap and does not need to query the database for them.
The includes method
Following the practice from ActiveRecord, Mongoid later included a method for doing this type of call in a single concise method.
Judge.all.includes(:user, :photo)
This will perform only 3 queries selecting only the users and photos that belong to the returned judges.
Here are links to the docs for both Mongoid and ActiveRecord:
Read more about eager loading in
Mongoid.
Read more about eager loading in
ActiveRecord.