Challenges faced in a little larger Rails API project

In the past three months we have been heavily developing SKL Genombrott project using Rails in the backend and the fancy Ember framework in the frontend. Given Embers philosophy, Rails was used almost exclusively for providing a restful API to Ember. I have to admit that as a junior developer there are cases that I hadn't seen before and not even imagined that Rails could restrict me. This post will focus on a Rails API app.

Back to basics

Before moving on though, I think it's a good idea to refresh what a regular Rails app look like.

The big picture: big picture

Since Rails 2.3, we have rack support and as a result the params that we get in a controller is nothing more than a (sanitized from Rails' ActionDispatch::Http::Parameters method) Rack::Request.params.

If we zoom in the Rails app it will look like: rails zoom

First and foremost, params filtering takes place in controllers. This happens by the (cumbersome IMHO) strong params Rails gem. A regular param filtering could be:

  def create_params
     params.require(:user).permit(
       :email, :password, :password_confirmation, :first_name, :last_name, :image_url, :unit_id
     )
  end

On the other end we have serializers. Serializers, output the resource data according to the selected format. Here we can either use Roar, Jbuilder, Oat or regular ActiveModel Serializers.

So essentially, Rails controllers are just Rack wrappers. They receive input, delegate work to other service objects and return (display) as output the processed input.

HATEOAS are still a vague idea in Rails

Coincidentally I had just read a book about REST (here HATEOAS is included in the term) ideas so I immediately thought that I would finally implement a HATEOAS API in Rails. JSONAPI API standard seemed the most rational standard to support. Other options were HAL and Siren. We have used HAL before in a mico Wizard Wars API but HAL is only perfect when your API supports only GET requests since it's far too simple and specs don't even specify how to access a resource, (the HTTP action), in the links. On the other hand, Siren is not widely supported (there is no Siren adapter for Ember). JSONAPI adapter for Ember was sufficient for our needs (although it's still needs improvements). Everything looked ok, we also have great JSONAPI serializer support in Rails using Roar or Oat so why not taking a chance applying what I learned!

I started implementing my first API controller. For the serializer I chose Oat as I feel it's the simplest yet most complete (it supports URI templates out of the box too!). Development continued for a while until I reached to the following JSONAPI section:

To-one relationships MAY be updated along with other attributes by including them in a links object within the resource object in a PUT request.

For instance, the following PUT request will update the title attribute and author relationship of an article:

PUT /articles/1
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "articles": {
    "title": "Rails is a Melting Pot",
    "links": {
      "author": "1"
    }
  }
}

However doing that in Rails takes some time and effort. First you have to dig in the params returned from Rack and find the relations and then structure them like ActiveModel params style. Essentially the above code will be like:

  params.require(:user).permit(
    :email, :password, :password_confirmation, :first_name, :last_name, :image_url
  ).merge(unit_id: params.require(:user).require(:links).permit(:unit)[:unit])

Yeap. That beautiful. And that's only for one relationship. Patch method? Not even standardized in the standards. Parsing embedded resourced? You have to do everything on your own. Eventually I gave up and went back to the Railsy non HATEOAS Rest way.

Authorization is tough

For authorization we used Pundit. It's a really simple gem. The guy might have written more code for the Rails template generators than for the authorize method itself.

I would like to step back and talk a bit about authorization. It's a common factor that Authentication and Authorization are different notions. Authentication takes place before authorization. However I would like to point that guest users are authenticated too and should also pass the regular authorization process. (again, my focus is on APIs). We might mix this concept by Pundits or Device philosophy that don't take into account unauthorized (better: 'users who haven't signed in' since an unauthorized user could be an admin in a request that is accessible by super-admin only) users at all. It's acceptable that when we know that a whole resource (GET/POST/PUT/DELETE) is not allowed to be accessed by a guest user to have a before_filter in Rails since it avoids hitting the db for no reason. But it's not always like that as we will see later.

In a regular (Rails) controller there are 3 steps of authorization (not necessarily in that order):

  • You need to authorize the input params (for instance a user might not have the authorization to update the created_at and updated_at fields of a resource).
  • You need authorize the user on the given resource on the given method. Does the user have authorization to update (PUT) the users/1 resource?
  • You need to filter what the user is authorized to get back. For instance a guest user might only be authorized to see specific attributes of a resource.

Coming back from authorization intro, let's take a look on how a regular Rails API controller method look like:

class Api::V1::UsersController < Api::V1::BaseController
  def show
    user = User.find(params[:id])
    authorize user

    render json: user, serializer: Api::V1::UserSerializer
  end
end

When I do authorize user I want to know 3 things:

  1. if user is authorized to access the resource in specific action
  2. if there are any authorization errors
  3. what permissions the user has in the attributes of the resource

If I don't know one of these that this means that I am going to duplicate the authorization somewhere else. Inside the controller or inside the serializer.

About 1, I think it's self explanatory. It's how most authorization gems work in Rails.

About 2, logging authorization errors is vital for the front-end developers. Consider the case where you have admins, users and posts which belong to events. Now let's say that an admin has the authorization to create any post in any event whereas a user can create a post in an event only if the latter hasn't passed (or add any other condition here). That's an authorization error. Not an 401 like error, more like a 403 error. But it should be handled by authorization policy (if we want to have a DRY code).

About 3, imagine the requirement that users should be visible to guest users too, but only their image and their first name.

Using Pundit to support all 3 authorization requirements, this is quite impossible to be DRY. Pundits philosophy is like everything are black or white. You either have access or you don't. Unfortunately it's not always the case.

To solve this issue, you can either monkeypatch Pundits authorize method or create a new one, called authorize_with_permissions:

Inside an initializer:

module AuthorizeWithReturn
  def authorize_with_permissions(record, query=nil)
    query ||= params[:action].to_s + '?'
    @_pundit_policy_authorized = true

    policy = policy(record)
    return true if policy.public_send(query)
  end
end

module Pundit
  prepend AuthorizeWithReturn
end

Now, calling that method, inside your controller, you know if the user has access or not by checking Pundits return value. But still, is that dry? Are you going to render a different serializer just for that?

What if you had more than 2 levels of permissions? Say that you had a super user who could see everything, an admin, a regular user and a guest. Pundit return value wouldn't help you figuring out user's permissions. I will say again that I want all authorization to take place inside Pundits policy, otherwise I will duplicate the code (probably leading to a bug).

So essentially I need to get an object from authorization that will tell me if there are any errors to show or the attributes that the serializer will serialize.

Returning the attributes the user has access to, in our case, was needed by the ActiveModel Serializer's :only option. But the general idea is that authorization should specify which attributes user has access to update in a Model(s) and which you will serialize from the updated resource. That's one more reason for not defining 'extra' methods inside the serializers. Serializers should only serialize an object to a specific format.

Consequently the controller action will look like:

class Api::V1::UsersController < Api::V1::BaseController
  def show
    user = User.find(params[:id])
    authorized_user = authorize_with_permissions user
    return api_error(status: 403, authorized_user.errors) unless authorized_user.errors.blank?

    render(
      json: Api::V1::UserSerializer.new(
        authorized_user.record,
        only: authorized_user.attributes
      ).to_json
    )
  end
end

You have to change Pundits authorize_with_permissions method to return the actual result of the policy object.

module AuthorizeWithReturn
  def authorize(record, query=nil)
    query ||= params[:action].to_s
    @_pundit_policy_authorized = true

    policy = policy(record)
    policy.public_send(query)
  end
end

module Pundit
  prepend AuthorizeWithReturn
end

Given that all actions (show/create/update/delete) have the same permissions for a specific user, inside the Pundit policy:

class UserPolicy < ApplicationPolicy
  def show
    #you can add errors to the object, if you have a non trivial authorization condition
    return Permissions::Admin.new(record) if user.super_admin?
    return Permissions::Owner.new(record) if record.association1.eql?(user)
    return Permissions::Regular.new(record) if record.association2.include?(user)
    return Permissions::Regular.new(record) if record.association3.users.include?(user)
    return Permissions::Guest.new(record)
  end

Now, embracing OO style, inside ApplicationPolicy define a ApplicationPermissions that all your Permissions object will inherit from:

  class ApplicationPermissions
    attr_accessor :attributes, :errors
    attr_reader :record

    def initialize(record)
      @record = record
      @errors = []
    end

    def attributes
      record.attributes.keys.map(&:to_sym).concat(
        record.class.reflect_on_all_associations.map{ |assoc| assoc.name}
      )
    end
  end

and finally define your desired Permissions classes inside the UserPolicy:

  class Permissions < ApplicationPermissions
    class Admin < self
    end

    class Owner < self
      def attributes
        super - [:created_at, :updated_at]
      end
    end

    class Regular < Owner
      def attributes
        super - [:association1, :association2]
      end
    end

    class Guest < Regular
      def attributes
        [:first_name, :public_image_url]
      end
    end
  end

Now, a lot of people might think that this is ugly and I won't blame them at all. Using OO it's getting veery verbose but personally I can't find any other DRY way. It should be noted that UserPolicy::Permissions could handle more methods like sanitizing the attributes that the user can update in a model (which might be different from the incoming params and the serializable attributes) and basically anything that you might need later and depends on the authorization.

We use that kind of structure in our latest project in just a couple of controllers that have many permission levels because it would be overkill to define all those POROS. You don't need to change your whole rails structure, since we declared a new Pundit authorize method. Personally, I think that if we had Pundits conventions in Permissions-like for sanitizing incoming params, specifing which model attributes are allowed to be accessed and specifing which object attributes should be visible to the user, we would have a very robust authorization system.

AR query chaining limits you in an index API method

Using where method in AR does not hit the db directly but instead returns a new ActiveRecord::Relation waiting for more specific queries. It's called lazy loaded. A regular index method (GET on /resources) in a Rails controller (again I focus on APIs) could look like:

class Api::V1::PdsasController < Api::V1::BaseController
  def index
    pdsas = Pdsa.all
    pdsas = pdsas.where(id: params[:ids]) if params[:ids]
    pdsas = pdsas.scope_by(params[:scope]) if params[:scope]

    [:unit_id, :act_status, :unit_category_id, :subject_id, :area_id].each do |prm|
      pdsas = pdsas.where(prm => params[prm]) unless params[prm].blank?
    end

    unless params[:start_date].blank?
      pdsas = pdsas.where('START_DATE >= ?', params[:start_date])
    end

    unless params[:end_date].blank?
      pdsas = pdsas.where('END_DATE <= ?', params[:end_date])
    end

    unless params[:name].blank?
      pdsas = pdsas.where('NAME ILIKE ?', "%#{params[:name]}%")
    end

    pdsas = policy_scope(pdsas)

    render json: pdsas, each_serializer: Api::V1::PdsaSerializer
  end
end

Taken from a real controller from SKL Genombrott project. Each request param specifies more and more the final query. Act status is a database enum and suppose that can take 4 values: 'terminated', 'ongoing', 'repeated' and 'planned'. In the current API you can specify which of the 4 act_status you want but what if you want a combination of them? Then this pattern obviously becomes quite complex. This can be solved using Arel though. The problem here is that we need an .or() method and only Arel seems to support it.

Writting API tests can be such a waste of time

Usually when I have to create a new API endpoint my process is the following:

  1. Create the model
  2. Add factories and model tests
  3. Add route
  4. Add API controller
  5. Add serializer
  6. Add API (controller) tests
  7. Add Pundit policies
  8. Add more (authorization) tests

Usually when I am out of time I tend to skip the model tests. Not that model tests are useless but more because I feel API tests catch many model tests so I might repeat myself. Also I think API tests are a unique set of tests. They are not unit tests but are not integration tests either (at least in the sense of scenarios).

With regular model tests, you create a factory/fixture and test the model under different inputs. In API tests, you still create a factory, only this time you tests test simultaneously , your controller, Pundit policies, your model and your serializer. You test at a slightly higher level (that includes multiple components) but at the same time you test only a specific component under specific input params. You are on the HTTP level and that helps a lot since it's a stateless protocol. So you don't need scenarios. Yes, you might create a scenario (better: a state) in your database using multiple factories but you test only 1 request per time.

A regular API controller would look like:

class Api::V1::CountiesController < Api::V1::BaseController

  # Needed for /inviterequest
  before_filter :authenticate_user!, except: [:index]

  def index
    counties = County.all
    counties = counties.where(id: params['ids']) if params['ids']

    render json: counties, each_serializer: Api::V1::CountySerializer
  end

  def show
    county = County.find(params[:id])

    render json: county, serializer: Api::V1::CountySerializer
  end

  def create
    county = County.new(create_params)
    return api_error(status: 422, errors: county.errors) unless county.valid?

    county.save!

    render(
      json: county,
      status: 201,
      location: api_v1_county_path(county.id),
      serializer: Api::V1::CountySerializer
    )
  end


  def update
    county = County.find(params[:id])

    if !county.update_attributes(update_params)
      return api_error(status: 422, errors: county.errors)
    end

    render(
      json: county,
      status: 200,
      location: api_v1_county_path(county.id),
      serializer: Api::V1::CountySerializer
    )
  end

  def destroy
    county = County.find_by(params[:id])

    if !county.destroy
      return api_error(status: 500)
    end

    head status: 204
  end

  private

  def create_params
     params.require(:county).permit(:name)
  end

  def update_params
    create_params
  end

end

This is one of the simplest controller APIs that we have. But still we need basic tests even for that.

For reference, here is the counties serializer:

class Api::V1::CountySerializer < Api::V1::BaseSerializer
  attributes :id, :name, :created_at, :updated_at

  has_many :units
  has_many :municipalities
end

The most basic tests that I will write are the following:

require 'rails_helper'

describe Api::V1::CountiesController, type: :api do
  context :index do
    before do
      create_and_sign_in_user
      5.times{ FactoryGirl.create(:county) }

      get api_v1_counties_path, format: :json
    end
    it 'returns the correct status' do
      expect(last_response.status).to eql(200)
    end
    it 'returns the correct number of data in the body' do
      body = HashWithIndifferentAccess.new(MultiJson.load(last_response.body))
      expect(body[:counties].length).to eql(5)
    end
  end

  context :create do
    before do
      create_and_sign_in_user
      county = FactoryGirl.attributes_for(:county)
      post api_v1_counties_path, county: county.as_json, format: :json
    end

    it 'returns the correct status' do
      expect(last_response.status).to eql(201)
    end

    it 'returns the data in the body' do
      county = County.last!
      body = HashWithIndifferentAccess.new(MultiJson.load(last_response.body))
      expect(body[:county][:name]).to eql(county.name)
      expect(body[:county][:updated_at]).to eql(county.updated_at.iso8601)
    end
  end

  context :show do
    before do
      create_and_sign_in_user
      @county = FactoryGirl.create(:county)

      get api_v1_county_path(@county.id), format: :json
    end

    it 'returns the correct status' do
      expect(last_response.status).to eql(200)
    end

    it 'returns the data in the body' do
      body = HashWithIndifferentAccess.new(MultiJson.load(last_response.body))
      expect(body[:county][:name]).to eql(@county.name)
      expect(body[:county][:updated_at]).to eql(@county.updated_at.iso8601)
    end
  end

  context :update do
    before do
      create_and_sign_in_user
      @county = FactoryGirl.create(:county)
      name = 'Another name'
      @county.name = name
      put api_v1_county_path(@county.id), county: @county.as_json, format: :json
    end

    it 'returns the correct status' do
      expect(last_response.status).to eql(200)
    end

    it 'returns the correct location' do
      expect(last_response.headers['Location'])
        .to include(api_v1_county_path(@county.id))
    end

    it 'returns the data in the body' do
      county = County.last!
      body = HashWithIndifferentAccess.new(MultiJson.load(last_response.body))
      expect(body[:county][:name]).to eql(@county.name)
      expect(body[:county][:updated_at]).to eql(county.updated_at.iso8601)
    end
  end

  context :delete do
    context 'when the resource does NOT exist' do
      before do
        create_and_sign_in_user
        @county = FactoryGirl.create(:county)
        delete api_v1_county_path(rand(100..1000)), format: :json
      end

      it 'returns the correct status' do
        expect(last_response.status).to eql(404)
      end
    end

    context 'when the resource does exist' do
      before do
        create_and_sign_in_user
        @county = FactoryGirl.create(:county)

        delete api_v1_county_path(@county.id), format: :json
      end

      it 'returns the correct status' do
        expect(last_response.status).to eql(204)
      end

      it 'actually deletes the resource' do
        expect(County.find_by(id: @county.id)).to eql(nil)
      end
    end
  end
end

What these tests test? From my business logic, literally nothing. But I think they are vital and should appear in every API controller test. They test:

  • the path input -> controller -> model -> controller -> serializer -> output actually works ok
  • controller returns the correct error statuses
  • controller responds to the API attributes.

What I am actually doing here is that I re-implement the RSpecs methods respond_to and rspec-rails' be_valid method at a higher level. Only that it takes me something more than 110 lines of code and I don't even test the associations. And what if I change my serializer and use HAL or JSONAPI instead? Then I have to change my whole test suite (even the next tests, like authorization tests etc).

I would love to see a gem that using the adapter pattern parses a HAL/JSONAPI/Siren/whatever api response and gives me the parsed object :)

Summarizing

To summarize my experience, I think with the help of a couple of gems, Rails could be an excellent platform for developing APIs.

  1. I would like to see a gem that sits on top of strong parameters and sanitizes the incoming parameters from a specific API standard to an activemodel params style. I will be able to choose which API standard I use (JSONAPI, HAL, etc). The gem should return 422 with the error if something required is not there according to the API standard.
  2. I would like to see a gem that sits on top of Pundit and specifies which params should be permitted and where. For instance if a user is supposed to update an attribute in a resource, or, which resource attributes the controller should return to the user.
  3. I would like to see a gem just like Pundit but with support of multiple permission levels. It could be co-developed with (2).
  4. I would like to see a gem for tests, that parses the controller output of a specific API format and creates a Ruby object from this response. Typically you will define here which adapter (API standard) your serializers like in 1. Again, the test should be red with the error if something required is not there according to the API standard. It will boost your productivity by writing more sophisticated tests in the controller level since you won't test for trivial attributes (like meta info, links, url templates etc).
Loading comments…
All rights reserved