BrigadeRest

Ready to run in production? Please check our deployment guides. Uses pinterest/elixir-thrift thrift parser and bindings for elixir,

See: https://github.com/pinterest/elixir-thrift

See: https://hexdocs.pm/thrift/readme.html#content

Elixir docs: https://hexdocs.pm/elixir/Kernel.html

Phoenix docs: https://hexdocs.pm/phoenix/Phoenix.html

Plug docs: https://hexdocs.pm/plug/readme.html (used as part of Phoenix)

Running an elixir service

Installation

Install elixir, on mac

brew install elixir

Install the dependencies for the project

mix deps.get

Development

Run the code as usual with mix phx.server. Now you can visit localhost:4000 from your browser.

The config in config/config.exs is applied and then the config in config/dev.exs is applied.

You can run the application with an interactive elixir terminal using iex as

iex -S mix phx.server

Where you can interact with any of the code from command-line. This is preferred while trying to debug, as there is tab completion for modules/methods and also better visibility. There is no saved state in functional programs so you can run any code from anywhere without too much difficulty.

Production

Run the code with MIX_ENV=prod PORT=$AVAILABLE_PORT_FROM_CONSUL mix phx.server

You should note that the config in config/config.exs will be applied and then the config in config/prod.exs will be applied. The prod.exs file has the production hosts hardcoded inside of it's declaration, those could be changed to System.get_env("$MY_ENV_VAR") if they are to be passed in through mesos/cryogen etc.

Overview of the service

The service is a Phoenix web server that uses generated Thrift clients (via pinterest/elixir-thrift) to contact internal Brigade services. It will attempt to translate basic thrift types automatically to provide a rudimentary querying ability.

Known shortcomings:

• filter parameters will need to be made to handle further nesting of parameters
• some services may not adhere to the naming conventions that most do, and will need custom code to make their interpretations work

Currently there are 2 types of endpoints:

• The dynamic endpoint module DynamicThriftController which will attempt to generate Thrift requests for any service and any method
• The verification search endpoint module VerificationController which is scoped specifically to the verification service, but only as a demonstration of a concrete application of Thrift clients since the dynamic controller may be hard to reason about.

/:service_name/:request_name endpoint and the lib/brigade_rest_web/router.ex file

This file is where the web server establishes its routing rules. The web server will match routes in the order of first match priority, and will pass the connection (Plug.Conn struct) to the appropriate handler method, for instance

get "/:service_name/:request_name", DynamicThriftController, :request

will take all page accesses matching any service name, and any request name and pass that to the DynamicThriftController module function request. That function will decide how to handle the incoming request. For example, a curl http://localhost:4000/voter-verifier/search would be making a call to the VoterVerifier client, and calling the search method.

Note that the voter-verifier has its own route that supersedes the above route,

get "/voter-verifier/search", VerificationController, :search

Which will then direct all requests via curl http://localhost:4000/voter-verifier/search to the VerificationController module function search.

Thrift client configuration

Each Thrift client must be setup manually currently. The Thrift client code is automatically generated by configuring the pinterest/elixir-thrift library to point to a directory containing thrift definitions. The mix.exs file contains the compiler instructions,

compilers: [:thrift, :phoenix, :gettext] ++ Mix.compilers,

and the target for the thrift files (files ending in .thrift)

thrift: [
  files: Path.wildcard("../thrift-defs/src/**/*.thrift")
],

This requires the thrift definitions to be in a directory beside this directory. The files will be compiled on start (mix phx.server) and compiled into lib/thrift/generated.

Accessing Thrift definitions

The thrift definitions will be available (once compiled) in the namespace

Thrift.Generated.*

Clients are only available in one format, a binary framed tcp client, which is accessible with

Thrift.Generated.<ServiceName>.Binary.Framed.Client

and will contain all the methods defined in the *.thrift definition files.

Starting and accessing Thrift clients

Elixir is a functional language, therefore there are no client "objects" there are object references or registries to access open ports, tcp connections etc.

To make things more simple for this application, each Thrift client is started in the lib/brigade_rest/application.ex file by creating a worker process that is managed by the application's Supervisor process. All this means is that there is a strategy that the Supervisor will implement to attempt to keep the child (worker) alive. In this case, it is :one_for_one strategy, see: https://hexdocs.pm/elixir/Supervisor.html#module-strategies for a detailed overview of how strategies work.

We also assign each client a name when it is made into a worker - this name is for the :global registry, so that when we call client operations it knows which client connection to use.

For instance when you call start_link on a thrift client, typically it returns a PID (process identifier), however we want to share this same PID to every http request, so that we are using a single connection (multiplexing) and not making new connections every time an http request is made. By passing the name parameter, we can replace the usage of the PID with the name, e.g.

{:ok, pid} = Thrift.Generated.CivicDataService.Binary.Framed.Client.start_link("mesos.brigade-rc.zone", 11064)
result = Thrift.Generated.CivicDataService.Binary.Framed.Client.get_served_terms(pid, ...)

# But inside the http handler, we don't don't want to keep track of that `pid` - it may die
# and be restarted by the supervisor! So instead, we want to reference it by namespace, using `name`

# The `application.ex` does this when it creates the `workers`
Thrift.Generated.CivicDataService.Binary.Framed.Client.start_link("mesos.brigade-rc.zone", 11064, name: :civic_data_service)

# Now inside the http handler we can simply call to the client by `name`
result = Thrift.Generated.CivicDataService.Binary.Framed.Client(:civic_data_service, ...)

Authentication/Authorization

To keep things extra simple, the project uses a simple list of strings to check against (in memory only) for the existence of an "api key". In this case the config/config.exs file contains a list of strings in the config :brigade_rest, BrigadeRestWeb.Endpoint, namespace under the key allowed_api_keys. To add your own simply add your own required string to the list. To generate on you can use mix phx.gen.secret for a hard-to-collide-with api key.

Add them to the config like

config :brigade_rest, BrigadeRestWeb.Endpoint,
  allowed_api_keys: [
    "some-generated-key",
    "some-other-generated-key",
  ]

Then ensure that the SimpleApiKeyAuthPlug is "plugged in" in the lib/brigade_rest_web/router.ex file like

plug BrigadeRestWeb.SimpleApiKeyAuthPlug

in the :browser pipeline, or anywhere it is required. The SimpleApiKeyAuthPlug checks that the incoming request has a "x-brigade-rest-api-key" header set with an api key that matches one of the values in the allowed_api_keys list.

Note: In development you may want to test without authorization on so you can make queries directly from the browser url bar. To do this, simply comment out the plug,

# plug BrigadeRestWeb.SimpleApiKeyAuthPlug

and request the page again - just don't forget to uncomment it again.

Accessing the API

You can access the APIs through restful urls

/:service_name/:request_name - maps to a service you name like voter-verifier and a method on that service like search, e.g. GET /voter-verifier/search

You can also pass parameters as get parameters like

GET /verification_service/search?last_name="Cumpson"&first_name="Jonathan"

Allowed parameters to the DynamicThriftController endpoint include

• limit the number of results to allow, default is 10
• direction ascending or descending, defaults to 0 (descending)
• cursor the uid of the starting point of the query, NOT including the provided uid. E.g. if you pass in the 10th item's UID to the next query as the cursor it will return the next batch after that cursor, not including it.

Other parameters need to be added via POST or less general methods.

Examples (make sure SimpleApiKeyAuthPlug is disabled unless you are requesting with headers):

http://localhost:4000/voter-verifier/search?last_name=%22Smith%22&first_name=%22Jane%22

Learn more

• Official website: http://www.phoenixframework.org/
• Guides: http://phoenixframework.org/docs/overview
• Docs: https://hexdocs.pm/phoenix
• Mailing list: http://groups.google.com/group/phoenix-talk
• Source: https://github.com/phoenixframework/phoenix