Skip to content

JSMike/BuckyServer

 
 

Repository files navigation

Bucky Server

Bucky uses a Node server to forward the HTTP requests with your monitoring data to Statsd/Graphite, OpenTSDB, or whatever other service you'd like.

Also see the Bucky Client.

Hosting

Everything you need to run Bucky on Heroku or Nodejitsu is included, just update the config file and push to the service of your choice.

Heroku

heroku create
git push heroku master

Nodejitsu

jitsu deploy

The jitsu application will ask you for a subdomain to run the service on, and will increment the version of the application whenever you deploy.

EC2 / Bare Metal

If you'd rather host Bucky on EC2 directly or on raw hardware, you just need something which will run ./start.js in a reliable way.

You can use environment variables to control runtime options, or put them in your config file in the server section.

You'll need to have nodejs installed. Anything in the 0.8.x series or above should work fine. We recommend using nvm, as it gives you an extra dimension of flexibility, but using your system's package manager should work just as well.

# In the project directory:

npm install
PORT=3333 APP_ROOT=bucky/ ./start.js

The APP_ROOT (or config.server.appRoot) will prefix all endpoints.

Bucky will respond to all requests at /APP_ROOT/v1/health-check, if you need a health check url.

Bucky can be setup to receive data at multiple endpoints, but by default it listens to /APP_ROOT/v1/send on whichever port you specify.

Ubuntu (12.04)

# Install nodejs
# This assumes you're on a 64 bit machine
wget https://nodejs.org/dist/v4.1.1/node-v4.1.1-linux-x64.tar.gz
tar xvf node-v4.1.1-linux-x64.tar.gz
sudo ln -s `pwd`/node-v4.1.1-linux-x64/bin/{node,npm} /usr/local/bin/

# Grab a Bucky release
# You should use the latest release available at https://github.com/HubSpot/BuckyServer/releases
wget https://github.com/HubSpot/BuckyServer/archive/v0.6.2.tar.gz -O BuckyServer.tar.gz
tar xvf BuckyServer.tar.gz
cd BuckyServer

# Install Bucky
sudo npm install -g

# Make any config changes by editing /usr/local/lib/node_modules/bucky-server/config/default.yaml

# You can start bucky by running bucky-server

# Add the upstart script so Bucky starts on startup and respawns
sudo cp init/bucky-server.conf /etc/init/

# Start Bucky with
sudo start bucky-server

# Log files will appear in /var/log/bucky.log by default

You can run bucky on a specific port (make sure to open that port in your security group if you're using EC2), or you can use a reverse proxy like Nginx or HAProxy to serve it on the same domain and port as your website, it's up to you.

Configuring

If you're not already running a stats collection service, you should take a look at our help doc.

Most people will only need to specify the config they're interested in and start up the server.

Configuration Options:

  • server: {Object}
    Use to set properties of the Bucky Server.

    • port: {Number}
      Use to set the port that Bucky Server will listen to.
    • appRoot: {String}
      Use to define the root of the endpoint.
    • https: {Object}
      Defines a set of options for running Bucky in https mode.
      • port: {Number}
        Use to specify the port for https, if not populated the default is the http server port + 1.
      • options: {Object}
        Use to define the options for https. key and cert are mandatory options, here is a full list of all available options. For all options that accept a buffer you can use the path to the file containing the option's data.
      • key: {Object|String|Buffer}
        key can be an Object that contains a filePath to the key file, or key can contain the entire String/Buffer for the key.
        • filePath: {String}
          Path to key file. The filePath is only required for loading the key from a file.
      • cert: {Object|String|Buffer}
        cert can be an Object that contains a filePath to the key file, or cert can contain the entire String/Buffer for the certificate.
        • filePath: {String}
          Path to certificate file. The filePath is only required for loading the certificate from a file.
    • httpsOnly: {Boolean}
      If this flag is set to true then Bucky Server will not run in http mode.
  • statsd:
    Configuration for connecting to statsd. Only required when using statsd module.

    • host: {String}
      The hostname for your statsd server.
    • port: {Number}
      The port for your statsd server.
  • opentsdb:
    Configuration for connecting to openTSDB. Only required when using openTSDB module.

    • host: {String}
      The hostname for your openTSDB server.
    • port: {Number}
      The port for your openTSDB server.
  • influxdb:
    Configuration for connecting to InfluxDB. Only required when using InfluxDB module.

    • host: {String}
      The hostname for your InfluxDB server.
    • port: {Number}
      The port for your InfluxDB server.
    • database: {String}
      The database to write data to inside InfluxDB.
    • username: {String}
      A user in InfluxDB that has write permissions to the specified database.
    • password: {String}
      The password for the specified user.
    • use_udp: {Boolean} (optional)
      When this option is set to true Bucky Server will communicate with InfluxDB using UDP instead of TCP.
    • retentionPolicy: {String} (optional)
      The name of a retention policy that's been created in InfluxDB.
    • version: {String} (optional)
      The major version of InfluxDB that you're using (either '0.8' or '0.9'). This defaults to '0.8' if it's omitted.
  • modules:
    Defines which modules will load when BuckyServer starts.

    • app:
      List of core modules to be required by BuckyServer.
    • collectors:
      List of modules that will be used by the collectors module for consuming, formatting, and handling the data that's sent to BuckyServer.

Modules

There are a few of types of modules:

  • Logger - Use to have Bucky log to something other than the console.
  • Config - Use to have Bucky pull config from somewhere other than the default file.
  • App - Use to do things when Bucky loads and/or on requests. Auth, monitoring initialization, etc.
  • Collectors - Use to send Bucky data to new and exciting places. We can only have one logger and one config, but you can specify as many app and collector modules as you like.

All modules follow the same basic sketch. You export a method which is called when Bucky starts up. That method is provided with as much of {app, config, logger} as we have loaded so far, and is expected to call a callback when it's ready for the loading to continue.

Logger

Used to log output. Defaults to a wrapper around console.log/console.error.

Should export a function which will be called when the server is started:

module.exports = ({logger}, next) ->
  # logger is the previous logger (just the console)

  next myNewLogger

This function should call the callback with a logger object which implements log and error:

module.exports = ({logger}, next) ->
  myNewLogger = {
    log: ->
      console.log "Bucky message:", arguments...
    error: ->
      console.error "Bucky error:", arguments...
  }

  next myNewLogger

Config

By default config comes from the config files loaded using the node config module.

If specified, this module will replace that config. Please note that the list of modules comes from the config module, so the list of modules must always be specified there. All other config options can be moved to your config solution of choice using this extension point.

At HubSpot, we're believers in config which can be changed without restarting services. For this reason, the config api is a bit more complex than you might expect. A wrapper is provided in lib/configWrapper.coffee for you to use should you wish to use a simpler solution.

module.exports = ({config, logger}, next) ->
  # config is the old config which was being used
  # before this module was loaded

  # logger.log and logger.error should be used rather than
  # console

  next myConfigObject

A config value will be retrieved from the config object the callback is called with like this:

  config.get('some.config').get()

  config.get('some.config').on 'change', ->
    # The config changed!

You are free to implement the on method as a dud if live reloading doesn't make sense using your config system. Take a look at lib/configWrapper.coffee for an example of how a basic object can be converted (and feel free to use it).

App

App modules get loaded once, and can optionally provide a function to be ran with each request.

Simple app modules are a good place to put any server config, initialization code, etc.

We use app modules to make little tweaks to how express works and enable monitoring.

App modules are called at initialize-time with a hash including a reference to the express app:

module.exports = ({app, logger, config}, next) ->

If your app module calls the callback with a function, that function will be executed on all requests to /v1/send, which is the default endpoint.

If the callback is called with a hash, it is expected to be a mapping between endpoints and handler functions.

module.exports = ({app, logger, config}, next) ->
  next
    send: (req, res, _next) ->
      # Standard express request handling stuff in here

    someOtherEndpoint: (req, res, _next) ->
       # Will get requests which are sent to /v1/someOtherEndpoint

These functions work like middleware, they are called sequentially. You can use them to implement things like auth if you need it.

Collectors

It's not a standard type of module (the core of Bucky has no idea about it), but the default collectors app module looks to a fourth type of module to know where to send data.

Statsd and OpenTSDB collectors are included.

Collectors should export a function which is called on initialize, and call the callback with a hash mapping endpoints to handlers.

module.exports = ({app, logger, config}, next) ->
  next
    send: (data) ->
      # This collector will receive any requests to /v1/send (the default endpoint)

      logger.log "We got some data!"

Format

If you are interested in writing new clients, there are two endpoints for inbound data. The default endpoint uses the same format as statsd:

default endpoint: {hostname}:{port}/v1/{appRoot} uses

<metric name>:<metric value>|<unit>[@<sample rate>]

For example:

my.awesome.metric:35|ms
some.other.metric:3|c@0.5

All post reqeusts sent to the default endpoint must use content-type text/plain.

JSON endpoint: {hostname}:{port}/v1/{appRoot}/json uses

{
  "<metric name>": "<metric value>[|<unit>[@<sample rate>]"
}

This allows for the ':' character to be included in your metrics. This is valid for InfluxDB Line Protocol.

For example:

{
  "page,browser=Chrome,browserVersion=44,url=http://localhost:3000/#customHash/%7Bexample%3A%22encoded%20data%22%7D,key=domContentLoadedEventEnd": "500|ms",
  "ajax,browser=Microsoft\\ Internet\\ Explorer,browserVersion=8,url=http://localhost:3000/#customHash/%7Bexample%3A%22encoded%20data%22%7D,endpoint=your/awesome/template.html,method=GET,status=200": "1|c"
}

All post request to the json endpoint will be converted to content-type 'application/json'. This allows for backwards compatibility with IE8 which can't send XDomainRequest with a content-type other than 'plain/text'.

About

Node server that receives metric data over HTTP & forwards to your service of choice #hubspot-open-source

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • CoffeeScript 99.3%
  • JavaScript 0.7%