This repo consists of 4 related projects:
- api
- feeder
- server
- web-interface
The api project builds the library jar which contains classes generated by thrift. This is shared by both the thrift-server and the web-interface. The feeder application pulls information about "push" events from github and then pushes them into a mongodb instance. The thrift-server exposes an interface to allow the mongodb store to be queried. The web-interface is a presentation layer application written with Spark, Twitter Bootstrap and Ractive.js which uses this interface.
The feeder traverses pages of events by calling the github api. This has been implemented as a lazy sequence of single events which transparently loads additional pages of events as required.
First, build the api project:
1.) Download thrift tarball:
./configureensuring that java support is enabled (depends on java 1.7+ and ant).make && make install
2.) Ensure the thrift executable is available on the path.
3.) run lein jar to generate the class files from thrift definitions.
4.) run lein install to install the library in the local maven repo.
This will make the library available to the thrift-server and web-interface.
To build the server jar:
cd server
lein uberjar
To run the server: java -jar server-0.1.0-SNAPSHOT-standalone.jar
This will launch a server that will respond to thrift requests by querying mongodb.
| ENV Variables: | Defaults |
|---|---|
| APP_MONGO_HOST | "mongodb" |
| APP_MONGO_PORT | 27017 |
| APP_SERVER_PORT | 8080 |
| APP_BIND_HOST | "0.0.0.0" |
These can be overriden:
APP_BIND_HOST=127.0.0.1 APP_SERVER_PORT=9000 java -jar server-0.1.0-SNAPSHOT-standalone.jar
The .deb can be built by running release.sh script in the release directory.
To build the feeder jar
cd feeder
lein uberjar
Get an api token from github.
To run the feeder: GITHUB_TOKEN="<TOKEN>" java -jar feeder-0.1.0-SNAPSHOT-standalone.jar
This will attempt to poll for latest events and insert them into mongo.
| ENV Variables: | Defaults |
|---|---|
| APP_MONGO_HOST | "mongodb" |
| APP_MONGO_PORT | 27017 |
| APP_MAX_EVENTS | 10000 |
| APP_GITHUB_TOKEN | <no default> |
These can be overriden:
APP_MONGO_HOST=127.0.0.1 APP_MONGO_PORT=9000 APP_GITHUB_TOKEN=blah-blah-blah java -jar feeder-0.1.0-SNAPSHOT-standalone.jar
The .deb can be built by running release.sh script in the release directory.
To build the web jar
cd web-interface
mvn clean install
To run the web-interface: java -jar web-interface-0.0.1-SNAPSHOT.jar
This will serve the web frontend.
| ENV Variables: | Defaults |
|---|---|
| APP_SERVER_PORT | 8080 |
| APP_PUSH_SERVICE_PORT | 8080 |
| APP_PUSH_SERVICE_HOST | thriftserver |
| APP_GRAPHITE_HOST | graphite |
| APP_GRAPHITE_PORT | 2003 |
| APP_GRAPHITE_PREFIX | ${HOSTNAME} OR RAND UUID |
This can be overriden:
APP_SERVER_PORT=8222 java -jar web-interface-0.0.1-SNAPSHOT.jar
This requires docker to be already installed. There a 5 containers, 1 for each of the following services:
- mongodb
- feeder
- thrift-server
- web-interface
- graphite (Based on nickstenning's repo)
The application containers are based on the Ubuntu base image provided by phusion.
There is a basejava8 dockerfile that extends this for running java applications; the feeder, thrift-server and web-interface dockerfiles all extend this file.
The application dockerfiles install the associated .deb file from the dist branch of this project. The .deb files are responsible for installing the jar and it's startup script (which will be run once the container is started).
There is a script for manipulating the docker containers: docker/docker.sh.
As a one off step, the images need to be built for each container:
cd docker/
sudo ./docker.sh build
After the images have been built, containers can be built and started with the following command:
GITHUB_TOKEN="${TOKEN}" ./docker.sh start
The containers can safely be stopped and disposed of with the following command:
./docker.sh stop
It takes a bit of time for mongodb to start up once its container has run, meaning the feeder may fail the first time it is scheduled to run.
- Use logback or some clj logging framework
- Proper error handling
- Prevent duplicate events from being added to mongo
- Vagrant?
- Add commits to frontend/pagination
- Add serverside pagination
Copyright © 2015 AL
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.