Note: Inspired by pyfab-dev, by Matthieu Morlon.
This tool was created to:
- shorten the time to run regular tasks
- no need to remember all the steps to do a regular task
- one command to consume a queue, without adding it into supervisor
Specification page here.
Please feel free to improve it via PRs.
If you don't have pip installed, please start by installing it:
sudo aptitude install python-pip
After you're done, install the libraries needed by the project:
sudo pip install -r requirements.txt
If you don't have pip installed, please go ahead and install it:
sudo easy_install pip
After you're done, install the libraries needed by the project:
cd ~/Work/bbc-dev-companion
sudo pip install -r requirements.txt
Now, we need a nice tool to send notifications when stuff is completed. Go ahead and install terminal-notifier:
brew install terminal-notifier
Start by configuring the application:
cp config.sample.py config.py
vim config.py
All workers sit in the workers directory, all admin related tools in the admin folder, etc.
The configuration is very flexible and it allows you to have any name for your directories, as long as you respect the purpose of the category.
Now, let's start configuring them one by one:
- DIR_WORK - this is the directory where you have all of your Blablacar projects
- DIR_CATEGORIES - the folders where your projects reside. When you see categories think of the ones used in stash ;)
- REMOTE_USER - the remote user that should be used when connection to comtools1. E.g.: anonymouse
Then, create an alias in your ~/.bashrc
or ~/.zshrc
file:
alias bbcdc='fab -f ~/Work/bbc-dev-companion/fabfile.py'
Get a list of tasks that you can use:
bbcdc --list
To run a specific task:
bbcdc <task_name>
Note: If you are in the project's folder you can use fab
to run the tasks. Otherwise, you need to use the bbcdc
alias.
bbcdc build.satis
Update the composer repository using satis.
bbcdc v3.fixtures
Apply the most recent fixtures to the V3 database.bbcdc v3.fixtures:-h
Give an extra command/arguments to the fixtures binary.bbcdc v3.warm_up|wu
Warm up the v3 cache for the dev envbbcdc v3.warm_up|wu:test
Warm up the v3 cache for the test envbbcdc v3.elasticsearch|es
Reset the elastic search mappings.bbcdc v3.reset
Run a reset for v3, for the dev envbbcdc v3.reset:test
Run a reset for v3, for the test envbbcdc v3.run_behat|rb
Init the behat containers in order to run Behat tests.bbcdsc v3.worker|w
Start the global worker for V3.
bbcdc worker.execute|e:segmentation
One command to consume queue using a worker. What it does: checkouts the segmentation worker, if it doesn't exist. If it does, run agit pull
,composer install
and aconsume
on the related queue. Timeout is set to one day.bbcdc worker.execute|e:segmentation,"branch_name"
Checkout the given branch and run the worker for it.
bbcdc worker.execute_remote|er:user-id-check
Remotely consume a worker's dl queue from production.bbcdc worker.execute_remote|er:user-id-check,retry_3
Remotely consume a worker's retry_3 queue.bbcdc worker.execute_remote|er:user-id-check,"dl","-m\=1 -r"
Extra options for the consume command. In this example, consume one message and return it to the queue if an error occurred.
bbcdc command.execute|e:indexer-contact
One command to run a bbc command. Same behaviour as for the worker. It will display the available commands.bbcdc command.execute|e:indexer-contact,check:indexation
Run a specific sub command for the given command.
bbcdc command.mse:1,/tmp/ids.csv
Runs the member search export command for an ES filter id.bbcdc command.execute_remote|er:total-ftp,'-h'
Run a given production command. Use the second argument, (-h in the example), to call sub commands.
bbcdc service.pm:start|stop,dev|test
Control the start or stop of the private messages service, for the dev or test environments.
bbcdc slot.init|i:6
Init a slot on comtools1(reserved via the mep web page), by running:git fetch
, checking out a branch(master, if not specified) and initializing the slot.bbcdc slot.init|i:6,"branch_name"
Init a slot using a given branch name.bbcdc slot.init_profiling|ip:22
Initialize the profiling environment for a given slot.bbcdc slot.reinit|ri:6
Re-initialize a slot.bbcdc slot.cmd:6,"composer install"
Run a given command on a specific slot.bbcdc slot.errors|err
Runtail -f
on thephp_errors.log
file on comtools1.bbcdc slot.free
Get a list of free(pointed to the master branch) slotsbbcdc slot.free:41
Check whether a slot is free.
Note: Use Ctrl + B, D to detach from the tmux session so that it's not re-created every time.
bbcdc deploy.v3
Deploy or finalize V3.bbcdc deploy.fac
Force the flushing of the admin.js and admin.css files in order for the UI changes pushed in production, for the Admin panel, to go live.bbcdc deploy.pa3
Build and deploy PA3 containers for V3.
bbcdc deploy.worker|w:user-id-check
Deploy a worker on the dev env.bbcdc deploy.worker|w:user-id-check,preprod
Deploy a worker on the preprod env.bbcdc deploy.worker|w:user-id-check,preprod,some_branch
Deploy a worker on the preprod env, using a given branch.bbcdc deploy.worker_envs|we:user-id-check
Deploy a worker on all environments(dev, preprod).bbcdc deploy.worker_envs|we:user-id-check,some_branch
Deploy a worker on all environments(dev, preprod), using a given branch.
bbcdc deploy.command|c:user-update-linkedin-information
Deploy a command on the dev env.bbcdc deploy.command|c:user-update-linkedin-information,preprod
Deploy a command on the preprod env.bbcdc deploy.command_envs|ce:user-update-linkedin-information
Deploy a command on all environments(dev, preprod).
- Add a command that generates the configuration for a project in misc and copies the new configuration files in the correct folder.
- 1st stage: Autocomplete for all projects that exist in the corresponding category folders(workers, commands, admin, micro-services) so that you can easily run a worker without manually typing its name.
- 2nd stage: Autocomplete for all projects that exist in the corresponding categories on stash(workers, commands, admin, micro-services) so that you can easily run a worker without manually typing its name.
- 3rd stage: Autocomplete for all branches of the current project(worker, command, etc.).
- Create a new project using the skeleton projects
- gateway-dev project - add it to satis, bamboo(?)
- worker/ command - add it to the misc folder, bamboo(?)
- Be able to tell in what projects the gateway and/or gateway-dev-* repo is used.
- Your suggestions are more than welcome for new tasks, new ways of improving the code etc.
- Please use PEP 8 as a style guide.
- Do not name a task run since it will be ran as the default one every time you intend to execute a command from a task file
v3.run_behat|rb
doesn't work on Mac OS X right now.