Skip to content

Latest commit

 

History

History
329 lines (245 loc) · 14 KB

README.md

File metadata and controls

329 lines (245 loc) · 14 KB
Raw

Quick project reference

About Taranis NG

Taranis NG is an OSINT gathering and analysis tool for CSIRT teams and organisations. It allows osint gathering, analysis and reporting; team-to-team collaboration; and contains a user portal for simple self asset management.

Taranis crawls various data sources such as web sites or tweets to gather unstructured news items. These are processed by analysts to create structured report items, which are used to create products such as PDF files, which are finally published.

Taranis supports team-to-team collaboration, and includes a light weight self service asset management which automatically links to the advisories that mention vulnerabilities in the software.

Deploying Taranis NG with docker-compose

Taranis NG supports deployment in Docker containers. The docker/ folder on GitHub repository contains a sample docker-compose.yml file which runs the whole application in one stack.

The same folder also contains additional support files for the creation of the Docker containers. These include start and pre-start scripts, the application entrypoint, and the gunicorn configuration file.

Prerequisites

or

  • Compose V2, which is part of standard Docker Engine installation
  • (Optional) Vim or other text editor - for configuration and development

Please note it is important to use the abovementioned version of docker-compose or newer, otherwise the build and deploy will fail.

Quickly build and run Taranis NG using docker-compose or docker compose

First, you need to clone the source code repository:

git clone https://github.com/SK-CERT/Taranis-NG.git
cd Taranis-NG

Then, remove .example extension from file docker/.env.example and files in docker/secrets. Use your favorite text editor and change default passwords. Taranis NG uses Docker secrets to store sensitive data. (Saving passwords in variables defined in docker/.env is not advised and you will need to modify Docker compose YAML files to make it work correctly. Also, make sure you do not have both POSTGRES_PASSWORD and POSTGRES_PASSWORD_FILE set - they are mutually exclusive)

vim docker/.env

Optionally: you may modify other settings in the docker/.env and docker/docker-compose.yml files to your liking. More information on container configuration can be found here.

Finally, either deploy the ready-made images from Docker hub with:

docker-compose -f docker/docker-compose.yml pull
docker-compose -f docker/docker-compose.yml up --no-build

or

docker compose -f docker/docker-compose.yml pull
docker compose -f docker/docker-compose.yml up --no-build

or, alternatively, build and run the containers with:

TARANIS_NG_TAG=build docker-compose -f docker/docker-compose.yml build --pull
TARANIS_NG_TAG=build docker-compose -f docker/docker-compose.yml up

or

TARANIS_NG_TAG=build docker compose -f docker/docker-compose.yml build --pull
TARANIS_NG_TAG=build docker compose -f docker/docker-compose.yml up

(--pull updates the base images)

Voila, Taranis NG is up and running. Visit your instance by navigating to https://localhost:4443/ using your web browser.

Your Taranis NG instance now needs to be configured. Continue here.

The default credentials are user / user and admin / admin.

Advanced build methods

Individually build the containers

To build the Docker images individually, you need to clone the source code repository.

git clone https://github.com/SK-CERT/Taranis-NG.git

Afterwards go to the cloned repository and launch the docker build command for the specific container image, like so:

cd Taranis-NG
docker build -t taranis-ng-bots . -f ./docker/Dockerfile.bots
docker build -t taranis-ng-collectors . -f ./docker/Dockerfile.collectors
docker build -t taranis-ng-core . -f ./docker/Dockerfile.core
docker build -t taranis-ng-gui . -f ./docker/Dockerfile.gui
docker build -t taranis-ng-presenters . -f ./docker/Dockerfile.presenters
docker build -t taranis-ng-publishers . -f ./docker/Dockerfile.publishers

There are several Dockerfiles and each of them builds a different component of the system. These Dockerfiles exist:

Configuration

Container variables

redis

Any configuration options are available at https://hub.docker.com/_/redis.

database

Any configuration options are available at https://hub.docker.com/_/postgres.

core

Environment variable Description Example
REDIS_URL Redis database URL. Used for SSE events. redis://redis
DB_URL PostgreSQL database URL. 127.0.0.1
DB_DATABASE PostgreSQL database name. taranis-ng
DB_USER PostgreSQL database user. taranis-ng
DB_PASSWORD PostgreSQL database password. supersecret
DB_POOL_SIZE SQLAlchemy QueuePool number of active connections to the database. 100
DB_POOL_RECYCLE SQLAlchemy QueuePool maximum connection age. 300
DB_POOL_TIMEOUT SQLAlchemy QueuePool connection timeout. 5
JWT_SECRET_KEY JWT token secret key. supersecret
OPENID_LOGOUT_URL Keycloak logout URL. https://example.com/auth/realms/master/protocol/openid-connect/logout
WORKERS_PER_CORE Number of gunicorn worker threads to spawn per CPU core. 4
SKIP_DEFAULT_COLLECTOR Set to true to prevent initialization of a default docker collector at first run ``

Taranis NG can use connection pooling to maintain multiple active connections to the database server. Connection pooling is required when your deployment serves hundreds of customers from one instance. To enable connection pooling, set the DB_POOL_SIZE, DB_POOL_RECYCLE, and DB_POOL_TIMEOUT environment variables.

bots, collectors, presenters, publishers

Environment variable Description Example
TARANIS_NG_CORE_URL URL of the Taranis NG core API. http://127.0.0.1:8080/api/v1
API_KEY Shared API key. cuBG/4H9lGTeo47F9X6DUg
WORKERS_PER_CORE Number of gunicorn worker threads to spawn per CPU core. 4

gui

Environment variable Description Example
VUE_APP_TARANIS_NG_CORE_API URL of the Taranis NG core API. http://127.0.0.1:8080/api/v1
VUE_APP_TARANIS_NG_CORE_SSE URL of the Taranis NG SSE endpoint. http://127.0.0.1:8080/sse
VUE_APP_TARANIS_NG_URL URL of the Taranis NG frontend. http://127.0.0.1
VUE_APP_TARANIS_NG_LOCALE Application locale. en
NGINX_WORKERS Number of NginX worker threads to spawn. 4
NGINX_CONNECTIONS Maximum number of allowed connections per one worker thread. 16

Management script how-to

Taranis NG core container comes with a simple management script that may be used to set up and configure the instance without manual interaction with the database.

To run the management script, launch a shell inside of the docker container for the core component with this command:

docker exec -it [CONTAINER] python manage.py [COMMAND] [PARAMETERS]

Currently, you may manage the following:

Command Description Parameters
account (WIP) List, create, edit and delete user accounts. --list, -l : list all user accounts
--create, -c : create a new user account
--edit, -e : edit an existing user account
--delete, -d : delete a user account
--username : specify the username
--name : specify the user's name
--password : specify the user's password
--roles : specify a list of roles, divided by a comma (,), that the user belongs to
role (WIP) List, create, edit and delete user roles. --list, -l : list all roles
--filter, -f : filter roles by their name or description
--create, -c : create a new role
--edit, -e : edit an existing role
--delete, -d : delete a role
--id : specify the role id (in combination with --edit or --delete)
--name : specify the role name
--description : specify the role description (default is "")
--permissions : specify a list of permissions, divided with a comma (,), that the role would allow
collector (WIP) List, create, edit, delete and update collector nodes. --list, -l : list all collector nodes
--create, -c : create a new node
--edit, -e : edit an existing node
--delete, -d : delete a node
--update, -u : re-initialize collector node
--all, -a : update all collector nodes (in combination with --update)
--show-api-key : show API key in plaintext (in combination with --list)
--id : specify the node id (in combination with --edit, --delete or --update)
--name : specify the node name
--description : specify the collector description (default is "")
--api-url : specify the collector node API url
--api-key : specify the collector node API key
dictionary Update CPE, CWE and CVE dictionaries. --upload-cpe : upload the CPE dictionary (expected on STDIN in XML format) to the path indicated by CPE_UPDATE_FILE environment variable, and update the database from that file.
--upload-cve : upload the CVE dictionary (expected on STDIN in XML format) to the path indicated by CVE_UPDATE_FILE environment variable, and update the database from that file.
--upload-cwe : upload the CWE dictionary (expected on STDIN in XML format) to the path indicated by CWE_UPDATE_FILE environment variable, and update the database from that file.
apikey List, create and delete apikeys. --list, -l : list all apikeys
--create, -c : create a new apikey
--delete, -d : delete a apikey
--name : specify the apikey name
--user : specify the user's name
--expires : specify the apikey expiration datetime

Example usage

Create a new role with a set of permissions
manage.py role \
    --create \
    --name "Custom role 1" \
    --description "Custom role with analysis and assessment access" \
    --permissions "ANALYZE_ACCESS, ANALYZE_CREATE, ANALYZE_UPDATE, \
    ANALYZE_DELETE, ASSESS_ACCESS, ASSESS_CREATE, ASSESS_UPDATE, \
    ASSESS_DELETE, MY_ASSETS_ACCESS, MY_ASSETS_CREATE, MY_ASSETS_CONFIG"

Command output:

Role 'Custom role 1' with id 3 created.
Role filter
manage.py role \
    --list \
    --filter "Custom role 1"

Command output:

Id: 3
	Name: Custom role 1
	Description: Custom role with analysis and assessment access
	Permissions: ['ANALYZE_ACCESS', 'ANALYZE_CREATE', 'ANALYZE_UPDATE', 'ANALYZE_DELETE', 'ASSESS_ACCESS', 'ASSESS_CREATE', 'ASSESS_UPDATE', 'ASSESS_DELETE', 'MY_ASSETS_ACCESS', 'MY_ASSETS_CREATE', 'MY_ASSETS_CONFIG']
Create a new collector node
manage.py collector \
    --create \
    --name "Docker collector" \
    --description "A simple collector hosted in a Docker container" \
    --api-url "http://collector.example.com" \
    --api-key "supersecret"

Command output:

Collector node 'Docker collector' with id 1 created.
Re-initialize a collector node
manage.py collector \
    --update \
    --name "Docker"

Command output:

Collector node 1 updated.
Collector node 2 updated.
Unable to update collector node 3.
    Response: [401] ""
Create a new user account
manage.py account \
    --create \
    --name "John Doe" \
    --username "test_user" \
    --password "supersecret" \
    --roles 3

Command output:

User 'test_user' created.
Upload a CPE dictionary
zcat official-cpe-dictionary_v2.3.xml.gz | manage.py dictionary --upload-cpe

Command output:

Processed CPE items: 1000
Processed CPE items: 2000
...
...
Processed CPE items: 789000
Processed CPE items: 789704
Dictionary was uploaded.
Create new ApiKey
manage.py apikey \
    --create \
    --name "My ApiKey"
Create a new API key for a user with an expiration date
manage.py apikey \
    --create \
    --name "My ApiKey" \
    --user "test_user" \
    --expire "2022-12-31 16:55"

Command output:

ApiKey 'My ApiKey' with id 3 created.