Project Leader Vincent Ferretti
Author Ivan Borozan
CSSSCL is a python package that uses Combined Sequence Similarity Scores for accurate taxonomic CLassification of long and short reads.
Distributor ID: Debian/Ubuntu
Description: Debian GNU/Linux 8.1 (jessie) / Ubuntu /12.04.3 LTS /14.04.1/19.04
Release: 8.1 64-bit / 12.04 64-bit / 14.04 64-bit
Codename: jessie / precise / trustyPython = 2.7.9 (biopython==1.67, Cython==0.29.10, numpy==1.9.2, pymongo==2.9.5, pysam==0.15.2, python-dateutil==2.5.3, scikit-learn==0.17.1, scipy==0.15.1, six==1.12.0)
We have setup three ways for installing the cssscl package:
- Quick deployment using Docker (small file).
- System wide installation from the source code (see the Installation Guide below).
| Installation Guide |
|---|
Option A - Installing cssscl using the Python's Virtual Environment |
We recommend to install the |
Note: if any of the following packages: jellyfish, BLAST or plzip are already installed on your system make sure that they are in your executable search path (i.e. PATH variable) (as shown in the examples below): |
BLAST |
|
|
jellyfish |
|
|
plzip |
|
|
Step 1. Install dependencies on Debian and Ubuntu |
In order to compile |
|
|
Step 2. Download the |
|
|
|
|
|
|
Step 3. Check that all packages necessary to run the |
|
|
Note: when prompted follow instructions to export when |
Note: for more information regarding the |
Step 4. In the |
|
Step 5. To begin using the virtual environment, it first needs to be activated as shown below: |
|
Step 6. Install |
|
Note: this will install all the python modules necessary for running the |
Step 7. Configure |
|
Accept all the values prompted by default by pressing [ENTER] |
Note: If you are done working in the virtual environment, you can deactivate it as shown below. |
|
If you would like to run the |
Option B - Install cssscl without using the Python's Virtual Environment |
Install the |
Note: if any of the following packages: jellyfish, BLAST or plzip are already installed on your system make sure that they are in your executable search path (i.e. PATH variable) (as shown in the examples below): |
BLAST |
|
|
jellyfish |
|
|
plzip |
|
|
Step 1. Install dependencies on Debian and Ubuntu |
Python: Only Python 2.7.3+ is supported. No support for Python 3 at the moment. |
In order to compile |
|
|
Step 2. Download the |
|
|
|
|
|
|
Step 3. Check that all packages necessary to run the |
|
|
Note: when prompted follow instructions to export when |
Note: for more information regarding the |
Step 4. Install |
|
Step 5. Configure |
|
Accept all the values prompted by default by pressing [ENTER] |
Additional instructions for non-automated installation of third party software necessary for running the cssscl package |
In case the cssscl_check_pre_installation.sh script (see the installation subsections above) fails please read the info below for the manual installation of individual third party software: |
Necessary Python modules: |
| - BioPython - Tools for biological computation. |
| - PyMongo - Python module needed for working with MongoDB (PyMongo = 2.8) |
| - Sklearn - Machine Learning in Python |
| - Numpy - NumPy is the fundamental package for scientific computing with Python |
| - Cython - Cython is an optimising static compiler for both the Python programming language and the extended Cython programming language (based on Pyrex) |
|
Installing python modules using pip manually: |
|
|
|
|
|
|
Third party software: |
| BLAST (version 2.2.30+ and higher) |
| Basic Local Alignment Search Tool. |
http://blast.ncbi.nlm.nih.gov/Blast.cgi?PAGE_TYPE=BlastDocs&DOC_TYPE=Download |
| JELLYFISH (version 1.1.+ but not 2.0.+) |
| JELLYFISH is a tool for fast, memory-efficient counting of k-mers in DNA. |
| PLZIP (version 1.1+) |
| Plzip is a massively parallel (multi-threaded) lossless data compressor based on the lzlib compression library, with a user interface similar to the one of lzip, bzip2 or gzip. |
Note: that the classification results in the paper were obtained using: Plzip 1.1 using Lzlib 1.5 |
To compile Plzip 1.1 and Lzlib 1.5: |
Step 1. Donwload lzlib-1.5.tar.gz |
|
Step 2. Install lzlib-1.5: |
|
|
|
|
|
|
Step 3. Donwload Plzip 1.1 |
|
Step 4. Install Plzip |
|
|
|
|
|
|
| For more information about plzip consult: |
| and for memory required to compress and decompress: |
http://www.nongnu.org/lzip/manual/plzip_manual.html#Memory-requirements |
Make sure that JELLYFISH, BLAST and Plzip are in your executable search path (see the examples below): |
|
|
|
|
Install MongoDB |
MongoDB should be installed using the following set of instructions: |
Ubuntu 12.04.3 LTS /14.04.1 |
First add the 10gen GPG key, the public gpg key used for signing these packages. It should be possible to import the key into apt's public keyring with a command like this: |
|
Add this line verbatim to your |
|
In order to complete the installation of the packages, you need to update the sources and then install the desired package |
|
|
Ubuntu 19.04 |
|
|
|
|
Start mongo service |
|
Debian |
|
|
|
|
Uninstall cssscl |
Note: this will only work if you installed cssscl with the cmd |
|
|
| ========== |
| User Guide |
Download taxon data:
https://drive.google.com/open?id=1okbaJkv6IgvWf8R1A97CX9lq10wV_INY
$ tar -zxvf taxon.tar.gz
Download test/train data:
https://drive.google.com/open?id=1glzuBJAqf5MPuO5_ivaFFLnZxpjnamFc
$ tar -zxvf test_data.tar.gz
Example 1 - run the cssscl classifier without the optimization using the taxon data and the test set provided
Step 1. Build the necessary databases from the training set
$ cssscl build_dbs -btax -c -blast -nt 2 PATH_TO/test_data/TRAIN.fa PATH_TO/taxon/
(the whole process should take ~ 37 min using 2 CPUs)
By default all databases will be outputted to the directory where the TRAIN.fa resides (note that all paths provided in the examples above are using absolute/full paths to the files/directories). The above command will build three databases (blast, compression and the kmer database) for sequences in the training set.
The cssscl's build_dbs module requires two positional arguments to be provided:
1. a file in the fasta format (e.g. TRAIN.fa as in the example above) that specifies the collection of reference genomes composing the training set.
2. a directory (taxon/ in the example above) that specifies the location where the taxon data is stored (more specifically the directory should contain the following files: gi_taxid_nucl.dmp, names.dmp and nodes.dmp, these files can be downloaded from the NCBI taxonomy database at ftp://ftp.ncbi.nlm.nih.gov/pub/taxonomy/).
The information about the additional optional arguments used in the command line above is provided here.
For more information please consult the cssscl's build_dbs help page by typing:
$ cssscl build_dbs --help
Step 2. Perform the classification of the sequences in the test set
# use cssscl to classify sequences in TEST.fa $ cssscl classify -c -blast blastn -tax genus -nt 2 PATH_TO/test_data/test/TEST.fa PATH_TO/test_data/
(the whole process should take ~ 29 min using 2 CPUs)
Note that in the above example the output file cssscl_results_genus.txt with classification results will be located in the directory where the TEST.fa resides.
Note: For the test set data provided above the values of the parameters used in the model have already been optimized and are included as part of the test set (see the optimum_kmer directory in the test_set/ directory provided). Thus for the test dataset the optimization is not required to be performed prior to running the classifier. On how to run the classifier by performing the optimization stage first please see the step 3 below.
The cssscl's classify module requires two positional arguments to be provided:
1. a file with test data with sequences in the FASTA format for classification (e.g. TEST.fa as in the example above)
2. a directory where the databases (built using the training set) reside
Note: This will run the classifier with all the similarity measures (including the compression and the blast measure) as described in: Borozan I et al. "Integrating alignment-based and alignment-free sequence similarity measures for biological sequence classification." Bioinformatics. 2015 Jan 7. pii: btv006.
The information about the additional optional arguments used in the command line above is provided here.
For more information please consult the cssscl's classify help page by typing
$ cssscl classify --help
Step 1. Build the necessary databases from the training set
Note: Only do this is you did not already built the database in Example 1 above.
$ cssscl build_dbs -btax -c -blast -nt 2 PATH_TO/test_data/TRAIN.fa PATH_TO/taxon/
(the whole process should take ~ 37 min using 2 CPUs)
Step 2. Perform the classification of the sequences in the test set by optimizing the cssscl's parameter values first
$ cssscl classify -c -blast blastn -opt -tax genus -nt 8 PATH_TO/test_data/test/TEST.fa PATH_TO/test_data/
More information about the optimization can be found here.
Note that the optimization phase will take considerably longer when -c (compression) argument is used as mentioned in the section Note regarding the compression measure below.
The information about the additional optional arguments used in the command line above is provided here.
The use of the compression measure will slow down considerably the optimization and the classification parts because of the running time complexity ~ O(n*n) (for the optimization phase) and ~ O(n*m) for the classification phase, where n and m are respectively the number of sequences in the training and test sets. Thus the compression measure should only be used with smaller genome databases (e.g. viruses) and/or with smaller datasets (i.e. smaller number of reads/contigs to classify).
Licensed under the GNU General Public License, Version 3.0. See LICENSE for more details.
Copyright 2015 The Ontario Institute for Cancer Research.
This project is supported by the Ontario Institute for Cancer Research (OICR) through funding provided by the government of Ontario, Canada.