This code is associated with the following manuscript. If you use any part of the source code, please cite us:
Sumaiya Nazeen, Yun William Yu, and Bonnie Berger*. 
"Carnelian uncovers hidden functional patterns across diverse study populations from whole metagenome sequencing reads." (Accepted for publication in Genome Biology). 

A preliminary version of the paper was presented at ECCB 2018 (Applications Track) and bioRxiv preprint can be found at https://doi.org/10.1101/375121

Upon publication, further information can be found at http://carnelian.csail.mit.edu/

0. Requirments
    Vowpal Wabbit 8.1.1
    scikit-learn
    R 3.3.2
    Python 2.7.13
    BioPython 1.70
    FragGeneScan

    This code has been tested with GCC 6.3.0 on Ubuntu 17.04, running
    under Bash 4.4.7(1) on a server with Intel Xeon E5-2695 v2 x86_64 2.40 GHz 
    processor and 320 GB RAM.
    
    Using the EC-2010-DB dataset as gold standard, Carnelian can be comfortably
    run on a machine with 16GB RAM using 1 CPU.

1. Directory structure
data/: EC-2010-DB dataset with gold standard EC labels.
scripts/: R scripts for abundance estimation and analysis from read counts in functional bins.
util/
    ext/: external libararies.
    test/: test drawfrag.c and fasta2skm.c
    drawfrag.c: draw fragments from fasta records.
    fasta2skm.c: construct feature (spaced k-mer profile), and convert to VW input format.
    ldpc.py: generate LSH function using LDPC code.
    sequtil.py: splitting and merging utilities for fasta files.
    merge_pairs.py: links paired-end read files using paired-end relationships.
    reduce.py: translate sequences using reduced amino acid alphabets.
    kseq.h: parse FASTA files
tests/
    demo_data/: data files required for unit and advanced tests.
    basictest_carnelian.py: contains the unit tests for Carnelian.
    advancedtest_carnelian.py: contains the end-to-end tests for Carnelian.
    config.py: configures unit tests and advanced tests for Carnelian.
    README.txt: contains the instructions to run the tests.
    
2. Install and test:
   bash SETUP.sh

3. Usage:

Modes:
    (default --optional-arguments such as k-mer length, fragment size,
    hash functions, etc. are set to work best with EC-2010-DB as used in the manuscript.
    If you're going to train on a different dataset, be sure to tune parameters.)

    1)  ./carnelian.py frag [--optional-arguments] test_dir frag_dir [-h]
        Looks for a fasta file in test_dir with matching label file.
        Randomly draws fragments of length and coverage specified in
        optional-arguments. (use "./carnelian.py frag -h" for details)

        Outputs these fragments with corresponding label into frag_dir.

    2) ./carnelian.py train [--optional-arguments] train_dir model_dir [-h]

        Looks for a fasta file in train_dir with matching label file.
        For each batch of training, randomly draw fragments and generate
        feature vectors using Opal LDPC hashes, and trains Vowpal_Wabbit
        One-Against-All classifier against all batches sequentially. To train 
        classifiers in precise mode, use "--precise" option which will make
        the learned model store probabilities.

        Outputs the generated classifier model into model_dir.

    3) ./carnelian.py retrain [--optional-arguments] old_model_dir new_model_dir new_exmpls_dir [-h]

        Looks for a vowpal-wabbit model with patterns and dictionary file
        in the old_model_dir and a fasta file with matching labels in the
        new_exmpls_dir. Starting with the old model, it updates the existing
        training model and merges new labels with old dictionary using the old
        LDPC patterns. Note that a model trained in default mode must be updated
        in default mode. Same is true for precise mode.
        Output model, dictionary, and pattern files will be generated in new_model_dir.
	
    4) ./carnelian.py translate [--optional-arguments] seq_dir out_dir fgsp_loc [-h]
    	
	Using FragGeneScan program located in the fgsp_loc directory, tries to find
	coding sequences in the input reads fasta file in seq_dir, and translated the
	coding sequences to possible ORFs outputting them in a fasta file in the out_dir.

    5) ./carnelian.py predict [--optional-arguments] model_dir test_dir predict_dir [-h]

        Looks for a classifier model in model_dir, and a fasta file in
        test_dir containing reads/fragments. To make predictions with probabilities,
        run in precise mode using "--precise" option and specify probability cutoff
        using "--cutoff <X>" option.
      
        Outputs the predictions in predict_dir as a fasta file with
        corresponding a corresponding label file.

    6) ./carnelian.py eval reference_file predicted_labels [-h]
    
        Evaluation of prediction accuracy in terms of micro and macro averaged
        precision, sensitivity, and F1-score. If run in "precise" mode, it will
        assume predicted_labels file to have two tab-separated columns: <readID, predLabel>

    7) ./carnelian.py abundance in_dir out_dir mapping_file gs_file [-h]

        Generates abundance estimates of functional terms. Looks for predicted labels for
        each sample in its own sub-directory in in_dir and sample mapping information
        and average protein length per label in mapping_file and gs_file respectively. Please
	note that, the sample ids must not start with digits. 

        Outputs raw counts and effective counts matrices in out_dir.

    8) ./carnelian.py simulate [--optional-arguments] test_dir train_dir out_dir [-h]

        Runs a full pipeline for performance evaluation starting from training on data 
        in train_dir, testing on data in test_dir, and outputting fragments, model and 
        predictions under out_dir in the following directory structure:

        1frag/
            simulated test data (drawn fragments) are saved here.
            (ignored if --do-not-fragment)
        2model/
            classifier will be saved here.
        3predict/
            fragment classifications are saved here.

    9) ./carnelian.py annotate [--optional-arguments] sample_dir model_dir out_dir fgsp_loc [-h]
	
	Annotates the input nucleotide reads starting from gene finding and translation
	on the reads fasta file in the sample_dir using FragGeneScan located in the fgsp_loc
	directory, then classifying the predicted ORFs using the model in model_dir, and 
	outputting the labels in the out_dir.
	
    Steps to be followed in a typical workflow is given in the workflow.txt file.
    
    To replicate our classification performance analysis the code in performance_analysis.R can be used. Before running
    the script the following packages need to be installed: 
        caret, pROC, ROCR, cvAUC, randomForest

Contact
    Sumaiya Nazeen, nazeen@mit.edu

Acknowledgement
    This implementation of Carnelian is adapted from the source code of the following papers:
    Yunan Luo, Y. William Yu, Jianyang Zeng, Bonnie Berger, and Jian Peng. Metagenomic binning through low density hashing.  Bioinformatics (2018), bty611, https://doi.org/10.1093/bioinformatics/bty611
    K. Vervier, P. Mahe, M. Tournoud, J.-B. Veyrieras, and J.-P. Vert. Large-scale Machine Learning for Metagenomics Sequence Classification , Technical report HAL-01151453, May, 2015.