sftp_downloader

This code may help to download IMAGEN data via sftp. You need to....

...have login credentials and enter these into the file secret.py (use secret_template.py to create it.) Caution: never push the file with your login credentials to a public github repository
...know what you want to download, look at the folder structure using an sftp browser before using this script

The downloader assumes the following folder structure on the sftp server:

remote_path/{time_dirs}/{overall_dirs}
or
remote_path/{time_dirs}/intermed_dir/{subjs}/{dirs}/{subj_files}

There are four modes that help to do different things

mode = "dirs"   # "files" or "dirs" or "subjects" or "overall"

"overall" mode: download a set of given folders recursively (including all subdirectories and files)
"subjects" mode: download given subject folders recursively (including all subdirectories and files)
"dirs" mode: download specific subdirectories within subject folders recursively
"files" mode: download files which match specific patterns within specific subdirectories within subject folders

clone the repository
use secret_template.py to create a new file secret.py
enter your login information in secret.py
use settings_template.py to create a new file settings.py
enter your local path settings and your download definitions in settings.py (some examples are given in settings_template.py)
start the script:

python get_data.py

logfiles are created for basic information (info_logger*) and debugging information (debug_logger*) to check if something did not work.

pysftp may not work properly with windows (e.g. recursive downloads may be buggy)
download is not super fast (own experience: 1.5GB per hour)

Provide feedback