Skip to content
This repository has been archived by the owner on Dec 5, 2023. It is now read-only.

cirrus-geo/cirrus-lib

Repository files navigation

build-status-image coverage-status-image pypi-version

Cirrus Library

A common set of functions used by the Cirrus geospatial pipeline. This library are convenience functions for interacting with the various Cirrus AWS resources (queues, Lambdas, Batch, DynamoDB, etc.) and should be preferred over writing separate code where possible.

Installation

Cirrus-lib requires boto3-utils, a library of convenience functions built upon boto3. Cirrus-lib is published to PyPi which is the preferred method of installing:

$ pip install cirrus-lib

# install specific version
$ pip install cirrus-lib==0.1.0

# in a requirements.txt file - the tilda (~) will install the highest compatible version
cirrus-lib~=0.1.0

Modules

Module Description
process_payload ProcessPayload and ProcessPayloads classes representing a Cirrus input payloads and a set of ProcessPayloads
errors Custom Error classes
statedb The StateDB class for interacting with the Cirrus state database (DynamoDB)
transfer Convenience functions for uploading and download assets in STAC Items
utils Miscellaneous utility functions

process payload

errors

The errors module contains a single class: InvalidInput, which can be thrown from a task if there is a problem with the input to the task. The Item will be marked as INVALID in the state database rather than FAILED to avoid running it again.

statedb

The StateDB class from the statedb module is used to interact with the Cirrus state database (DynamoDB): creating new items in the database, querying, and updating statuses. Each item in the state database represents one input payload into Cirrus. The StateDB class is abstracted from what is actually stored in the DynamoDB, which is optimized for queries. The actual entry in the database can be retrieved with the StateDB.get_dbitem() and StateDB.get_dbitems() functions which are used internally, but clients should generally opt to use StateDB.get_items(), which translates the dbitem into:

Field Name Type Description
payload_id string The ID of the Cirrus input payload
collections string A list of all the collections used by the input items
workflow string The name of the workflow used to process the input data
items string A list of all the input Item IDs
payload string The s3 URL of the original input payload JSON
state string One of: 'PROCESSING', 'COMPLETED', 'FAILED', 'INVALID'
created string The datetime that this item was created in the database
updated string The datetime stamp that this item's state was last updated
executions [string] A list of http URLs to the State Machine executions, in chronological order
outputs [string] If state is COMPLETED, a list of s3 (if private payload) or http (if public) URLs of all STAC Items generated by this process.
last_error string This is the error message from the last FAILED or INVALID execution, if any. It is not cleared if state is PROCESSING or COMPLETED

payload_id: The payload ID is formed by a combination of all the input collections, output collections, and the name of the workflow.

transfer

The transfer module contains convenience functions for transferring data.

Function Description
get_s3_session Get a boto3-utils s3 object for transfer (used by the other functions) to/from s3
download_from_http Download a file from an HTTP url
download_item_assets For a given STAC Item download 1 or more assets to a local path
upload_item_assets For a given STAC Item upload 1 or more assets to s3

utils

This module contains miscellaneous utility functions.

Function Description
submit_batch_job Start a Cirrus Batch job with a given payload

About

Cirrus is an open-source pipeline for processing geospatial data in AWS. Cirrus was developed by Element 84 originally under a [NASA Access project].