Prep 0.7.0

CHANGELOG.md

@@ -6,32 +6,168 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
-### Fixed
 
-- rerun feeder has required permissions ([#131])
-- `update-state` lambda supports payload URLs ([#135])
+## [v0.7.0] - 2022-09-12
 
-## [v0.6.0] - 2022-02-18
+### ⚠️ Breaking changes
 
-### Fixed
+- Serverless versions through 3.x now supported. Minimum serverless of 2.3.0 is now
+  required per pseudo parameters now being parsed within cirrus, rather than via the
+  `serverless-pseudo-parameters` plugin. ([#139])
 
-- `cirrus build` will now rmtree for build dirs on rebulids ([#105])
+  Tested `package.json` dependency versions:
+
+  ```
+    "serverless": "~3.18.0",
+    "serverless-python-requirements": "~5.4.0",
+    "serverless-step-functions": "~3.7.0",
+    "serverless-iam-roles-per-function": "~3.2.0"
+  ```
+
+  Note that upgrading to serverless v3 changes the type of EventBridge Rules
+  resources. In testing we found that the existing `update-state` rule needed
+  to be deleted by CloudFormation before adding the new rule. Manually
+  deleteing the existing rule before deployment was not sufficient, as
+  CloudFormation ended up removing the new rule after it was created.
+
+  In short, the simplest thing to do after upgrading serverless is to deploy
+  twice, once with `update-state` disabled and again with it re-enabled.
+
+  To disable it, it is easiest to run this from your project root:
+
+  ```
+  mkdir functions/update-state
+  echo "description: temporaily disabled" > functions/update-state/definition.yml
+  ```
+
+  Then, run the deploy as normal. Once that is complete, remove the
+  `update-state` override:
+
+  ```
+  rm -r functions/update-state
+  ```
+
+  Deploy again and `update-state` and its event should be re-created
+  successfully.
+
+  Note that state tracking/workflow chaining _will be broken_ between the first
+  deploy and the second. It is _strongly_ recommended to do this only when your
+  pipelines are not processing workflows.
+
+- All lambda component definitions need the `handler` populated if not already.
+  Previously cirrus was defaulting `handler` to `lambda_function.lambda_handler`
+  if it were omitted. Now the default lambda `definition.yml` includes
+  `handler: lambda_function.lambda_handler`, allowing users to remove it if not
+  compatible with their needs (i.e., specifying a container `image`). ([#139])
+
+- S3 buckets `Data` and `Payload` are no longer defined as builtins. Projects that
+  do not otherwise define their required S3 buckets should ensure they have both
+  of these buckets defined in their cloudformation resource templates. The
+  previously-default configuration looks like this ([#147]):
+
+  ```
+  # cloudformation/s3.yml
+  Resources:
+    # Main data bucket
+    Data:
+      Type: AWS::S3::Bucket
+      Properties:
+        BucketName: "#{AWS::StackName}-data"
+    # Bucket for storing input catalogs
+    Payloads:
+      Type: AWS::S3::Bucket
+      Properties:
+        BucketName: "#{AWS::StackName}-payloads"
+        LifecycleConfiguration:
+          Rules:
+            - ExpirationInDays: 10
+              Prefix: batch/
+              Status: Enabled
+            - ExpirationInDays: 10
+              Prefix: payloads/
+              Status: Enabled
+  ```
+
+  Note that if a bucket _not part your existing cirrus project_ already exists with
+  the same name specified here, cloudformation will fail. Ensure you are only using
+  these default bucket names if your project was previously relying on these
+  built-in resources.
+
+- Batch IAM role best practices have changed, and some builtin roles have changed
+  or been removed. See [#149] for additional context. In summary:
+  - Do not specify the service role on batch compute environments. The builtin
+    `BatchServiceRole` has been removed from cirrus. The default role automatically
+    provided by `AWS` automatically is sufficient.
+  - All non-standard permissions have been removed from the `BatchInstanceRole`.
+    If you have been overriding that role with custom permissions review the new
+    `BatchJobRole` and override with any further permissions instead. Better yet,
+    create a unique role per batch task based on the `BatchJobRole`.
+  - When using `BatchJobRole` or a custom role per batch task, ensure it is specified
+    on the job definition as the `ContainerProperties` `JobRoleArn`.
+
+- The cli command to create new tasks now uses `-t`/`--type` to specify the task type,
+  instead of `--has-batch`/`--no-batch` and `--has-lambda`/`--no-lambda`. `-t`/`--type`
+  has no default value and is required. It can be specified multiple times in the case
+  of a task that uses both batch and lambda. ([#123])
+
+### ⚠️ Deprecations
+
+- `ProcessPayload.process['output_options']` has been renamed to `'upload_options'` ([#128])`
 
 ### Added
 
-- Support for cli plugins ([#138])
-- Documentation within the package for projects to build into their docs ([#138])
+- experimental support for lambdas using container images ([#139])
+- `init` creates a minimal .gitignore in the project root ([#140])
+- `init` will now create cloudformation templates for the minimum set of resources
+  not provided by builtins ([#147])
+- explicit error message when lambda package requirements have duplicates ([#106])
+- support for cirrus plugins providing components or cloudformation via the
+  `cirrus.resources` entrypoint ([#104])
+- support for cli plugins via the `cirrus.plugins` entrypoint ([#138])
+- documentation within the package for projects to build into their docs ([#138])
 - `update-state` now supports `ABORTED` and `TIMED_OUT` step function events ([#85])
 
+### Changed
 
-### Fixed
+- `BatchInstanceRole` no longer has cirrus-specific permissions. Specify a `JobRoleArn`
+  on batch job definitions pointing to the builtin `BatchJobRole` or a custom role. ([#149])
+- support for modern versions of serverless; minimum version supported is now 2.3.0 ([#139])
+- lambda components definitions require `handler` to be specified when not using container
+  images (previous default: `handler: lambda_function.lambda_handler`) ([#139])
 
-- Component README template missing space in header ([#138])
+### Fixed
 
+- `rerun` feeder has required permissions ([#131])
+- cirrus.yml default template now has missing provider vpc configuration ([#132])
+- `sls`/`serverless` command returns non-0 on errors ([#134])
+- `update-state` lambda supports payload URLs ([#135])
+- omit lambda block from batch-only task `definition.yml` on create ([#123])
+- test payloads output to non-terminal FDs will not have lines broken by terminal width ([#145])
+- component README templates missing space in header ([#138])
 
 ### Removed
 
-- Cleaned up old docs ([#138])
+- all non-core builtins ([#78]):
+  - feed-s3-inventory (feeder)
+  - feed-stac-api (feeder)
+  - feed-stac-crawl (feeder)
+  - add-preview (task)
+  - convert-to-cog (task)
+  - copy-assets (task)
+  - cog-archive (workflow)
+  - mirror-with-preview (workfow)
+  - mirror (workflow)
+- builtin S3 bucket resources ([#147])
+- builtin `BatchServiceRole` resource ([#149])
+- dependency on `servereles-pseudo-parameters` ([#139])
+- old docs ([#138])
+
+
+## [v0.6.0] - 2022-02-18
+
+### Fixed
+
+- `cirrus build` will now rmtree for build dirs on rebulids ([#105])
 
 
 ## [v0.5.4] - 2022-02-10
@@ -320,7 +456,8 @@ Initial release
 
 
 
-[Unreleased]: https://github.com/cirrus-geo/cirrus-geo/compare/v0.6.0...main
+[Unreleased]: https://github.com/cirrus-geo/cirrus-geo/compare/v0.7.0...main
+[v0.7.0]: https://github.com/cirrus-geo/cirrus-geo/compare/v0.6.0...v0.7.0
 [v0.6.0]: https://github.com/cirrus-geo/cirrus-geo/compare/v0.5.4...v0.6.0
 [v0.5.4]: https://github.com/cirrus-geo/cirrus-geo/compare/v0.5.3...v0.5.4
 [v0.5.3]: https://github.com/cirrus-geo/cirrus-geo/compare/v0.5.2...v0.5.3
@@ -350,20 +487,31 @@ Initial release
 [#65]: https://github.com/cirrus-geo/cirrus-geo/issues/65
 [#74]: https://github.com/cirrus-geo/cirrus-geo/issues/74
 [#75]: https://github.com/cirrus-geo/cirrus-geo/issues/75
+[#78]: https://github.com/cirrus-geo/cirrus-geo/issues/78
 [#79]: https://github.com/cirrus-geo/cirrus-geo/issues/79
 [#82]: https://github.com/cirrus-geo/cirrus-geo/issues/82
 [#85]: https://github.com/cirrus-geo/cirrus-geo/issues/85
 [#98]: https://github.com/cirrus-geo/cirrus-geo/issues/98
 [#99]: https://github.com/cirrus-geo/cirrus-geo/issues/99
 [#102]: https://github.com/cirrus-geo/cirrus-geo/issues/102
 [#105]: https://github.com/cirrus-geo/cirrus-geo/issues/105
+[#106]: https://github.com/cirrus-geo/cirrus-geo/issues/106
 [#107]: https://github.com/cirrus-geo/cirrus-geo/issues/107
 [#108]: https://github.com/cirrus-geo/cirrus-geo/issues/108
 [#111]: https://github.com/cirrus-geo/cirrus-geo/issues/111
 [#114]: https://github.com/cirrus-geo/cirrus-geo/issues/114
 [#116]: https://github.com/cirrus-geo/cirrus-geo/issues/116
+[#123]: https://github.com/cirrus-geo/cirrus-geo/issues/123
+[#128]: https://github.com/cirrus-geo/cirrus-geo/issues/128
 [#131]: https://github.com/cirrus-geo/cirrus-geo/issues/131
+[#132]: https://github.com/cirrus-geo/cirrus-geo/issues/132
 [#135]: https://github.com/cirrus-geo/cirrus-geo/issues/135
+[#134]: https://github.com/cirrus-geo/cirrus-geo/issues/134
+[#139]: https://github.com/cirrus-geo/cirrus-geo/issues/139
+[#140]: https://github.com/cirrus-geo/cirrus-geo/issues/140
+[#145]: https://github.com/cirrus-geo/cirrus-geo/issues/145
+[#147]: https://github.com/cirrus-geo/cirrus-geo/issues/147
+[#149]: https://github.com/cirrus-geo/cirrus-geo/issues/149
 
 [#71]: https://github.com/cirrus-geo/cirrus-geo/pull/72
 [#72]: https://github.com/cirrus-geo/cirrus-geo/pull/72

README.md

@@ -128,7 +128,7 @@ mirror-with-preview (built-in): Mirror items with selected assets
 publish-only (built-in): Simple example that just published input Collections and items
 
 # To create a new task, for example, we can do this
-❯ cirrus create task a_task "A task that doesn't do much yet"
+❯ cirrus create task -t lambda a_task "A task that doesn't do much yet"
 task a_task created
 
 ❯ cirrus show tasks

requirements.txt

@@ -3,4 +3,4 @@ click~=8.0
 click-plugins~=1.1
 rich~=10.6
 cfn-flip~=1.2
-cirrus-lib~=0.7.0
+cirrus-lib~=0.8.0

setup.py

@@ -89,5 +89,7 @@ def git_version(gitdir, default='0.0.0'):
     entry_points='''
         [console_scripts]
         cirrus=cirrus.cli.__main__:main
+        [cirrus.resources]
+        built-in=cirrus.builtins
     ''',
 )

src/cirrus/builtins/cloudformation/basic-compute-environments.yml

@@ -4,8 +4,6 @@ Resources:
     Type: AWS::Batch::ComputeEnvironment
     Properties:
       Type: MANAGED
-      ServiceRole:
-        Fn::GetAtt: [ BatchServiceRole, Arn ]
       ComputeEnvironmentName: '#{AWS::StackName}-basic-ondemand'
       ComputeResources:
         MaxvCpus: ${self:custom.batch.BasicComputeEnvironments.MaxvCpus}
@@ -35,8 +33,6 @@ Resources:
     Type: AWS::Batch::ComputeEnvironment
     Properties:
       Type: MANAGED
-      ServiceRole:
-        Fn::GetAtt: [ BatchServiceRole, Arn ]
       ComputeEnvironmentName: '#{AWS::StackName}-basic-spot'
       ComputeResources:
         MaxvCpus: ${self:custom.batch.BasicComputeEnvironments.MaxvCpus}
@@ -62,4 +58,4 @@ Resources:
           ComputeEnvironment: !Ref BasicSpotComputeEnvironment
       State: ENABLED
       Priority: 1
-      JobQueueName: '#{AWS::StackName}-basic-spot'
+      JobQueueName: '#{AWS::StackName}-basic-spot'

src/cirrus/builtins/cloudformation/resources.yml

@@ -1,22 +1,4 @@
 Resources:
-  # Main data bucket
-  Data:
-    Type: AWS::S3::Bucket
-    Properties:
-      BucketName: "#{AWS::StackName}-data"
-  # Bucket for storing input catalogs
-  Payloads:
-    Type: AWS::S3::Bucket
-    Properties:
-      BucketName: "#{AWS::StackName}-payloads"
-      LifecycleConfiguration:
-        Rules:
-          - ExpirationInDays: 10
-            Prefix: batch/
-            Status: Enabled
-          - ExpirationInDays: 10
-            Prefix: payloads/
-            Status: Enabled
   # SNS Topic for adding input data to Cirrus
   ProcessTopic:
     Type: 'AWS::SNS::Topic'
@@ -32,21 +14,6 @@ Resources:
     Type: 'AWS::SNS::Topic'
     Properties:
       TopicName: "#{AWS::StackName}-failed"
-  # Allow anyone to subscribe to publish topic
-  #PublishTopicPolicy:
-  #  Type: 'AWS::SNS::TopicPolicy'
-  #  Properties:
-  #    Topics:
-  #      - !Ref PublishTopic
-  #    PolicyDocument:
-  #      Statement:
-  #        - Sid: public-sns-subscriptions
-  #          Effect: Allow
-  #          Principal: "*"
-  #          Action:
-  #            - sns:Subscribe
-  #          Resource:
-  #            - arn:aws:sns:#{AWS::Region}:#{AWS::AccountId}:${self:service}-${self:provider.stage}-publish
   # Process Queue
   ProcessQueue:
     Type: AWS::SQS::Queue
@@ -121,14 +88,7 @@ Resources:
             ProjectionType: ALL
       BillingMode: PAY_PER_REQUEST
       TableName: "#{AWS::StackName}-state"
-  # Batch IAM Roles
-  BatchInstanceProfile:
-    Type: AWS::IAM::InstanceProfile
-    Properties:
-      Path: '/'
-      Roles:
-        - Ref: BatchInstanceRole
-  BatchInstanceRole:
+  BatchJobRole:
     Type: AWS::IAM::Role
     Properties:
       AssumeRolePolicyDocument:
@@ -137,12 +97,15 @@ Resources:
           - Effect: 'Allow'
             Principal:
               Service:
-                - 'ec2.amazonaws.com'
+                - 'ecs-tasks.amazonaws.com'
             Action:
               - 'sts:AssumeRole'
+            Condition:
+              ArnLike:
+                aws:SourceArn: "arn:aws:ecs:#{AWS::Region}:#{AWS::AccountId}:*"
+              StringEquals:
+                aws:SourceAccount: "#{AWS::AccountId}"
       Path: '/'
-      ManagedPolicyArns:
-        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
       Policies:
         - PolicyName: 'Cirrus'
           PolicyDocument:
@@ -177,7 +140,13 @@ Resources:
                   - lambda:GetFunction
                 Resource:
                   - arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:#{AWS::StackName}-*
-  BatchServiceRole:
+  BatchInstanceProfile:
+    Type: AWS::IAM::InstanceProfile
+    Properties:
+      Path: '/'
+      Roles:
+        - Ref: BatchInstanceRole
+  BatchInstanceRole:
     Type: AWS::IAM::Role
     Properties:
       AssumeRolePolicyDocument:
@@ -186,12 +155,12 @@ Resources:
           - Effect: 'Allow'
             Principal:
               Service:
-                - 'batch.amazonaws.com'
+                - 'ec2.amazonaws.com'
             Action:
               - 'sts:AssumeRole'
       Path: '/'
       ManagedPolicyArns:
-        - arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole
+        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
   EC2SpotRole:
     Type: AWS::IAM::Role
     Properties:

src/cirrus/builtins/feeders/feed-rerun/definition.yml

@@ -2,6 +2,7 @@ description: Rerun items in the database
 lambda:
   memorySize: 128
   timeout: 900
+  handler: lambda_function.lambda_handler
   iamRoleStatements:
     - Effect: "Allow"
       Action: