Add IoT fleet provisioning event type

[fredericbarthelet]

Closes: #8296

[codecov-commenter]

Codecov Report

Merging #8324 into master will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #8324      +/-   ##
==========================================
+ Coverage   88.01%   88.05%   +0.03%     
==========================================
  Files         249      250       +1     
  Lines        9307     9335      +28     
==========================================
+ Hits         8192     8220      +28     
  Misses       1115     1115              

Impacted Files	Coverage Δ
lib/plugins/index.js	100.00% <ø> (ø)
lib/plugins/aws/lib/naming.js	98.18% <100.00%> (+0.02%)	⬆️
...ckage/compile/events/iotFleetProvisioning/index.js	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2fb696...220237c. Read the comment docs.

[fredericbarthelet]

@medikoo my PR is ready for review when you have time.

I made a small change from original plan, changing templateBody type from string to object.

Unlike AWS::ApiGateway::Model Schema property which typically accept a JSON, AWS::IoT::ProvisioningTemplate TemplateBody property requires a "JSON formatted contents of the fleet provisioning template" (see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iot-provisioningtemplate.html#cfn-iot-provisioningtemplate-templatebody).

Such JSON formatted string is quite painful to write. It either requires a yaml literal block and the whole template to be dumped as is in the serverless.yml configuration file.

functions:
  provisioningYaml:
    handler: 'handler.hello'
    events:
        - iotFleetProvisioning:
            templateBody: |
            {
              "Parameters": {
                "SerialNumber": {
                  "Type": "String"
                },
                "AWS::IoT::Certificate::Id": {
                  "Type": "String"
                }
              },
              "Resources": {
                "certificate": {
                  "Properties": {
                    "CertificateId": {
                      "Ref": "AWS::IoT::Certificate::Id"
                    },
                    "Status": "Active"
                  },
                  "Type": "AWS::IoT::Certificate"
                },
                "policy": {
                  "Properties": {
                    "PolicyName": "iotPolicy"
                  },
                  "Type": "AWS::IoT::Policy"
                }
              }
            }
            templateName: MyTemplateYaml
            provisioningRoleArn: arn:aws:iam::123456789:role/test-provisioning

or it requires content to be manually stringified :

functions:
  provisioningYaml:
    handler: 'handler.hello'
    events:
        - iotFleetProvisioning:
            templateBody: "{\"Parameters\":{\"SerialNumber\", ...}"
            templateName: MyTemplateYaml
            provisioningRoleArn: arn:aws:iam::123456789:role/test-provisioning

Using an object, and serializing within the event compiler code allows much more fluent configuration :

functions:
  provisioningYaml:
    handler: 'handler.hello'
    events:
        - iotFleetProvisioning:
            templateBody: ${file(template.json)}
            templateName: MyTemplateYaml
            provisioningRoleArn: arn:aws:iam::123456789:role/test-provisioning

WDYT ?

[medikoo]

Thank you @fredericbarthelet That looks very promising! Please see my comments

Using an object, and serializing within the event compiler code allows much more fluent configuration

I totally agree! I believe we should definitely support just objects in such case

[medikoo]

@fredericbarthelet thank you ! That looks great. I have just one cleanup suggestion towards the fixture.

Do you think it's possible to add integration test for it?

[fredericbarthelet]

Do you think it's possible to add integration test for it?

@medikoo for sure, I will add those :)
The current PR version includes a working provisioned service with the IoT Provisioning Template.

In order to test a device provisioning, IoT Core uses a public mqtt endpoint for me to publish to in order to initiate a Thing registration using the previously created provisioning template, and thus triggering the Pre-Provision lambda. In order to publish this message, my idea was to create a dedicated test/utils/iotFleetProvisioning.js service with the desired API to simulate the interaction a Thing will have with IoT Core service. This will require me to install a client mqtt library as part of the serverless project, https://github.com/mqttjs/MQTT.js for example. Is that OK ?

[medikoo]

This will require me to install a client mqtt library as part of the serverless project, https://github.com/mqttjs/MQTT.js for example. Is that OK ?

Yes, we already have some tests which require installing an extra dependencies, but so far it happens in a context of a fixture (for which npm dependencies are automatically installed when testing), e.g. it's how MSK events are tested, see:

• test/integration/infra-dependent/msk.test.js
• test/fixtures/functionMsk

Also for cases, where we need some extra infrastructure which is time taking to setup. We have dedicated infra cloudformation service, which is expected to be setup before tests that depend in infra are being run. Not sure if it's needed for this case, but it's good to be aware of such possibility.

[fredericbarthelet]

@medikoo I'm a bit stuck with this feature integration test, and I would like your feedback on one of the 2 ways I see this coming through. I'm currently facing issues with the IoT sdk and opened the corresponding issue on their repo - aws/aws-iot-device-sdk-js-v2#100, but no response so far. This prevent me from simulating an IoT device within a lambda, making the required API interactions to register and use the new IoT fleet provisioning feature. The code is working fine when not executed within a lambda.

Considering the situation, I could :

• Option 1: Simulate the IoT device directly in the test code base, meaning test/integration/iotFleetProvisioning.test.js, but this require to add dependency to aws-iot-device-sdk-js-v2 within Serverless framework dev dependencies.
• Option 2: Skip integration test for the scope of this PR until the issue is solved on sdk side.

What's your opinion on that ?

[medikoo]

I'm currently facing issues with the IoT sdk and opened the corresponding issue on their repo

Are you bundling aws-iot-device-sdk-v2 or is it available there by default? In latter case, I would confirm on version you're using there, as usually they're locked to specific (and outdated) versions.

Skip integration test for the scope of this PR until the issue is solved on sdk side.

If I see correctly registerDevice logic is more a test setup logic than functionality testing logic (?) In light of that I think it'll be even better to invoke it naturally in tests scripts and not make it part of a serverless service.

However I would still keep it in a fixture. Fixtures engine have baked in support for setup logic. Simply create a _setup.js scripts in fixture folder. It should export a function, which would be invoked as part of fixture setup operation right after dependencies for fixture are installed (and script file will be removed afterwards).
See:

• https://github.com/serverless/test/blob/master/docs/get-fixtures-engine.md#preparing-a-fixture
• https://github.com/serverless/test/blob/7dfc92e1d0d93c91f1c9affc872afa41fcb1c89b/get-fixtures-engine.js#L60-L64

[fredericbarthelet]

Are you bundling aws-iot-device-sdk-v2 or is it available there by default? In latter case, I would confirm on version you're using there, as usually they're locked to specific (and outdated) versions.

I'm bundling, see test/fixtures/iotFleetProvisioning/package.json

If I see correctly registerDevice logic is more a test setup logic than functionality testing logic (?) In light of that I think it'll be even better to invoke it naturally in tests scripts and not make it part of a serverless service.

Here is small schema of a real life scenario I'm trying to reproduce through the integration test to make it clearer :

• 1 : user (through a mobile app for exemple) request to AWS a claim with createProvisioningClaim API. This is done in test/integration/iotFleetProvisioning.test.js
• 2: user forward returned certificates from createProvisioningClaim to IoT device. In my plan, device behavior is occuring within the registerDevice handler shipped in the project. It is similar to the produce handler of MSK integration test. So test/integration/iotFleetProvisioning.test.js invoke registerDevice lambda providing the certificates from 1 as event payload.
• 3, 4 and 5: all interactions made from device (from lambda) with IoT AWS services. This is the purpose of aws-iot-device-sdk-v2 use within the registerDevice lambda. It is not a test setup, it is actually doing what this feature is intended for. The final 5 call is the one triggering the pre-provision lambda. My test actually aims to ensure this lambda has been invoked and a new Thing is listed in IoT Device Management service.

There are no requirement for initial test logic setup, everything is already included within the serverless.yml fixture file.

My question was wether I could move 3, 4 and 5 from registerDevice to iotFleetProvisioning.test.js. If I can, this requires aws-iot-device-sdk-v2 to be listed as dev dependency of Serverless framework. If I can't, I need to wait for resolution on bundle side.

@medikoo , is it more clear ?

[medikoo]

@fredericbarthelet I understand that in real world scenarios, the logic that currently part of registerDevice will usually be part of a service in which we configure iotFleetProvisioning handler.

If that's the case I would keep the things the way you prepared initially.

Concerning issue you reported here aws/aws-iot-device-sdk-js-v2#100. async/await is a core language construct, and I don't think it's possible for it to work differently in lambda context.

From what you describe it appears that there's some network or lambda setup issue. If 'connected' log doesn't appear at all, it means that promise as returned by connection.connect() simply didn't resolve before lambda time out (is 3s a lambda timeout ?)

[fredericbarthelet]

Concerning issue you reported here aws/aws-iot-device-sdk-js-v2#100. async/await is a core language construct, and I don't think it's possible for it to work differently in lambda context.

I agree, however I see a difference which is currently driving me crazy ^^

In test/integration/iotFleetProvisioning.test.js, doing a lambda invocation with

const resp = await awsRequest('Lambda', 'invoke', {
      FunctionName: `${stackName}-registerDevice`,
      InvocationType: 'RequestResponse',
      Payload: JSON.stringify(certificates),
});

or doing a direct code execution with

const { registerDevice } = require('../fixtures/iotFleetProvisioning/index');
const resp = await registerDevice(certificates);

does NOT result in the same output.
So without any changes in the handler, the execution context does change the behavior of registerDevice

@medikoo do you see something obvious I'm missing out ?

I'll continue on working on my test implementing the latter require style to check my development until the end and see what happens.

[codecov-io]

Codecov Report

Merging #8324 (1ffa428) into master (715ba60) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #8324      +/-   ##
==========================================
+ Coverage   87.25%   87.29%   +0.03%     
==========================================
  Files         255      256       +1     
  Lines        9614     9642      +28     
==========================================
+ Hits         8389     8417      +28     
  Misses       1225     1225              

Impacted Files	Coverage Δ
lib/plugins/index.js	100.00% <ø> (ø)
lib/plugins/aws/lib/naming.js	98.19% <100.00%> (+0.02%)	⬆️
...aws/package/compile/events/iotFleetProvisioning.js	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 715ba60...1ffa428. Read the comment docs.

[fredericbarthelet]

Alright, thanks @medikoo for the explanation. I updated the comments as you suggested :)

[medikoo]

Thank you @fredericbarthelet !