SAM vs Serverless: switch to Serverless again?

[mnapoli]

Yes let's discuss switching back again from SAM to the serverless framework (documentation) 😄 (see #99)

I open this issue because things have moved forward with the serverless framework. Mainly: it supports layers and it can run things locally with Docker.

But another big reason is that, after trying again serverless, it solves a lot of small pain points with SAM. For example:

• the region is defined in the config, you never have to deal with it in the CLI
• deploy in only one step (instead of 3 with SAM)

and it is extensible. That opens very interesting doors, for example to simplify a lot some things like setting layer ARNs:

Resources:
    SimpleFunction:
        Type: AWS::Serverless::Function
        Properties:
            FunctionName: 'bref-demo-function'
            Handler: function.php
            Runtime: provided
            Layers:
                - 'arn:aws:lambda:us-east-2:209497400698:layer:php-72:4'

After with serverless:

functions:
    bref-demo-function:
        handler: function.php
        runtime: provided
        layers:
            - ${bref:layer.php-72}

What's great with that as well is that runtime versions could be handled by Bref itself, making upgrading Bref as simple as composer update.

In short serverless could help make things simpler.

BC break

If we switch we would direct tooling efforts towards serverless.

However Bref would still work with SAM.

But since the documentation would be rewritten to target serverless I think it could be considered a BC break (or a major change).

Full comparison SAM vs Serverless

Here is a complete comparison of features:

https://docs.google.com/spreadsheets/d/1DP2uVbg6TLfBa56XJ1GwlSSKQM1sM87-S3LQ2Pl9sPI/edit?usp=sharing

Please have a look and comment (here or in Google Docs) what is missing and what should be added to the list.

As you will see Serverless has a lot of advantages, the only big question left is local development.

WDYT?

[atrope]

Hey @mnapoli,

I will always favor Serverless Framework because it's one less vendor lock we have.
I also think SLS framework is better and more documented than SAM.

I will have a look in local dev with custom layer the next few days and will let you know if I get anywhere. With python I know it works great.

[atrope]

Ok, some updates.

Definitely Possible to run bref locally with SLS.

The test:

the YML:

service: localTest
plugins:
  - serverless-offline
provider:
  name: aws
  runtime: provided
functions:
  hello:
    handler: index.php
    layers:
      - arn:aws:lambda:us-east-1:209497400698:layer:php-72-fpm:6
    events:
      - http: ANY /
      - http: 'ANY {proxy+}'

Install serverless-offline (npm install serverless-offline --save-dev).

In Serverless offline need to edit 2 files,

index.js
Lines 367 to 381: (need to add --docker if it is provided) need to remove the error they put for provided and allow it.

  // if (serviceRuntime === 'provided') {
   //   if (this.options.providedRuntime) {
   //     serviceRuntime = this.options.providedRuntime;
   //   }
   //   else {
   //     throw new Error('Runtime "provided" is unsupported. Please add a --providedRuntime CLI option.');
   //   }
   // }

   if (!( serviceRuntime.startsWith('provided') || serviceRuntime.startsWith('nodejs') || serviceRuntime.startsWith('python') || serviceRuntime.startsWith('ruby'))) {
     this.printBlankLine();
     this.serverlessLog(`Warning: found unsupported runtime '${serviceRuntime}'`);

     return;
   }

functionHelper.js - (need to add --docker if it is provided)

line 16:
const args = funOptions.runtime.startsWith("provided") ? ['invoke', 'local', '-f', funOptions.funName, '--docker'] : ['invoke', 'local', '-f', funOptions.funName];

That's it. on console, execute
sls offline

it will start a Webserver in port 3000. Every call he builds the layer and execute.
Everything is working with a test php file.

Of course we need to Edit some things in serverless-offline and make a PR, but we can Definitely run it.

You can edit in the Google Sheets :)

[deleugpn]

I would ask not to rewrite (a.k.a remove) the SAM documentation. If you decide to change, I understand switching the entire focus of the docs, but keep the bare minimum (layer arn) documented. I would personally stick with CloudFormation since that's where everything else is deployed and the vendor lock-in argument doesn't work at all for my company since we make heavy use of AWS-only services. Having AWS structure declared in the SLS framework means leaked abstraction which means vendor lock in anyway.
The only commitment I would expect from Bref is to at least never stop providing the layer. I can write my own templates without any help needed from the bref tools and my deployments are powered by AWS CodePipeline with CF so I dont need anything more than aws cloudformation package.
I say this as someone that would love to put bref in production but haven't done so for the fear of instability of an early project. Once it's there I would start using Lambda-only features that would lock me in there and if bref doesn't allow me to update my functions it would be a disaster.

I forgot to mention that one strong reason for me to not switch to SLS is because my pipeline has no js dependency at all. All I need is AWS CLI and an authorized Role, which is convenient.

Things that are not a problem if you're using CodePipeline (from the spreadsheet):

Define the region once and for all
Define the CF stack name once and for all
Include configuration variable from file
Deploy multiple stages
Auto-create the S3 bucket

[mnapoli]

@atrope thanks that sounds really cool, I'll have a look at it. And yes if it's a matter of a quick fix in the plugin I'm sure this is something we can sort out over time.

@deleugpn good points.

Just to be clear I still plan on targeting AWS only for now.

But yes since layers work with SAM or Serverless or any other tool, people can still use Bref tooling with whatever they prefer 👍 The switch to serverless would be mostly useful for those starting and those looking for an all-in-one solution with complete documentation.

[mnapoli]

Just dumping the link here: https://www.jeremydaly.com/developing-serverless-applications-locally-with-the-serverless-cloudside-plugin/

This new Serverless Framework plugin allows you to use AWS CloudFormation intrinsic functions (such as !Ref and !GetAtt) to reference cloud resources during local development. When used in your environment variables, these values are replaced with the same identifiers used when deployed to the cloud!

It's compatible with serverless-offline as well.

That's another advantage for serverless here ^^

[deleugpn]

Gotta be extra careful with that one. Imagine pushing local development messages to a Queue and having your deployed worker inside AWS picking those up and trashing your staging / live database.
Pretty cool to see the advancement on tooling around serverless nonetheless.
Personally, I use and recommend localstack. You can provision S3, SQS, SNS, etc in a local container and network it with your application using docker-compose and environment variables. I have a few test code that actually pushes messages to SQS and work them before performing assertions using localstack.

[mnapoli]

Localstack looks really interesting! There are so many things to try out 😅 But it's great!

[akrabat]

FWiW, I'm in favour of serverless.

[Sh1d0w]

I would also ask not to remove CloudFormation docs. This will be a huge breaking change. I personally use BREF in my job and we have a setup our "infrastructure as code". Using AWS with BREF, CloudFormation template can be imported in other stacks, and can import stacks as well.

IMO best case will be to choose what to use during setup time - Serverless or CloudFormation, and then a template will be generated based on your choice. This way changes you plan to introduce will be backwards compatible.

But please don't remove support for CloudFormation as it will break a lot of production apps that currently opted to use Bref and relying on CloudFormation.

[mnapoli]

@Sh1d0w supporting 2 deployment config (both in the code and documentation) would be too much effort at the moment. One of Bref's principle is this:

simplify problems by removing choices instead of trying to address every need

We should support one format perfectly instead of spreading our effort and making it more confusing for users.

But please don't remove support for CloudFormation as it will break a lot of production apps that currently opted to use Bref and relying on CloudFormation.

Rest assured however that anything you wrote will still work. Also you will still be able to use SAM if you want for new projects. For example right now I use serverless for some projects even though Bref recommends SAM. The PHP layer works with both solutions. Nothing should break.

Also I want to reassure you that serverless configuration supports CloudFormation syntax. There are a few specific bits that a different but anything you do with SAM can be done with serverless. But if you don't want to change that's fine, it will still work.

[madhavtummala]

By reading the documentation, I felt like SAM is not completely local. There is some communication taking place with AWS Cloud although the function itself runs locally.

[esurov]

I support switching to serverless. And I agree that we can't maintain both. Serverless it is!

[mnapoli]

For those interested there is a WIP in this branch: https://github.com/mnapoli/bref/tree/serverless

[esurov]

For those interested there is a WIP in this branch: https://github.com/mnapoli/bref/tree/serverless

Wow! @mnapoli ! Thank you for the heads up! Can't wait for it to be released. Currently I need to deploy my lambda to multiple regions and doing with sam is a pain.

[mnapoli]

Nice! You can start using Serverless right now, I've been using it for some projects. It's already compatible.

[esurov]

@mnapoli You made my day! That's really great news! Yay!

[esurov]

@mnapoli A question on serverless plugin usage. Is it supposed I need to copy bref.js to my project .serverless_plugins/ ?
What about ${bref:layer.php-73} variable in serverless.yaml? How is it resolved? Do I need to include bref.yaml into my serverless.yaml?
Thank you!

[mnapoli]

That is experimental, you can ignore all this and keep using the layer ARN explicitly.

[esurov]

But how can I do it if I need to deploy it to multiple regions? I guess I want to run:

serverless deploy --stage production --region us-east-1
serverless deploy --stage production --region us-west-1
serverless deploy --stage production --region eu-central-1
serverless deploy --stage production --region ap-southeast-1

but if I have the layer ARN specified explicitly it will not work. I was thinking maybe I could modify the plugin to make some sort of switch there and specify ARN for each region?

[mnapoli]

You can use region variables (see https://serverless.com/framework/docs/providers/aws/guide/variables/) but I'm sorry I can't document everything in this issue.

My first priority is to keep the same set of features. Right now you can't deploy easily to multiple regions with SAM and Bref.

[esurov]

Yeah, it works like this:

        layers:
            - 'arn:aws:lambda:${opt:region}:209497400698:layer:php-73-fpm:6'

awesome! thank you @mnapoli for all your work on bref! can I help you with anything to make it better?

[mnapoli]

I confirm that @atrope's PR makes it possible to run API Gateway + Lambda locally with Serverless (it works through Docker): dherault/serverless-offline#648

This isn't as fast as it could be (a bit like SAM, the lambda starts from scratch every time) but it can be optimized over time. We need dherault/serverless-offline#648 to be merged (or the worst case would be to fork the plugin to apply the change, but that's way more work).

[atrope]

Hi @mnapoli as i said, i think some work could be done in the future so the docker container would be reutilized instead of initializing from scratch all the time.

I updated the code from master in the PR and fixed what needed to be fixed now we must wait and see if they will approve it.