aws_autoscaling_traffic_source_attachment add downtime VS target_group_arns

[florianmagnin]

Description

Hello,

We migrate code from:

target_group_arns = [module.alb.outputs.target_groups["front_backend"].arn]

to

create_traffic_source_attachment = true                                                                                                                                                     
traffic_source_identifier        = module.alb.outputs.target_groups["front_backend"].arn                                                                                                    
traffic_source_type              = "elbv2"

But this change add like 2 minutes of downtime (ASG time creation) when we have an ASG re-creation because Terraform:
1 - delete existing aws_autoscaling_traffic_source_attachment
2 - create ASG
3 - create new existing aws_autoscaling_traffic_source_attachment

LOGS (trigger a new creation of ASG):

module.env.module.front_asg.aws_autoscaling_policy.scale_in[0]: Destruction complete after 0s
module.env.module.front_asg.aws_autoscaling_policy.scale_out[0]: Destruction complete after 0s
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 10s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 20s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 30s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb5a8aa, 40s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb5a8aa, 50s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb5a8aa, 1m0s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb5a8aa, 1m10s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Destruction complete after 1m14s
module.env.module.front_asg.module.main.aws_launch_template.this[0]: Modifying... [id=lt-03d0898f38662d5f0]
module.env.module.front_asg.module.main.aws_launch_template.this[0]: Modifications complete after 1s [id=lt-03d0898f38662d5f0]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Creating...
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Still creating... [10s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Still creating... [20s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Creation complete after 26s [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Creating...
module.env.module.front_asg.aws_autoscaling_policy.scale_in[0]: Creating...
module.env.module.front_asg.aws_autoscaling_policy.scale_out[0]: Creating...
module.env.module.front_asg.aws_cloudwatch_metric_alarm.underusage_alarm[0]: Creating...
module.env.module.front_asg.aws_cloudwatch_metric_alarm.overusage_alarm[0]: Creating...
module.env.module.front_asg.aws_cloudwatch_metric_alarm.overusage_alarm[0]: Creation complete after 0s [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC_overusage]
module.env.module.front_asg.aws_cloudwatch_metric_alarm.underusage_alarm[0]: Creation complete after 0s [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC_underusage]
module.env.module.front_asg.aws_autoscaling_policy.scale_out[0]: Creation complete after 1s [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC_scaleout]
module.env.module.front_asg.aws_cloudwatch_metric_alarm.scaleout_alarm[0]: Creating...
module.env.module.front_asg.aws_autoscaling_policy.scale_in[0]: Creation complete after 1s [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC_scalein]
module.env.module.front_asg.aws_cloudwatch_metric_alarm.scalein_alarm[0]: Creating...
module.env.module.front_asg.aws_cloudwatch_metric_alarm.scaleout_alarm[0]: Creation complete after 0s [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC_scaleout]
module.env.module.front_asg.aws_cloudwatch_metric_alarm.scalein_alarm[0]: Creation complete after 0s [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC_scalein]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still creating... [10s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Creation complete after 14s [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC,elbv2,arn:aws:elasticloadbalancing:eu-west-3:654654406548:targetgroup/FRTprd20240304105646607100000007/ab45916d8cb5a8aa]

With "target_group_arns" there is no downtime (checked with $ watch -n 0.2 'curl -ILk -s https://...'). But your comment in the main.tf looks scary

  # TODO - remove at next breaking change. Use `traffic_source_identifier`/`traffic_source_type` instead
  load_balancers = var.load_balancers
  # TODO - remove at next breaking change. Use `traffic_source_identifier`/`traffic_source_type` instead
  target_group_arns         = var.target_group_arns

I tried to fixed it by adding a lifecycle in "aws_autoscaling_traffic_source_attachment":

resource "aws_autoscaling_traffic_source_attachment" "this" {
  count = local.create && var.create_traffic_source_attachment ? 1 : 0

  autoscaling_group_name = var.ignore_desired_capacity_changes ? aws_autoscaling_group.idc[0].id : aws_autoscaling_group.this[0].id

  traffic_source {
    identifier = var.traffic_source_identifier
    type       = var.traffic_source_type
  }
  lifecycle {
    create_before_destroy = true
  }
}

It's better, but traffic is mixed between old and new asg, and during 1minute with some http 200 and http 502. Still shitty.

LOGS with lifecycle:

module.env.module.front_asg.module.main.aws_launch_template.this[0]: Modifying... [id=lt-03d0898f38662d5f0]
module.env.module.front_asg.module.main.aws_launch_template.this[0]: Modifications complete after 1s [id=lt-03d0898f38662d5f0]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Creating...
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Still creating... [10s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Still creating... [20s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Creation complete after 25s [id=front-asg-prd-front-Thu-21-Mar-2024-09-36-43-UTC]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Creating...
module.env.module.front_asg.aws_autoscaling_policy.scale_in[0]: Creating...
module.env.module.front_asg.aws_autoscaling_policy.scale_out[0]: Creating...
module.env.module.front_asg.aws_cloudwatch_metric_alarm.underusage_alarm[0]: Creating...
module.env.module.front_asg.aws_cloudwatch_metric_alarm.overusage_alarm[0]: Creating...
module.env.module.front_asg.aws_cloudwatch_metric_alarm.overusage_alarm[0]: Creation complete after 0s [id=front-asg-prd-front-Thu-21-Mar-2024-09-36-43-UTC_overusage]
module.env.module.front_asg.aws_cloudwatch_metric_alarm.underusage_alarm[0]: Creation complete after 0s [id=front-asg-prd-front-Thu-21-Mar-2024-09-36-43-UTC_underusage]
module.env.module.front_asg.aws_autoscaling_policy.scale_out[0]: Creation complete after 0s [id=front-asg-prd-front-Thu-21-Mar-2024-09-36-43-UTC_scaleout]
module.env.module.front_asg.aws_cloudwatch_metric_alarm.scaleout_alarm[0]: Creating...
module.env.module.front_asg.aws_autoscaling_policy.scale_in[0]: Creation complete after 0s [id=front-asg-prd-front-Thu-21-Mar-2024-09-36-43-UTC_scalein]
module.env.module.front_asg.aws_cloudwatch_metric_alarm.scalein_alarm[0]: Creating...
module.env.module.front_asg.aws_cloudwatch_metric_alarm.scaleout_alarm[0]: Creation complete after 1s [id=front-asg-prd-front-Thu-21-Mar-2024-09-36-43-UTC_scaleout]
module.env.module.front_asg.aws_cloudwatch_metric_alarm.scalein_alarm[0]: Creation complete after 1s [id=front-asg-prd-front-Thu-21-Mar-2024-09-36-43-UTC_scalein]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Creation complete after 2s [id=front-asg-prd-front-Thu-21-Mar-2024-09-36-43-UTC,elbv2,arn:aws:elast
icloadbalancing:eu-west-3:654654406548:targetgroup/FRTprd20240304105646607100000007/ab45916d8cb5a8aa]
module.env.aws_autoscaling_policy.front_asg_phpfpm_workers_scaleout_policy: Creating...
module.env.aws_autoscaling_policy.front_asg_apache_workers_scaleout_policy: Creating...
module.env.aws_cloudwatch_dashboard.cw_scaling: Modifying... [id=Scaling-on-front-asg-prd]
module.env.aws_cloudwatch_metric_alarm.front_asg_phpfpm_workers_metric_alarm_scalein: Modifying... [id=front-asg-prd-phpfpm-workers-metric-alarm-scalein]
module.env.aws_cloudwatch_metric_alarm.front_asg_apache_workers_metric_alarm_scalein: Modifying... [id=front-asg-prd-apache-workers-metric-alarm-scalein]
module.env.aws_cloudwatch_metric_alarm.front_asg_phpfpm_workers_metric_alarm_scalein: Modifications complete after 0s [id=front-asg-prd-phpfpm-workers-metric-alarm-scalein]
module.env.aws_cloudwatch_metric_alarm.front_asg_apache_workers_metric_alarm_scalein: Modifications complete after 0s [id=front-asg-prd-apache-workers-metric-alarm-scalein]
module.env.aws_autoscaling_policy.front_asg_phpfpm_workers_scaleout_policy: Creation complete after 0s [id=front-asg-prd-phpfpm-workers-scaleout-policy]
module.env.aws_autoscaling_policy.front_asg_apache_workers_scaleout_policy: Creation complete after 0s [id=front-asg-prd-apache-workers-scaleout-policy]
module.env.aws_cloudwatch_metric_alarm.front_asg_phpfpm_workers_metric_alarm_scaleout: Modifying... [id=front-asg-prd-phpfpm-workers-metric-alarm-scaleout]
module.env.aws_cloudwatch_metric_alarm.front_asg_apache_workers_metric_alarm_scaleout: Modifying... [id=front-asg-prd-apache-workers-metric-alarm-scaleout]
module.env.aws_cloudwatch_metric_alarm.front_asg_phpfpm_workers_metric_alarm_scaleout: Modifications complete after 0s [id=front-asg-prd-phpfpm-workers-metric-alarm-scaleout]
module.env.aws_cloudwatch_metric_alarm.front_asg_apache_workers_metric_alarm_scaleout: Modifications complete after 1s [id=front-asg-prd-apache-workers-metric-alarm-scaleout]
module.env.aws_cloudwatch_dashboard.cw_scaling: Modifications complete after 1s [id=Scaling-on-front-asg-prd]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0] (deposed object dec1bb0f): Destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC,elbv2,
arn:aws:elasticloadbalancing:eu-west-3:654654406548:targetgroup/FRTprd20240304105646607100000007/ab45916d8cb5a8aa]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 10s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 20s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 30s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 40s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 50s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 1m0s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-...304105646607100000007/ab45916d8cb
5a8aa, 1m10s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_traffic_source_attachment.this[0]: Destruction complete after 1m14s
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0] (deposed object 21af4693): Destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC, 10s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC, 20s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC, 30s elapsed]
module.env.module.front_asg.module.main.aws_autoscaling_group.this[0]: Still destroying... [id=front-asg-prd-front-Thu-21-Mar-2024-09-25-35-UTC, 40s elapsed]

Please DO NOT REMOVE "target_group_arns" with way better than additional attachments.

Thanks,
Regards
Florian

Versions

$ terraform -version
Terraform v1.7.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.41.0

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[florianmagnin]

Bump

[bryantbiggs]

Its not clear what you are looking for with this issue

[florianmagnin]

Hello!

Hmm... It looks like your strategy on some modules is to use extra resources when they are available even if we can configure the parameter in the native resource.

Here the example with "aws_autoscaling_group" and the parameter "target_group_arns" that you quote "to remove" in the module. You prefer to use the extra resource "aws_autoscaling_traffic_source_attachment" for the same result.

But with extra resource to configure the same thing than the main resource you add lifecycle issues and downtime to HTTP traffic. Detailed in the main post.

Regards,
Florian

[vesl]

Hello same issue here

[bryantbiggs]

I would suggest opening an issue upstream with the AWS provider. The aws_autoscaling_traffic_source_attachment is the recommended resource since it support attaching to multiple different services such as VPC lattice, gateway load balancer, etc.

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days