[Bug]: svc deploy does not update ECS Service

[rluisr]

Description:

svc deploy with "Load Balanced Web Service" uploads a new container image to ECR but ECS Service is not updated.

related: #3343

Details:

copilot: 1.33.2

svc deploy

svc deploy --app app --name api --env prd --tag prd --force does not update ECS Service.

#11 exporting to image
#11 exporting layers
#11 exporting layers 1.8s done
#11 writing image sha256:done
#11 naming to done
#11 naming to  done
#11 DONE 1.9s
api:latest
api:prd
25l- Updating the infrastructure for stack app-prd-api                       [update in progress]  [29.6s]
  - An autoscaling target to scale your service's desired count               [not started]         
  - A custom resource returning the ECS service's running task count          [update complete]     [4.6s]
  - An ECS service to run and maintain your tasks in the environment cluster  [update in progress]  [18.6s]
    Deployments                                                                                     
               Revision  Rollout      Desired  Running  Failed  Pending                             
      PRIMARY  2         [completed]  5        5        0       0                                   
- Updating the infrastructure for stack app-prd-api                       [update complete]   [51.0s]
  - An autoscaling target to scale your service's desired count               [not started]       
  - A custom resource returning the ECS service's running task count          [update complete]   [4.6s]
  - An ECS service to run and maintain your tasks in the environment cluster  [update complete]   [35.7s]
    Deployments                                                                                   
               Revision  Rollout      Desired  Running  Failed  Pending                           
      PRIMARY  2         [completed]  5        5        0       0                                 
25h✔ Deployed service api.
Recommended follow-up action:
  - Your service is accessible at https://api over the internet.

deploy

but deploy --app app --name api --env prd --tag prd --force works perfectly.

#11 exporting to image
#11 exporting layers
#11 exporting layers 1.8s done
#11 writing image sha256:done
#11 naming to api:latest done
#11 naming to api:prd done
#11 DONE 1.8s
api:latest
api:prd
25l- Updating the infrastructure for stack app-prd-api                       [update complete]   [13.6s]
  - An autoscaling target to scale your service's desired count               [not started]       
  - A custom resource returning the ECS service's running task count          [update complete]   [2.1s]
  - An ECS service to run and maintain your tasks in the environment cluster  [not started]       
25h✔ Deployed service api.
Recommended follow-up action:
  - Your service is accessible at https://api over the internet.

manifest

image:
  build: Dockerfile

cloudformation

Observed result:

svc deploy does not update ECS Service.

Expected result:

svc deploy should update ECS Service.

Debugging:

[iamhopaul123]

Sorry @rluisr. Did you actually mean deploy --force doesn't update your ECS service? Because from the description svc deploy --force worked well.

[rluisr]

Hi @iamhopaul123.

No, just the opposite.
svc deploy --force doesn't update the ECS service. but deploy --force worked well.

I couldn't see any events on ECS Service like increment task definition, deploy, etc. with svc deploy --force

[iamhopaul123]

But from the screenshot you posted and the log, when you did svc deploy --force there was a CFN event for ECS service update, and update info in the progress tracker for your CLI.

[rluisr]

Yes, we can see from the CFn logs and CLI output that the ECS has been updated, but it has done nothing.
svc deploy --force just uploading a new container image to ECR.

[iamhopaul123]

I've been trying to reproduce the issue and this is what i tried.

copilot svc deploy --force

demo git:(main) ✗ copilot svc deploy --force
Found only one service, defaulting to: frontend
Only found one option, defaulting to: test
Login Succeeded
[+] Building 0.6s (7/7) FINISHED                                                                                                                                                               docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                           0.0s
 => => transferring dockerfile: 413B                                                                                                                                                                           0.0s
 => [internal] load metadata for public.ecr.aws/nginx/nginx:latest                                                                                                                                             0.6s
 => [internal] load .dockerignore                                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                                0.0s
 => [internal] load build context                                                                                                                                                                              0.0s
 => => transferring context: 126B                                                                                                                                                                              0.0s
 => [1/2] FROM public.ecr.aws/nginx/nginx:latest@sha256:0d6cee50bcf761ecf2d42024566522d1f299c50a578847a4c45c88477aa637d5                                                                                       0.0s
 => CACHED [2/2] COPY index.html /usr/share/nginx/html                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                         0.0s
 => => exporting layers                                                                                                                                                                                        0.0s
 => => writing image sha256:ef7703770b211f018e444afc5e9663bcf2b7986a9379c81bc2f14df45ebed1e0                                                                                                                   0.0s
 => => naming to 403971813171.dkr.ecr.us-west-2.amazonaws.com/demo/frontend:latest                                                                                                                             0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/vxrqheq1375op0pqkkz8u2rl0

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
The push refers to repository [1234567890.dkr.ecr.us-west-2.amazonaws.com/demo/frontend]
495fc3d2ec25: Layer already exists 
2a4e0f85c473: Layer already exists 
d672eb98862f: Layer already exists 
285cf1fde295: Layer already exists 
25aa6aa4ec97: Layer already exists 
7e028cfbf374: Layer already exists 
39c300f46f6c: Layer already exists 
8560597d922c: Layer already exists 
latest: digest: sha256:1e07389691e8cd8a73df3a312d442200a8b5fc67ab2c0f4d1cd8262bd0acd352 size: 1986
- No new infrastructure changes for stack demo-test-frontend
✔ Forced an update for service frontend from environment test.
✔ Deployed service frontend.
Recommended follow-up action:
  - Your service is accessible at https://v1.copilot.penghaoh.com over the internet.

Seems like it works as expected - building/uploading to ECR and then force update the service.

copilot deploy --app demo --env test --name frontend --tag prod --force

I got the exact same output as above. And I used v1.33.3.

[iamhopaul123]

Also just wanted to clarify: when you do --force we are calling ECS's update service API directly under the hood so it is not expected to see any CFN infra changes. As long as you see new ECS tasks are spinning up and old tasks are draining, it should be working as expected.