Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] wash app deploy succeeds but does not deploy neither on Fedora nor on Ubuntu #1727

Open
1 of 9 tasks
LostInBrittany opened this issue Mar 20, 2024 · 20 comments
Open
1 of 9 tasks

[BUG] wash app deploy succeeds but does not deploy neither on Fedora nor on Ubuntu #1727

LostInBrittany opened this issue Mar 20, 2024 · 20 comments
Labels
bug Something isn't working

Comments

@LostInBrittany
Copy link
Contributor

Affected project(s)

  • documentation
  • examples
  • wasmCloud host
  • wasmCloud CLI (wash)
  • wasmCloud dashboard UI (washboard)
  • capability providers
  • provider bindgen
  • control interface client
  • other / not sure

Describe the bug

According to my tests, wash app deploy completes but does not properly deploy apps on Fedora Linux

For example, with the Tinygo Hello World example:

$ wash new component hello --template-name hello-world-tinygo
🔧   Cloning template from repo wasmCloud/wasmCloud subfolder examples/golang/components/http-hello-world...
🔧   Using template subfolder examples/golang/components/http-hello-world...
🔧   Generating template...
[ 1/63]   Done: .gitignore                                                                                                             [ 2/63]   Done: README.md                                                                                                              [ 3/63]   Done: go.mod                                                                                                                 [ 4/63]   Done: hello.go                                                                                                               [ 5/63]   Skipped: project-generate.toml                                                                                               [ 6/63]   Done: wadm.yaml                                                                                                              [ 7/63]   Done: wasmcloud.toml                                                                                                         [ 8/63]   Done: wit/deps/cli/command.wit                                                                                               [ 9/63]   Done: wit/deps/cli/environment.wit                                                                                           [10/63]   Done: wit/deps/cli/exit.wit                                                                                                  [11/63]   Done: wit/deps/cli/imports.wit                                                                                               [12/63]   Done: wit/deps/cli/run.wit                                                                                                   [13/63]   Done: wit/deps/cli/stdio.wit                                                                                                 [14/63]   Done: wit/deps/cli/terminal.wit                                                                                              [16/63]   Done: wit/deps/clocks/monotonic-clock.wit                                                                                    [17/63]   Done: wit/deps/clocks/wall-clock.wit                                                                                         [18/63]   Done: wit/deps/clocks/world.wit                                                                                              [20/63]   Done: wit/deps/filesystem/preopens.wit                                                                                       [21/63]   Done: wit/deps/filesystem/types.wit                                                                                          ✨   Done! New project created /home/horacio/git/wasmcloud/hello

Project generated and is located at: /home/horacio/git/wasmcloud/hello

$ wash build
Actor built and signed and can be found at "/home/horacio/git/wasmcloud/hello/build/http_hello_world_s.wasm"

$ wash app deploy wadm.yaml 
Successfully deployed model tinygo-http-hello-world v0.0.1

$ wash app list                                                                                                                                                                                                        
  Name                      Latest Version   Deployed Version   Deploy Status   Description                                                                                                             
  tinygo-http-hello-world   v0.0.1           v0.0.1                Undeployed   HTTP hello world demo in Golang (TinyGo), using the WebAssembly Component Model and WebAssembly Interfaces Types (WIT)

Steps to reproduce

  1. wash app deploy your-app.yaml
  2. wash app list
  3. Notice that the app list says that your app is undeployed

Expected behavior

The application should have been successfully deployed

Environment

  • OS: Fedora Linux
  • Shell: bash
  • wasmcloud Version: latest

Screenshots / Logs / Additional context

No response
@LostInBrittany LostInBrittany added the bug Something isn't working label Mar 20, 2024
@LostInBrittany
Copy link
Contributor Author

I have put wash logs to debug, no error message.
I have rolled back to Wash 0.25, and it deploys as usual.

Any idea of how to get more info on where is the problem?

@LostInBrittany
Copy link
Contributor Author

OK, I have just tested on my Mac M1. I had wasmcloud installed (wash version 0.26, wasmcloud version 0.82, nats-server version 2.9.14) and everything worked.
I have deleted my .wash folder, and doing a wash up has downloaded me a new version (wash version 0.26, wasmcloud version 0.82, nats-server version 2.10.7), and now I have the same problem than in my linux, when I deploy an app it doesn't deploy, it ends in a Reconciling state:

$ wash app list
  Name                          Latest Version   Deployed Version   Deploy Status   Description
  typescript-http-hello-world   v0.0.1           v0.0.1               Reconciling   Demo of Typescript HTTP hello world server

The only difference I see is the nats-server version.

I can't believe I am th eonly one with this problem 😔

@thomastaylor312
Copy link
Contributor

Sorry for the delay here @LostInBrittany. A lot of the maintainers were out at KubeCon this past week.

A couple of notes:

  • We definitely run tests on main of wadm against NATS server 2.10.7, but we've also seen patch releases of NATS fix this before
  • Moving NATS versions on your Mac might have possibly been the issue with being stuck in reconciling, but I'd have to try it myself to see. Sometimes Jetstream state can be a little finicky
  • Uploading your wadm logs would probably help out here. They should be located at ~/.wash/downloads/wadm.log
  • If you'd like a work around, you should be able to stop NATS and remove the jetstream directory. If you look at the logs from NATS (~/.wash/downloads/nats.log) you should find a "Store Directory" that you can remove and then restart to get things working again

Once we have a look at the logs, if that doesn't tell us anything, then I can start diving in to debug some more.

@LostInBrittany
Copy link
Contributor Author

Hi! I have started again, deleting my .wash folder and the jetstream folder (thanks for the tip, I wasn't sure where to find it!).

I've restarted the process from the beginning, and now it gets stuck on undeployed:

$ wash build
Actor built and signed and can be found at "/home/horacio/git/wasmcloud/hello-typescript/build/http_hello_world_s.wasm"

$ wash app deploy wadm.yaml
Successfully deployed model typescript-http-hello-world v0.0.1

$ wash app list   
  Name                          Latest Version   Deployed Version   Deploy Status   Description                                 
  typescript-http-hello-world   v0.0.1           v0.0.1                Undeployed   Demo of Typescript HTTP hello world server

Here you have the logs:

wadm.log

$ cat ~/.wash/downloads/wadm.log 
RUST_LOG was not set or the given directive was invalid: FromEnvError { kind: Env(NotPresent) }
Defaulting logger to `info` level
2024-04-02T08:29:38.730761Z  INFO async_nats::options: event: connected
2024-04-02T08:29:38.736846Z  INFO new: wadm::server: Creating API subscriber topic=wadm.api.>

nats.log

$ cat ~/.wash/downloads/nats.log 
[117406] 2024/04/02 10:29:30.619666 [INF] Starting nats-server
[117406] 2024/04/02 10:29:30.619793 [INF]   Version:  2.10.7
[117406] 2024/04/02 10:29:30.619804 [INF]   Git:      [fa8464d]
[117406] 2024/04/02 10:29:30.619807 [INF]   Name:     NDMTRKQKL4746RJMD6ZEB5ZJKHNXMMVIV2IKMRNNMKFEUZ246LXW3WE5
[117406] 2024/04/02 10:29:30.619818 [INF]   Node:     NVoUT4Pu
[117406] 2024/04/02 10:29:30.619821 [INF]   ID:       NDMTRKQKL4746RJMD6ZEB5ZJKHNXMMVIV2IKMRNNMKFEUZ246LXW3WE5
[117406] 2024/04/02 10:29:30.619836 [INF] Using configuration file: /home/horacio/.wash/downloads/nats.conf
[117406] 2024/04/02 10:29:30.620232 [INF] Starting JetStream
[117406] 2024/04/02 10:29:30.620366 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[117406] 2024/04/02 10:29:30.620374 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[117406] 2024/04/02 10:29:30.620377 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[117406] 2024/04/02 10:29:30.620379 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[117406] 2024/04/02 10:29:30.620381 [INF] 
[117406] 2024/04/02 10:29:30.620383 [INF]          https://docs.nats.io/jetstream
[117406] 2024/04/02 10:29:30.620385 [INF] 
[117406] 2024/04/02 10:29:30.620390 [INF] ---------------- JETSTREAM ----------------
[117406] 2024/04/02 10:29:30.620394 [INF]   Max Memory:      46.90 GB
[117406] 2024/04/02 10:29:30.620398 [INF]   Max Storage:     23.32 GB
[117406] 2024/04/02 10:29:30.620400 [INF]   Store Directory: "/tmp/wash-jetstream-4222/jetstream"
[117406] 2024/04/02 10:29:30.620402 [INF]   Domain:          core
[117406] 2024/04/02 10:29:30.620404 [INF] -------------------------------------------
[117406] 2024/04/02 10:29:30.620903 [INF] Listening for websocket clients on ws://0.0.0.0:4001
[117406] 2024/04/02 10:29:30.620913 [WRN] Websocket not configured with TLS. DO NOT USE IN PRODUCTION!
[117406] 2024/04/02 10:29:30.621164 [INF] Listening for client connections on 127.0.0.1:4222
[117406] 2024/04/02 10:29:30.621173 [INF] Server is ready

Any tips on how to dig deeper?

@LostInBrittany
Copy link
Contributor Author

LostInBrittany commented Apr 2, 2024 •
edited

Same thing with other apps, like the TinyGo example:

$ wash app get typescript-http-hello-world   v0.0.1
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: typescript-http-hello-world
  annotations:
    description: Demo of Typescript HTTP hello world server
    experimental: 'true'
    version: v0.0.1
spec:
  components:
  - name: typescript-http-hello-world
    type: actor
    properties:
      image: file:///home/horacio/git/wasmcloud/hello-typescript/build/http_hello_world_s.wasm
    traits:
    - type: spreadscaler
      properties:
        instances: 1
        spread: []
    - type: linkdef
      properties:
        target: httpserver
        values:
          ADDRESS: 0.0.0.0:8080
  - name: httpserver
    type: capability
    properties:
      image: wasmcloud.azurecr.io/httpserver:0.19.1
      contract: wasmcloud:httpserver

$ wash app history typescript-http-hello-world                         
  Version   Deployed?  
  v0.0.1    true       

$ wash app list                                                                                                                                                                                                            
  Name                          Latest Version   Deployed Version   Deploy Status   Description                                                                                                             
  tinygo-http-hello-world       v0.0.1           v0.0.1                Undeployed   HTTP hello world demo in Golang (TinyGo), using the WebAssembly Component Model and WebAssembly Interfaces Types (WIT)  
  typescript-http-hello-world   v0.0.1           v0.0.1                Undeployed   Demo of Typescript HTTP hello world server

@thomastaylor312
Copy link
Contributor

Hmmm...this is really odd. I don't think we allow increasing the log level for wadm from wash up so that would be the next step. If you really want to try, you can download the latest wadm release 0.10 and then do wash up --disable-wadm and then in another terminal run RUST_LOG=wadm=debug ./wadm with the wadm release you downloaded.

Otherwise, I'll try to take a look at this locally and see if I can reproduce the issue. I remember seeing something like this when we first released wadm, but haven't seen it in a while

@LostInBrittany
Copy link
Contributor Author

LostInBrittany commented Apr 3, 2024 •
edited

Thanks for the suggestion!

Just tried it. App still remains in Undeployed state:

$ wash app deploy wadm.yaml 
Successfully deployed model tinygo-http-hello-world v0.0.1

Congratulations on installing wash!  Shell auto-complete is available. For instructions on setting up auto-complete for your shell, please see 'https://github.com/wasmCloud/wasmCloud/blob/main/crates/wash-cli/Completions.md'

$ wash app list                                                                                                                                                                                                        
  Name                      Latest Version   Deployed Version   Deploy Status   Description                                                                                                             
  tinygo-http-hello-world   v0.0.1           v0.0.1                Undeployed   HTTP hello world demo in Golang (TinyGo), using the WebAssembly Component Model and WebAssembly Interfaces Types (WIT)

Here you have wadm logs:

$ RUST_LOG=wadm=debug ./bin/wadm
2024-04-03T14:30:40.312197Z  INFO new: wadm::server: Creating API subscriber topic=wadm.api.>
2024-04-03T14:31:18.550975Z DEBUG serve:put_model{account_id=None lattice_id="default"}:get{account_id=None lattice_id="default" model_name=tinygo-http-hello-world}: wadm::server::storage: Fetching model from storage key=default-tinygo-http-hello-world
2024-04-03T14:31:18.559080Z DEBUG serve:put_model{account_id=None lattice_id="default"}:set{account_id=None lattice_id="default" current_revision=Some(0) model_name=tinygo-http-hello-world}: wadm::server::storage: Storing model in storage
2024-04-03T14:31:18.561438Z DEBUG serve:deploy_model{account_id=None lattice_id="default" name="tinygo-http-hello-world"}:get{account_id=None lattice_id="default" model_name=tinygo-http-hello-world}: wadm::server::storage: Fetching model from storage key=default-tinygo-http-hello-world
2024-04-03T14:31:18.561911Z DEBUG serve:deploy_model{account_id=None lattice_id="default" name="tinygo-http-hello-world"}:list{account_id=None lattice_id="default"}: wadm::server::storage: Fetching list of models from storage
2024-04-03T14:31:18.562352Z DEBUG serve:deploy_model{account_id=None lattice_id="default" name="tinygo-http-hello-world"}:list{account_id=None lattice_id="default"}:get{account_id=None lattice_id="default" model_name=tinygo-http-hello-world}: wadm::server::storage: Fetching model from storage key=default-tinygo-http-hello-world
2024-04-03T14:31:18.562661Z DEBUG serve:deploy_model{account_id=None lattice_id="default" name="tinygo-http-hello-world"}:set{account_id=None lattice_id="default" current_revision=Some(1) model_name=tinygo-http-hello-world}: wadm::server::storage: Storing model in storage
2024-04-03T14:31:18.563166Z DEBUG serve:deploy_model{account_id=None lattice_id="default" name="tinygo-http-hello-world"}:set{account_id=None lattice_id="default" current_revision=Some(1) model_name=tinygo-http-hello-world}:retry_model_update{account_id=None lattice_id="default" operation=Add("tinygo-http-hello-world")}: wadm::server::storage: Model was already in list, returning early
2024-04-03T14:31:34.764514Z DEBUG serve:list_models{account_id=None lattice_id="default"}:list{account_id=None lattice_id="default"}: wadm::server::storage: Fetching list of models from storage
2024-04-03T14:31:34.765219Z DEBUG serve:list_models{account_id=None lattice_id="default"}:list{account_id=None lattice_id="default"}:get{account_id=None lattice_id="default" model_name=tinygo-http-hello-world}: wadm::server::storage: Fetching model from storage key=default-tinygo-http-hello-world

I also tried with an older wadm version, 0.9.1, same result.

@LostInBrittany
Copy link
Contributor Author

LostInBrittany commented Apr 3, 2024 •
edited

I've just discoverer that wash drain all allows to delete a cache...
Tried again, with the cache deleted, and got a Reconciling status. And the issue has looped...

$ wash drain all
Successfully cleared caches at: ["/tmp/wasmcloudcache", "/tmp/wasmcloud_ocicache", "/home/horacio/.wash/downloads"]

$ wash app deploy wadm.yaml 
Successfully deployed model tinygo-http-hello-world v0.0.1

$ wash app list                                                                                                                                                                                                        
  Name                      Latest Version   Deployed Version   Deploy Status   Description                                                                                                             
  tinygo-http-hello-world   v0.0.1           v0.0.1               Reconciling   HTTP hello world demo in Golang (TinyGo), using the WebAssembly Component Model and WebAssembly Interfaces Types (WIT)  

$ cat ~/.wash/downloads/nats.log 
[235708] 2024/04/03 17:34:23.339534 [INF] Starting nats-server
[235708] 2024/04/03 17:34:23.339616 [INF]   Version:  2.10.7
[235708] 2024/04/03 17:34:23.339622 [INF]   Git:      [fa8464d]
[235708] 2024/04/03 17:34:23.339627 [INF]   Name:     NAYERVFHQPDPORBNF75UVFDDDGR3KKRSDPRWQQYSQS57R6DAX4UZ6TQI
[235708] 2024/04/03 17:34:23.339631 [INF]   Node:     0tz3OiY1
[235708] 2024/04/03 17:34:23.339633 [INF]   ID:       NAYERVFHQPDPORBNF75UVFDDDGR3KKRSDPRWQQYSQS57R6DAX4UZ6TQI
[235708] 2024/04/03 17:34:23.339646 [INF] Using configuration file: /home/horacio/.wash/downloads/nats.conf
[235708] 2024/04/03 17:34:23.339930 [INF] Starting JetStream
[235708] 2024/04/03 17:34:23.340097 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[235708] 2024/04/03 17:34:23.340104 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[235708] 2024/04/03 17:34:23.340106 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[235708] 2024/04/03 17:34:23.340108 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[235708] 2024/04/03 17:34:23.340110 [INF] 
[235708] 2024/04/03 17:34:23.340112 [INF]          https://docs.nats.io/jetstream
[235708] 2024/04/03 17:34:23.340113 [INF] 
[235708] 2024/04/03 17:34:23.340115 [INF] ---------------- JETSTREAM ----------------
[235708] 2024/04/03 17:34:23.340118 [INF]   Max Memory:      46.90 GB
[235708] 2024/04/03 17:34:23.340121 [INF]   Max Storage:     23.32 GB
[235708] 2024/04/03 17:34:23.340123 [INF]   Store Directory: "/tmp/wash-jetstream-4222/jetstream"
[235708] 2024/04/03 17:34:23.340125 [INF]   Domain:          core
[235708] 2024/04/03 17:34:23.340127 [INF] -------------------------------------------
[235708] 2024/04/03 17:34:23.340504 [INF]   Starting restore for stream '$G > KV_CONFIGDATA_default'
[235708] 2024/04/03 17:34:23.340699 [INF]   Restored 0 messages for stream '$G > KV_CONFIGDATA_default' in 0s
[235708] 2024/04/03 17:34:23.340755 [INF]   Starting restore for stream '$G > KV_LATTICEDATA_default'
[235708] 2024/04/03 17:34:23.340870 [INF]   Restored 2 messages for stream '$G > KV_LATTICEDATA_default' in 0s
[235708] 2024/04/03 17:34:23.340916 [INF]   Starting restore for stream '$G > KV_wadm_manifests'
[235708] 2024/04/03 17:34:23.341049 [INF]   Restored 2 messages for stream '$G > KV_wadm_manifests' in 0s
[235708] 2024/04/03 17:34:23.341092 [INF]   Starting restore for stream '$G > KV_wadm_state'
[235708] 2024/04/03 17:34:23.341202 [INF]   Restored 3 messages for stream '$G > KV_wadm_state' in 0s
[235708] 2024/04/03 17:34:23.341251 [INF]   Starting restore for stream '$G > wadm_commands'
[235708] 2024/04/03 17:34:23.341393 [INF]   Restored 0 messages for stream '$G > wadm_commands' in 0s
[235708] 2024/04/03 17:34:23.341438 [INF]   Starting restore for stream '$G > wadm_events'
[235708] 2024/04/03 17:34:23.341554 [INF]   Restored 0 messages for stream '$G > wadm_events' in 0s
[235708] 2024/04/03 17:34:23.341606 [INF]   Starting restore for stream '$G > wadm_mirror'
[235708] 2024/04/03 17:34:23.341738 [INF]   Restored 1 messages for stream '$G > wadm_mirror' in 0s
[235708] 2024/04/03 17:34:23.341778 [INF]   Starting restore for stream '$G > wadm_notify'
[235708] 2024/04/03 17:34:23.341876 [INF]   Restored 0 messages for stream '$G > wadm_notify' in 0s
[235708] 2024/04/03 17:34:23.341926 [INF]   Starting restore for stream '$G > wadm_status'
[235708] 2024/04/03 17:34:23.342042 [INF]   Restored 5 messages for stream '$G > wadm_status' in 0s
[235708] 2024/04/03 17:34:23.342079 [INF]   Recovering 1 consumers for stream - '$G > wadm_commands'
[235708] 2024/04/03 17:34:23.342328 [INF]   Recovering 1 consumers for stream - '$G > wadm_events'
[235708] 2024/04/03 17:34:23.342475 [INF]   Recovering 1 consumers for stream - '$G > wadm_mirror'
[235708] 2024/04/03 17:34:23.342583 [INF]   Recovering 2 consumers for stream - '$G > wadm_notify'
[235708] 2024/04/03 17:34:23.343129 [INF] Listening for websocket clients on ws://0.0.0.0:4001
[235708] 2024/04/03 17:34:23.343137 [WRN] Websocket not configured with TLS. DO NOT USE IN PRODUCTION!
[235708] 2024/04/03 17:34:23.343510 [INF] Listening for client connections on 127.0.0.1:4222
[235708] 2024/04/03 17:34:23.343527 [INF] Server is ready

I feel simply lost now...

@ricochet
Copy link
Contributor

ricochet commented Apr 3, 2024

Since this is fedora specific, I'd like to also double-check that the http server is starting without issue. The way I usually debug those issues is by looking at the host logs ~/.wash/downloads/wasmcloud.log.

We can also simplify the test case to verify that the error in reporting state (still a wadm bug but isn't masking another underlying issue) by using the imperative commands in wash (which don't use wadm):

# make sure you don't have any apps deployed
# so that wadm isn't trying to reconcile or do anything underneath our test
wash app typescript-http-hello-world delete v0.0.1

wash start provider wasmcloud.azurecr.io/httpserver:0.19.1
# if you've recently cleared your OCI cache, this will need a bit to pull the image

wash get inventory

# check the host log for errors
less ~/.wash/downloads/wasmcloud.log

# if the provider starts without errors and everything looks ok,
# then it's probably not the provider but continuing on with a complete imperative test

wash start actor `file:///home/horacio/git/wasmcloud/hello-typescript/build/http_hello_world_s.wasm`

wash link put <actor-id> <provider-id> <contract-id>

@LostInBrittany
Copy link
Contributor Author

Thanks for the suggestion!

Here you have the result, it seems we advance...

$ wash start provider wasmcloud.azurecr.io/httpserver:0.19.1

Provider [VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M] (ref: [wasmcloud.azurecr.io/httpserver:0.19.1]) started on host [NB3HEIF4UVKFGTZN7ZLMI7RKENGBKOCXNAQEIVNTORZWCEMX3PDM3STI]

$ cat ~/.wash/downloads/wasmcloud.log
2024-04-04T19:52:54.103676Z DEBUG new: wasmcloud_host::wasmbus: adding cluster key to cluster issuers cluster_pub_key="CDSFWMJTQVQ5PPA7VGHY5FJPK6U45KEF2F3ASOI2DLMXT3JNYCXN2V3A"
2024-04-04T19:52:54.104115Z DEBUG new: wasmcloud_host::wasmbus: connecting to NATS control server ctl_nats_url="nats://127.0.0.1:4222"
2024-04-04T19:52:54.104225Z DEBUG new: wasmcloud_host::wasmbus: connecting to NATS RPC server rpc_nats_url="nats://127.0.0.1:4222"
2024-04-04T19:52:54.106386Z  INFO async_nats::options: event: connected
2024-04-04T19:52:54.106846Z  INFO async_nats::options: event: connected
2024-04-04T19:52:54.107946Z  INFO new:create_bucket: wasmcloud_host::wasmbus: bucket already exists. Skipping creation. bucket=LATTICEDATA_default
2024-04-04T19:52:54.108561Z  INFO new:create_bucket: wasmcloud_host::wasmbus: bucket already exists. Skipping creation. bucket=CONFIGDATA_default
2024-04-04T19:52:54.110778Z  INFO new: wasmcloud_host::wasmbus: wasmCloud host started host_id="NB3HEIF4UVKFGTZN7ZLMI7RKENGBKOCXNAQEIVNTORZWCEMX3PDM3STI"
2024-04-04T19:53:08.567691Z  INFO handle_auction_provider: wasmcloud_host::wasmbus: handling auction for provider provider_ref="wasmcloud.azurecr.io/httpserver:0.19.1" link_name="default" constraints={}
2024-04-04T19:53:10.569892Z  INFO handle_launch_provider: wasmcloud_host::wasmbus: handling launch provider provider_ref="wasmcloud.azurecr.io/httpserver:0.19.1" link_name="default"
2024-04-04T19:53:11.754903Z  INFO handle_launch_provider_task: wasmcloud_host::wasmbus: provider started provider_ref="wasmcloud.azurecr.io/httpserver:0.19.1" link_name="default"
2024-04-04T19:53:11.754937Z DEBUG process_claims_put: wasmcloud_host::wasmbus: process claim entry put pubkey="VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M"
Log level was not set by host or environment variable.
Defaulting logger to `info` level
Starting capability provider VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M instance 018eaaac-234a-799c-b2b0-5262904069c7 with nats url nats://127.0.0.1:4222
2024-04-04T19:53:11.759860Z  INFO wasmbus_rpc::rpc_client: nats client connected

$ wash get inventory
                                  Host Inventory (NB3HEIF4UVKFGTZN7ZLMI7RKENGBKOCXNAQEIVNTORZWCEMX3PDM3STI)                                 
                                                                                                                                            
  hostcore.os                                                                    linux                                                      
  hostcore.arch                                                                  x86_64                                                     
  hostcore.osfamily                                                              unix                                                       
                                                                                                                                            
  No actors found                                                                                                                           
                                                                                                                                            
  Provider ID                                                 Name               Link Name          Image Reference                         
  VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M    HTTP Server        default            wasmcloud.azurecr.io/httpserver:0.19.1  

$ wash start actor file:///home/horacio/git/wasmcloud/hello-tinygo/build/http_hello_world_s.wasm 

Actor [MBOA7KKDM5G5G77YIMJEDWD4QOQVKTXL2VIGLHOOOB5ALARKE6UWOGHN] (ref: [file:///home/horacio/git/wasmcloud/hello-tinygo/build/http_hello_world_s.wasm]) started on host [NB3HEIF4UVKFGTZN7ZLMI7RKENGBKOCXNAQEIVNTORZWCEMX3PDM3STI]


$ wash link put MBOA7KKDM5G5G77YIMJEDWD4QOQVKTXL2VIGLHOOOB5ALARKE6UWOGHN VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M contract

Published link (MBOA7KKDM5G5G77YIMJEDWD4QOQVKTXL2VIGLHOOOB5ALARKE6UWOGHN) <-> (VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M) successfully

$ curl localhost:8000 -v
* processing: localhost:8000
*   Trying [::1]:8000...
* connect to ::1 port 8000 failed: Connection refused
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.2.1
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< content-length: 0
< date: Thu, 04 Apr 2024 20:11:02 GMT
< 

$ cat ~/.wash/downloads/wasmcloud.log
[...]
2024-04-04T20:10:55.187966Z ERROR handle_rpc_message: wasmcloud_host::wasmbus: failed to handle request origin=WasmCloudEntity { public_key: "VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M", link_name: "default", contract_id: "contract" } target=WasmCloudEntity { public_key: "MBOA7KKDM5G5G77YIMJEDWD4QOQVKTXL2VIGLHOOOB5ALARKE6UWOGHN", link_name: "", contract_id: "" } operation="HttpServer.HandleRequest" invocation_id="df2f075ae2724b6a8822239ffb26ae40" e=failed to handle invocation

Caused by:
    0: failed to call actor
    1: failed to call component
    2: failed to instantiate instance (no bindings satisfied exports):
       
       `wasmcloud:bus/guest` error: exported instance `wasmcloud:bus/guest` not present
       
       `wasi:command/command` error: exported instance `wasi:cli/run@0.2.0` not present
       
2024-04-04T20:10:55.188500Z ERROR wasmbus_rpc::rpc_client: rpc error response error=failed to handle invocation
2024-04-04T20:10:55.188546Z  WARN httpserver: actor responded with error error=rpc: failed to handle invocation
2024-04-04T20:10:55.188564Z ERROR wasmcloud_provider_httpserver: Error sending HttpRequest to actor error=rpc: failed to handle invocation
2024-04-04T20:10:55.188584Z ERROR warp::filters::trace: unable to process request (internal error) status=500 error=None
2024-04-04T20:11:02.359969Z  INFO warp::filters::trace: processing request
2024-04-04T20:11:02.361754Z ERROR handle_rpc_message: wasmcloud_host::wasmbus: failed to handle request origin=WasmCloudEntity { public_key: "VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M", link_name: "default", contract_id: "contract" } target=WasmCloudEntity { public_key: "MBOA7KKDM5G5G77YIMJEDWD4QOQVKTXL2VIGLHOOOB5ALARKE6UWOGHN", link_name: "", contract_id: "" } operation="HttpServer.HandleRequest" invocation_id="a3fa5917bf9f4a55943a768a9af324a5" e=failed to handle invocation

Caused by:
    0: failed to call actor
    1: failed to call component
    2: failed to instantiate instance (no bindings satisfied exports):
       
       `wasmcloud:bus/guest` error: exported instance `wasmcloud:bus/guest` not present
       
       `wasi:command/command` error: exported instance `wasi:cli/run@0.2.0` not present
       
2024-04-04T20:11:02.362273Z ERROR wasmbus_rpc::rpc_client: rpc error response error=failed to handle invocation
2024-04-04T20:11:02.362324Z  WARN httpserver: actor responded with error error=rpc: failed to handle invocation
2024-04-04T20:11:02.362342Z ERROR wasmcloud_provider_httpserver: Error sending HttpRequest to actor error=rpc: failed to handle invocation
2024-04-04T20:11:02.362359Z ERROR warp::filters::trace: unable to process request (internal error) status=500 error=None

@LostInBrittany
Copy link
Contributor Author

LostInBrittany commented Apr 4, 2024 •
edited

Important update

OK, I have done tests in both Fedora 39 and Ubuntu 23.10 linux. Both fresh virtual machines, created specifically for the tests.

Installation

To install wash I had to use the Rust installation method in both machines, as the .rpm and .deb for Fedora 39 and Ubuntu 23.10 aren't pushed to Packagecloud. After #1719 and #1718 GitHubActions can generate the binaries for latest versions of both Fedora and Ubuntu, but it seems that those versions aren't pushed to Packagecloud.

After installing I have wash 0.26 in both machines

Trying to initiate, build and deploy a Hello World

I choose the TinyGo version:

$ wash new actor hello --git wasmcloud/wasmcloud --subfolder examples/golang/actors/http-hello-world --branch 0.82-examples
🔧   Cloning template from repo wasmcloud/wasmcloud subfolder examples/golang/actors/http-hello-world...
🔧   Using template subfolder examples/golang/actors/http-hello-world...
🔧   Generating template...
[ 1/63]   Done: .gitignore                                                                                                             [ 2/63]   Done: README.md                                                                                                              [ 3/63]   Done: go.mod                                                                                                                 [ 4/63]   Done: hello.go                                                                                                               [ 5/63]   Skipped: project-generate.toml                                                                                               [ 6/63]   Done: wadm.yaml                                                                                                              [ 7/63]   Done: wasmcloud.toml                                                                                                         [ 8/63]   Done: wit/deps/cli/command.wit                                                                                               [ 9/63]   Done: wit/deps/cli/environment.wit                                                                                           [10/63]   Done: wit/deps/cli/exit.wit                                                                                                  [11/63]   Done: wit/deps/cli/imports.wit                                                                                               [12/63]   Done: wit/deps/cli/run.wit                                                                                                   [13/63]   Done: wit/deps/cli/stdio.wit                                                                                                 [14/63]   Done: wit/deps/cli/terminal.wit                                                                                              [16/63]   Done: wit/deps/clocks/monotonic-clock.wit                                                                                    [17/63]   Done: wit/deps/clocks/wall-clock.wit                                                                                         [18/63]   Done: wit/deps/clocks/world.wit                                                                                              [20/63]   Done: wit/deps/filesystem/preopens.wit                                                                                       [21/63]   Done: wit/deps/filesystem/types.wit                                                                                          ✨   Done! New project created /home/horacio/git/wasmcloud/hello

Project generated and is located at: /home/horacio/git/wasmcloud/hello

Congratulations on installing wash!  Shell auto-complete is available. For instructions on setting up auto-complete for your shell, please see 'https://github.com/wasmCloud/wasmCloud/blob/main/crates/wash-cli/Completions.md'


$ wash build
Actor built and signed and can be found at "/home/horacio/git/wasmcloud/hello/build/http_hello_world_s.wasm"


$ wash app deploy wadm.yaml 
Successfully deployed model tinygo-http-hello-world v0.0.1

$ wash app list                                                                                                                                                                                                        
  Name                      Latest Version   Deployed Version   Deploy Status   Description                                                                                                             
  tinygo-http-hello-world   v0.0.1           v0.0.1               Reconciling   HTTP hello world demo in Golang (TinyGo), using the WebAssembly Component Model and WebAssembly Interfaces Types (WIT)

Exactly the same thing in the two instances.

The wadm logs:

$ cat ~/.wash/downloads/wadm.log 
RUST_LOG was not set or the given directive was invalid: FromEnvError { kind: Env(NotPresent) }
Defaulting logger to `info` level
2024-04-04T23:37:22.770297Z  INFO async_nats::options: event: connected
2024-04-04T23:37:22.778102Z  INFO new: wadm::server: Creating API subscriber topic=wadm.api.>
2024-04-04T23:38:55.911101Z ERROR observe{subscribe_topics=["wasmbus.evt.*"]}:consumer_worker{topic=wadm.evt.default worker_type=wadm::workers::event::EventWorker<wadm::storage::nats_kv::NatsKvStore, wasmcloud_control_interface::Client, async_nats::jetstream::context::Context>}: wadm::consumers::manager: Got error from worker error=Other(NATS error: Error { kind: Other, source: Some(Error { kind: WrongLastSequence, source: Some(Error { code: 400, err_code: ErrorCode(10071), description: Some("wrong last sequence: 14") }) }) }

Caused by:
    failed getting entry: wrong last sequence: 400 (code wrong last sequence: 14, error code 10071))
2024-04-04T23:39:55.912872Z  WARN wadm::storage::reaper: Error when storing updated providers. Will retry on next tick error=NATS error: Error { kind: Other, source: Some(Error { kind: WrongLastSequence, source: Some(Error { code: 400, err_code: ErrorCode(10071), description: Some("wrong last sequence: 25") }) }) }
2024-04-04T23:43:55.912400Z  WARN wadm::storage::reaper: Error when storing updated providers. Will retry on next tick error=NATS error: Error { kind: Other, source: Some(Error { kind: WrongLastSequence, source: Some(Error { code: 400, err_code: ErrorCode(10071), description: Some("wrong last sequence: 64") }) }) }
2024-04-04T23:46:55.912093Z  WARN wadm::storage::reaper: Error when storing updated providers. Will retry on next tick error=NATS error: Error { kind: Other, source: Some(Error { kind: WrongLastSequence, source: Some(Error { code: 400, err_code: ErrorCode(10071), description: Some("wrong last sequence: 93") }) }) }
2024-04-04T23:47:55.913373Z  WARN wadm::storage::reaper: Error when storing updated providers. Will retry on next tick error=NATS error: Error { kind: Other, source: Some(Error { kind: WrongLastSequence, source: Some(Error { code: 400, err_code: ErrorCode(10071), description: Some("wrong last sequence: 102") }) }) }
2024-04-04T23:49:55.912678Z  WARN wadm::storage::reaper: Error when storing updated providers. Will retry on next tick error=NATS error: Error { kind: Other, source: Some(Error { kind: WrongLastSequence, source: Some(Error { code: 400, err_code: ErrorCode(10071), description: Some("wrong last sequence: 121") }) }) }
2024-04-04T23:51:55.913604Z  WARN wadm::storage::reaper: Error when storing updated providers. Will retry on next tick error=NATS error: Error { kind: Other, source: Some(Error { kind: WrongLastSequence, source: Some(Error { code: 400, err_code: ErrorCode(10071), description: Some("wrong last sequence: 140") }) }) }

And the wasmcloud logs:

$ cat ~/.wash/downloads/wasmcloud.log 
2024-04-04T23:37:55.902368Z  INFO async_nats::options: event: connected
2024-04-04T23:37:55.903584Z  INFO async_nats::options: event: connected
2024-04-04T23:37:55.905210Z  INFO wasmcloud_host::wasmbus: created bucket with 1 replica bucket=LATTICEDATA_default
2024-04-04T23:37:55.906141Z  INFO wasmcloud_host::wasmbus: created bucket with 1 replica bucket=CONFIGDATA_default
2024-04-04T23:37:55.910310Z  INFO wasmcloud_host::wasmbus: wasmCloud host started host_id="NAMRQYDSYOPNLKW5HLN4EFORRDDH6FCJIVK3D2OGGCDM6W76RGR3DP6Z"
2024-04-04T23:38:03.685975Z  INFO wasmcloud_host::wasmbus: handling launch provider provider_ref="wasmcloud.azurecr.io/httpserver:0.19.1" link_name="default"
2024-04-04T23:38:03.845264Z  INFO wasmcloud_host::wasmbus: actor started actor_ref="file:///home/horacio/git/wasmcloud/hello/build/http_hello_world_s.wasm"
2024-04-04T23:38:37.036856Z  INFO wasmcloud_host::wasmbus: provider started provider_ref="wasmcloud.azurecr.io/httpserver:0.19.1" link_name="default"
Log level was not set by host or environment variable.
Defaulting logger to `info` level
Starting capability provider VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M instance 018eab7a-846c-fa63-9906-1dcc3b04f5d3 with nats url nats://127.0.0.1:4222
2024-04-04T23:38:37.044968Z  INFO wasmbus_rpc::rpc_client: nats client connected
2024-04-04T23:38:42.044759Z  INFO wasmcloud_host::wasmbus: handling put link definition actor_id="MAYZWT4UTBB7BDSJBZKZSW5G4KGMKDBPI5Q43UUW2A5BDQX472YLNLPY" provider_id="VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M" link_name="default" contract_id="wasmcloud:httpserver"
2024-04-04T23:38:42.045836Z  INFO wasmcloud_host::wasmbus: process link definition entry put actor_id="MAYZWT4UTBB7BDSJBZKZSW5G4KGMKDBPI5Q43UUW2A5BDQX472YLNLPY" provider_id="VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M" link_name="default" contract_id="wasmcloud:httpserver"
2024-04-04T23:38:42.046567Z  INFO wasmbus_rpc::provider: Linking actor with provider
2024-04-04T23:38:42.046613Z  INFO wasmcloud_provider_httpserver: httpserver starting listener for actor addr=127.0.0.1:8080 actor_id=MAYZWT4UTBB7BDSJBZKZSW5G4KGMKDBPI5Q43UUW2A5BDQX472YLNLPY

Conclusions

  • I can't install the latest release of wash in any of the machines, as it isn't available neither as cargo module nor as .deb or .rpm, so I had to install 0.26
  • I get the same result, Reconciling state, than in the first post of the thread, so it seems it can be reproduced.

Any idea of how could I try to advance?

@LostInBrittany LostInBrittany changed the title [BUG] wash app deploy succeeds but does not deploy on Fedora Linux [BUG] wash app deploy succeeds but does not deploy neither on Fedora nor on Ubuntu Apr 9, 2024
@sunny-g
Copy link

sunny-g commented Apr 17, 2024

I think I'm encountering the same/similar issue on MacOS 14.4.1:
wadm: v0.11.0-alpha.4
wash: 0.27.0-alpha.3
wasmcloud: 1.0.0-rc.2
starting command: wash up --wasmcloud-version=v1.0.0-rc.2 --log-level=trace --disable-wadm --experimental -d

deploying the following yaml results in a Reconciling deploy status, with the component deployed but the provider hangs:

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: helloworld
  annotations:
    version: v0.0.1
    description: "HTTP hello world demo, using WAC, WIT and the datalove:http provider server."
  experimental: true
spec:
  components:
    - name: applet
      type: component
      properties:
        image: file://./examples/hello-world/build/hello_world_s.wasm
      traits:
        - type: spreadscaler
          properties:
            replicas: 1

    - name: localhttp
      type: capability
      properties:
        image: file://./providers/localhttp/build/localhttp.par.gz
      traits:
        - type: link
          properties:
            target: applet
            namespace: wasi
            package: http
            interfaces: [incoming-handler]
            source_config:
              - name: applet-http
                properties:
                  address: 127.0.0.1:8080

I can manually start the provider, but manually configuring the link also doesn't seem to get everything up and running.

@brooksmtownsend
Copy link
Member

brooksmtownsend commented Apr 17, 2024 •
edited

@sunny-g if you're on wadm v0.11.0-alpha.4 you shouldn't hit this issue, but with any previous version of wadm you would have needed to create the configuration ahead of time. Do you see any other information in your ~/.wash/downloads/wasmcloud.log file?

(for posterity)
To create that configuration: wash config put applet-http address=127.0.0.1:8080

@sunny-g
Copy link

sunny-g commented Apr 17, 2024

I see a ton in the log file :) One previous version of the tooling would error if that config was not put in place already; however, with these versions I not only don't see an error when deploying without the configuration, but wash ui doesn't show the configuration even after wash config put ....

Likewise, I am seeing errors related to the provider deployment, but they persist even after nuking ~/.wash, suggesting that there's extra state being persisted somewhere beyond ~/.wash that doesn't get wash drain'ed correctly or purged from NATS correctly:

2024-04-17T16:41:51.418083Z  INFO handle_ctl_message{subject=wasmbus.ctl.v1.default.provider.start.NANKKKWVPL2IUTX2PBYKNKGZ32PFSCO3ZGGXTZN4MZE6TDYZQGLHYPCS}:handle_start_provider: wasmcloud_host::wasmbus: handling start provider provider_ref="file:///Users/sunnyg/gitdev/datalove/providers/localhttp/build/localhttp.par.gz" provider_id="helloworld-localhttp"
2024-04-17T16:41:51.418138Z TRACE handle_ctl_message{subject=wasmbus.ctl.v1.default.provider.start.NANKKKWVPL2IUTX2PBYKNKGZ32PFSCO3ZGGXTZN4MZE6TDYZQGLHYPCS}: wasmcloud_host::wasmbus: handled control interface request subject=wasmbus.ctl.v1.default.provider.start.NANKKKWVPL2IUTX2PBYKNKGZ32PFSCO3ZGGXTZN4MZE6TDYZQGLHYPCS
2024-04-17T16:41:51.418270Z TRACE handle_start_provider_task: wasmcloud_host::wasmbus: start provider task provider_ref="file:///Users/sunnyg/gitdev/datalove/providers/localhttp/build/localhttp.par.gz" provider_id="helloworld-localhttp"
2024-04-17T16:41:51.418470Z TRACE handle_ctl_message{subject=wasmbus.ctl.v1.default.link.get}: wasmcloud_host::wasmbus: handling control interface request subject=wasmbus.ctl.v1.default.link.get
2024-04-17T16:41:51.418500Z DEBUG handle_ctl_message{subject=wasmbus.ctl.v1.default.link.get}:handle_links: wasmcloud_host::wasmbus: handling links
2024-04-17T16:41:51.418514Z TRACE handle_ctl_message{subject=wasmbus.ctl.v1.default.link.get}: wasmcloud_host::wasmbus: handled control interface request subject=wasmbus.ctl.v1.default.link.get
2024-04-17T16:41:51.418779Z TRACE handle_ctl_message{subject=wasmbus.ctl.v1.default.config.get.helloworld-applet_http}: wasmcloud_host::wasmbus: handling control interface request subject=wasmbus.ctl.v1.default.config.get.helloworld-applet_http
2024-04-17T16:41:51.418807Z TRACE handle_ctl_message{subject=wasmbus.ctl.v1.default.config.get.helloworld-applet_http}:handle_config_get{config_name="helloworld-applet_http"}: wasmcloud_host::wasmbus: handling get config config_name=helloworld-applet_http
2024-04-17T16:41:51.418991Z TRACE handle_ctl_message{subject=wasmbus.ctl.v1.default.config.get.helloworld-applet_http}: wasmcloud_host::wasmbus: handled control interface request subject=wasmbus.ctl.v1.default.config.get.helloworld-applet_http
2024-04-17T16:41:51.442441Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 77 })}:resolve_def{ty=TypeDef { name: None, kind: Handle(Own(Id { idx: 75 })), owner: None, docs: Docs { contents: None } }}:resolve_def{ty=TypeDef { name: Some("incoming-request"), kind: Type(Id(Id { idx: 28 })), owner: Interface(Id { idx: 11 }), docs: Docs { contents: None } }}:resolve{ty=Id(Id { idx: 28 })}:resolve_def{ty=TypeDef { name: Some("incoming-request"), kind: Resource, owner: Interface(Id { idx: 2 }), docs: Docs { contents: None } }}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "incoming-request" })))
2024-04-17T16:41:51.442480Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 77 })}:resolve_def{ty=TypeDef { name: None, kind: Handle(Own(Id { idx: 75 })), owner: None, docs: Docs { contents: None } }}:resolve_def{ty=TypeDef { name: Some("incoming-request"), kind: Type(Id(Id { idx: 28 })), owner: Interface(Id { idx: 11 }), docs: Docs { contents: None } }}:resolve{ty=Id(Id { idx: 28 })}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "incoming-request" })))
2024-04-17T16:41:51.442497Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 77 })}:resolve_def{ty=TypeDef { name: None, kind: Handle(Own(Id { idx: 75 })), owner: None, docs: Docs { contents: None } }}:resolve_def{ty=TypeDef { name: Some("incoming-request"), kind: Type(Id(Id { idx: 28 })), owner: Interface(Id { idx: 11 }), docs: Docs { contents: None } }}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "incoming-request" })))
2024-04-17T16:41:51.442504Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 77 })}:resolve_def{ty=TypeDef { name: None, kind: Handle(Own(Id { idx: 75 })), owner: None, docs: Docs { contents: None } }}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "incoming-request" })))
2024-04-17T16:41:51.442511Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 77 })}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "incoming-request" })))
2024-04-17T16:41:51.442524Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 78 })}:resolve_def{ty=TypeDef { name: None, kind: Handle(Own(Id { idx: 76 })), owner: None, docs: Docs { contents: None } }}:resolve_def{ty=TypeDef { name: Some("response-outparam"), kind: Type(Id(Id { idx: 11 })), owner: Interface(Id { idx: 11 }), docs: Docs { contents: None } }}:resolve{ty=Id(Id { idx: 11 })}:resolve_def{ty=TypeDef { name: Some("response-outparam"), kind: Resource, owner: Interface(Id { idx: 2 }), docs: Docs { contents: None } }}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "response-outparam" })))
2024-04-17T16:41:51.442547Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 78 })}:resolve_def{ty=TypeDef { name: None, kind: Handle(Own(Id { idx: 76 })), owner: None, docs: Docs { contents: None } }}:resolve_def{ty=TypeDef { name: Some("response-outparam"), kind: Type(Id(Id { idx: 11 })), owner: Interface(Id { idx: 11 }), docs: Docs { contents: None } }}:resolve{ty=Id(Id { idx: 11 })}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "response-outparam" })))
2024-04-17T16:41:51.442554Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 78 })}:resolve_def{ty=TypeDef { name: None, kind: Handle(Own(Id { idx: 76 })), owner: None, docs: Docs { contents: None } }}:resolve_def{ty=TypeDef { name: Some("response-outparam"), kind: Type(Id(Id { idx: 11 })), owner: Interface(Id { idx: 11 }), docs: Docs { contents: None } }}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "response-outparam" })))
2024-04-17T16:41:51.442560Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 78 })}:resolve_def{ty=TypeDef { name: None, kind: Handle(Own(Id { idx: 76 })), owner: None, docs: Docs { contents: None } }}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "response-outparam" })))
2024-04-17T16:41:51.442571Z TRACE handle_scale_actor_task:fetch_actor:new:resolve{ty=Id(Id { idx: 78 })}: wrpc_types: return=Ok(Resource(Dynamic(DynamicResource { instance: "wasi:http/types@0.2.0", name: "response-outparam" })))
2024-04-17T16:41:51.442766Z DEBUG handle_scale_actor_task:start_actor: wasmcloud_host::wasmbus: starting new component component_ref="file:///Users/sunnyg/gitdev/datalove/examples/hello-world/build/hello_world_s.wasm" max_instances=1
2024-04-17T16:41:51.442796Z TRACE handle_scale_actor_task:start_actor:store_claims: wasmcloud_host::wasmbus: storing claims claims=Component(StoredComponentClaims { call_alias: "", issuer: "ACDKMRG2F3WMBTOURHFOUUBHSQI55B2JVY2L4T2HQRAAWBIRECIZYMRZ", name: "localhttp-hello-world", revision: "0", subject: "MBUNUFDRQWRBOQMTAOGOCZ4ZHUH27FIT6MTKQHRG7NY575VZNJZWIB3W", tags: ["wasmcloud.com/experimental"], version: "0.0.1" }) key="CLAIMS_MBUNUFDRQWRBOQMTAOGOCZ4ZHUH27FIT6MTKQHRG7NY575VZNJZWIB3W"
2024-04-17T16:41:51.443449Z DEBUG process_entry:process_claims_put: wasmcloud_host::wasmbus: process claim entry put pubkey="MBUNUFDRQWRBOQMTAOGOCZ4ZHUH27FIT6MTKQHRG7NY575VZNJZWIB3W"
2024-04-17T16:41:51.443741Z DEBUG process_entry:process_component_spec_put: wasmcloud_host::wasmbus: process component spec put id="helloworld-applet"
2024-04-17T16:41:51.443761Z TRACE handle_scale_actor_task:start_actor:instantiate_actor: wasmcloud_host::wasmbus: instantiating component component_ref="file:///Users/sunnyg/gitdev/datalove/examples/hello-world/build/hello_world_s.wasm" max_instances=1
2024-04-17T16:41:51.443807Z  INFO handle_scale_actor_task:start_actor: wasmcloud_host::wasmbus: component started component_ref="file:///Users/sunnyg/gitdev/datalove/examples/hello-world/build/hello_world_s.wasm"
2024-04-17T16:41:51.445292Z TRACE handle_ctl_message{subject=wasmbus.ctl.v1.default.link.get}: wasmcloud_host::wasmbus: handling control interface request subject=wasmbus.ctl.v1.default.link.get
2024-04-17T16:41:51.445307Z DEBUG handle_ctl_message{subject=wasmbus.ctl.v1.default.link.get}:handle_links: wasmcloud_host::wasmbus: handling links
2024-04-17T16:41:51.445319Z TRACE handle_ctl_message{subject=wasmbus.ctl.v1.default.link.get}: wasmcloud_host::wasmbus: handled control interface request subject=wasmbus.ctl.v1.default.link.get
2024-04-17T16:41:51.532177Z TRACE handle_start_provider_task:store_claims: wasmcloud_host::wasmbus: storing claims claims=Provider(StoredProviderClaims { issuer: "ABFXM3XVPSEH5A5GEBUR7IFQW6UYGOP4UAG7TQPS5DS72VYG6VLBFJCG", name: "datalove:localhttp", revision: "0", subject: "VBZA7BB7SJ5ZQCUQ7KWR3PBSVVEBNVAVSPD3EO7FDUE3S2FNUYNJDVNG", version: "0.0.1", config_schema: None }) key="CLAIMS_VBZA7BB7SJ5ZQCUQ7KWR3PBSVVEBNVAVSPD3EO7FDUE3S2FNUYNJDVNG"
2024-04-17T16:41:51.532510Z DEBUG process_entry:process_claims_put: wasmcloud_host::wasmbus: process claim entry put pubkey="VBZA7BB7SJ5ZQCUQ7KWR3PBSVVEBNVAVSPD3EO7FDUE3S2FNUYNJDVNG"
2024-04-17T16:41:51.532619Z ERROR wasmcloud_host::wasmbus: failed to start provider provider_ref="file:///Users/sunnyg/gitdev/datalove/providers/localhttp/build/localhttp.par.gz" provider_id="helloworld-localhttp" err=existing component specification URL [ghcr.io/wasmcloud/http-server:0.20.0] for provider ID [helloworld-localhttp] does not match provider reference [file:///Users/sunnyg/gitdev/datalove/providers/localhttp/build/localhttp.par.gz] (you may need to pick a new ID)

@brooksmtownsend
Copy link
Member

brooksmtownsend commented Apr 17, 2024 •
edited

Ah yep @sunny-g , this is what I fixed in #1909, which was an issue where we kept that old provider reference around. For a quick fix if you have the NATS CLI, you can run this to delete the entry:

nats kv del CONFIGDATA_default COMPONENT_helloworld-localhttp

Sorry that you hit this before we could release that fix!

edit: Otherwise everything looks okay, so that should resolve the issue (you can even run it while everything is deployed and wadm will start it eventually)

@sunny-g
Copy link

sunny-g commented Apr 17, 2024 •
edited

Oh no problem, I've been a bit too eager to use wasmcloud so bleeding edge problems are to be expected :)

However I'm still getting the same behavior: when starting wash up, I'm still seeing

2024-04-17T18:12:13.897600Z DEBUG new:process_entry:process_component_spec_put: wasmcloud_host::wasmbus: process component spec put id="localhttp"
2024-04-17T18:12:13.897681Z DEBUG new:process_entry:process_claims_put: wasmcloud_host::wasmbus: process claim entry put pubkey="MBUNUFDRQWRBOQMTAOGOCZ4ZHUH27FIT6MTKQHRG7NY575VZNJZWIB3W"
2024-04-17T18:12:13.897761Z DEBUG new:process_entry:process_component_spec_put: wasmcloud_host::wasmbus: process component spec put id="helloworld-applet"
2024-04-17T18:12:13.897836Z DEBUG new:process_entry:process_claims_put: wasmcloud_host::wasmbus: process claim entry put pubkey="VBZA7BB7SJ5ZQCUQ7KWR3PBSVVEBNVAVSPD3EO7FDUE3S2FNUYNJDVNG"

messages despite having ran wash drain all and rm -r ~/.wash; likewise, even after running those commands wash get claims is not empty.

Is there a command to drain/clear all outstanding claims (assuming that would be useful for getting to a clean slate)? Likewise, is there a way I can easily clear all wash/wadm/wasmcloud state from NATS?

I know I'm being a pedant haha, but lingering state makes everything harder to debug, especially if its not or can't be thoroughly documented yet.

@brooksmtownsend
Copy link
Member

brooksmtownsend commented Apr 17, 2024 •
edited

We don't currently have the functionality to drain all outstanding claims (we have planned to do it in #1740, actually)

What you can do in the meantime is clear out the NATS Jetstream directory, which will do it 😄 This is basically the functionality we'd implement in a wash drain nats type of command.

> cat ~/.wash/downloads/nats.log|grep Store
[77685] 2024/04/17 10:34:01.447656 [INF]   Store Directory: "/var/folders/rf/kgpfv4t9437cj3h3c77vglrr0000gn/T/wash-jetstream-4222/jetstream"

> rm -rf /var/folders/rf/kgpfv4t9437cj3h3c77vglrr0000gn/T/wash-jetstream-4222/jetstream

edit: we specify this in the NATS config as well:

~/.wash/downloads/nats.conf

jetstream {
    domain=core
    store_dir="/var/folders/rf/kgpfv4t9437cj3h3c77vglrr0000gn/T/wash-jetstream-4222"
}

websocket {
    port: 4223
    no_tls: true
}

@sunny-g
Copy link

sunny-g commented Apr 17, 2024 •
edited

Strangely, there used to be a ~/.wash/downloads/nats.conf with 0.82, but with these newer versions I'm not seeing it (though I can't pin the reason on the version change, just haven't seen it lately).

Regardless, I:

  • updated to wadm@0.11.0, wash-cli@0.27.0 and wasmcloud@1.0.0
  • nuked all wash-related folders from /var/folders..., did the wash drain all and rm -r ~/.wash just in case,
  • ran wash up --log-level=trace --experimental -d(note: not disabling wadm) and now wash ui can't find any hosts, though it can be found by wash get hosts; likewise, deploying my wadm.yaml from above now results in Could not put manifest to deploy Internal storage error.

Thanks for your patience and sorry if this issue has gotten off-track; let me know what you recommend I try next and how I can help make this easier to debug or document in a PR.

@brooksmtownsend
Copy link
Member

Oh boy, Internal storage error. @sunny-g would you mind opening up another issue with any errors you can see from ~/.wash/downloads/wasmcloud.log and ~/.wash/downloads/wadm.log? I really appreciate your patience here, I have a feeling theres a bug that we can fix when it comes to migrating from 0.82 to 1.0 to make it smoother for folks. Just suggesting a new issue so we can approach it fresh 😄

@LostInBrittany is this still an issue that you're reproducing on a fresh machine?

@sunny-g
Copy link

sunny-g commented Apr 23, 2024

@brooksmtownsend new issue reported here
No problem, I appreciate your active participation :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@LostInBrittany @ricochet @sunny-g @thomastaylor312 @brooksmtownsend