Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Vespa Deploy fails for application package with model files #30813

Open
rharige opened this issue Apr 4, 2024 · 3 comments
Open

Vespa Deploy fails for application package with model files #30813

rharige opened this issue Apr 4, 2024 · 3 comments
Assignees
@thomasht86
Milestone
soon

Comments

@rharige
Copy link

rharige commented Apr 4, 2024 •
edited

Describe the bug
When models are included in the application package, the vespa deploy fails with either of the below two errors:

Uploading application package... failed
Error: invalid application package (400 Bad Request)
{
    "error-code": "BAD_REQUEST",
    "message": "Unable to decompress application stream: Truncated ZIP file"
}

OR

Uploading application package... failed
Error: error from deploy API at 127.0.0.1:19071 (503 Idle timeout expired: 180005/180000 ms):
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503</title>
</head>
<body>
<h2>HTTP ERROR: 503</h2>
<p>Problem accessing /application/v2/tenant/default/prepareandactivate. Reason:
<pre>    Idle timeout expired: 180005/180000 ms</pre></p>
<hr/>
</body>
</html>

To Reproduce
The issue can be reproduced via before / after introducing model files in application package.
Before //

  • Clone and deploy sample app: $vespa clone multi-vector-indexing my-app && cd my-app - this works because it seems the model files are being read from Vespa hosted URLs (see service.xml supplied in the sample app) and not manually created using optimum-cli

After //
We will continue to use the same sample app, but introduce model files in it.

  • Create model files and delete unnecessary files from the command
    $ optimum-cli export onnx --task sentence-similarity -m intfloat/multilingual-e5-small multilingual-e5-small-onnx
  • Move the model files under my-app\models and make refer to the model files in service.xml
  • Retry vespa deploy --wait 600 -- this is where the error occurs

Expected behavior
Vespa Deploy to succeed.

Screenshots
If applicable, add screenshots to help explain your problem.
Adding Vespa Logs snippet:

INFO    configserver     Container.com.yahoo.vespa.config.server.http.HttpErrorResponse	Returning response with response code 400, error-code:BAD_REQUEST, message=Unable to decompress application stream: Truncated ZIP file

Environment (please complete the following information):

  • OS: MacOS 14.0 (23A344)
  • Infrastructure: self-hosted
  • Docker: Container memory usage: 3.95GB / 7.48GB
  • Docker: Container CPU usage: 11.13% / 1200% (12 CPUs available)
% docker version
Client:
 Cloud integration: v1.0.35+desktop.11
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:26 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.28.0 (139021)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:22 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Vespa version
Include Vespa version(s) used.

  • Vespa version: 8.325.46
  • Vespa Docker Image aee351eb05b1b2f54c6541b9cd38eff68ba30a0d1d290e996a290049c2942992
  • Vespa CLI version 8.323.45 compiled with go1.22.1 on darwin/arm64

Additional context
Add any other context about the problem here.
@bjormel bjormel assigned mpolden and thomasht86 and unassigned mpolden Apr 5, 2024
@thomasht86
Copy link

Hi!
I was able to (almost) reproduce your issue, encountering

Uploading application package... failed
Error: invalid application package (400 Bad Request)
Invalid application:
Could not import model 'sentencepiece_bpe':
Could not import a Vespa model from '/opt/vespa/var/db/vespa/config_server/serverdb/tenants/default/sessions/13/.preprocessed/models/sentencepiece.bpe.model'

It seems like the issue stems from the file sentencepiece.bpe.model, which you probably have located in the models-directory. The naming of this file likely causes some unexpected behavior, and removing or renaming it will hopefully solve your problem.
We will follow up internally for potential fix/doc improvement.

@thomasht86
Copy link

See also https://docs.vespa.ai/en/reference/stateless-model-reference.html

@rharige
Copy link
Author

rharige commented Apr 5, 2024 •
edited

Hi @thomasht86 - As noted in the steps to reproduce:

Create model files and delete unnecessary files from the command

The sentencepiece.bpe.model file is already removed from my my-app\models directory, it contains only onnx file and tokenizer.json. The issue still occurs, and I can reproduce this issue consistently - hence I raised the bug here.

@frodelu frodelu added this to the soon milestone Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thomasht86 thomasht86

Labels
None yet
Projects
None yet
Milestone
soon
Development

No branches or pull requests
4 participants
@frodelu @mpolden @rharige @thomasht86