Replies: 3 comments
-
Are you using the method |
Beta Was this translation helpful? Give feedback.
-
HI @sivanantha321 , The multiprocessing.set_start_method('fork') function is utilized within the Kserve model server (kserve/model_server.py). Here is the transformers_inference.py script that is employed alongside it. ` import kserve def mean_pooling(model_output, attention_mask): class KServeDistilrobertaModel(kserve.Model):
if name == "main": model = KServeDistilrobertaModel("all-distilroberta-v1") model_server = ModelServer() Running this inference.py using The problem occurs after building the Docker image using the Dockerfile and attempting to include the --workers argument in the docker run command. When the --workers argument is not provided, docker run functions perfectly, but it encounters issues when the --workers argument is included. Without --worker args With --worker args |
Beta Was this translation helpful? Give feedback.
-
Hello @sivanantha321 ,By the way, is this problem associated with this pull request? https://github.com/kserve/kserve/pull/3407. |
Beta Was this translation helpful? Give feedback.
-
In the inference.py when we mention the workers in the ModelServer, then the building the docker image and running it fails with below error.
if name == "main":
model = KServeBERTSentimentModel("onnx-all-distilroberta-v1")
model.load()
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
INFO:kserve:Registering model: onnx-all-distilroberta-v1
INFO:kserve:Setting max asyncio worker threads as 9
INFO:kserve:Starting uvicorn with 4 workers
INFO:kserve:Starting gRPC server on [::]:8081
Traceback (most recent call last):
File "/app/onnx_inference.py", line 87, in
model_server.start([model])
File "/usr/local/lib/python3.11/site-packages/kserve/model_server.py", line 226, in start
asyncio.run(servers_task())
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kserve/model_server.py", line 224, in servers_task
await asyncio.gather(*servers)
File "/usr/local/lib/python3.11/site-packages/kserve/model_server.py", line 211, in serve
multiprocessing.set_start_method('fork')
File "/usr/local/lib/python3.11/multiprocessing/context.py", line 247, in set_start_method
raise RuntimeError('context has already been set')
RuntimeError: context has already been set
Beta Was this translation helpful? Give feedback.
All reactions