Skip to content

Python implementation of a multi-processed gRPC client/server with streaming capabilities.

License

Notifications You must be signed in to change notification settings

flavienbwk/gRPC-multiprocessing-streaming

Repository files navigation

gRPC multiprocessing with streaming

This repo is a boilerplate showing how you can use multiprocessing with gRPC with streaming capabilities in Python

Payloads streaming allows bypassing gRPC's 2Gb limit in its Python implementation.

Getting started

This project offers a minimum reproducible example of clients sending batches of videos to gRPC servers.

  • Input : video(s)
  • Output : thumbnail(s)

client.py starts a pool of several gRPC channels. Each client process is attributed a video and cuts it into chunks that are sent to one of the gRPC servers parallely to other client processes.

Each response including the video thumbnail gets returned and cut if exceeding 2Gb as well.

  1. Adjust the NUM_WORKERS and NUM_JOBS variables in docker-compose.yml if desired

  2. Add your videos in the input/ directory. Thumbnails will be saved in output/.

# Download 2 videos (to test multiprocessing)
wget -O ./input/video.mp4 https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_640_3MG.mp4
wget -O ./input/video_2.mp4 https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_640_3MG.mp4
  1. Start playing around by running :
docker-compose build
docker-compose run client

Re-compile proto

You must have python3 and pip installed

# In project root directory
python3 -m pip install grpcio==1.32.0 protobuf==3.12.2
python3 -m grpc_tools.protoc --proto_path=./grpc_config --python_out=./grpc_config --grpc_python_out=./grpc_config ./grpc_config/video_thumbnail.proto

Credits

About

Python implementation of a multi-processed gRPC client/server with streaming capabilities.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published