Generic Multimodal Support #1021
Conversation
|
This is amazing. When this is merged, please ping me. I would like to adapt it for OpenAI + Gemini 1.5 Pro. ✌️ |
Saghen
force-pushed
the
feat/multimodal-anthropic
branch
from
e33c2e6
to
6c952e9
Compare
|
This is amazing! Would be nice to extend this to openai api as well if possible. |
|
Yes amazing! It would be so great to have also OpenAI-like API compatibility, so many Open sources multimodal models are available like Idefics2, Llava, llama-3-vision, ... :) |
|
Hey @Saghen, PR looking great from my local testing! We changed a few things last week since we switched our docker image to a new build process. That probably introduced some conflicts but I don't mind fixing them for you since I created them 😅 If you're ok giving me write-access on the PR then I can just do the merge commit directly. |
|
@nsarrazin that'd be great, thanks! granted you permission |
|
|
||
| const module = await import("browser-image-resizer"); | ||
| // currently, only IDEFICS is supported by TGI | ||
| // the size of images is hardcoded to 224x224 in TGI | ||
| // this will need to be configurable when support for more models is added | ||
| const resizedImages = await Promise.all( | ||
| files.map(async (file) => { | ||
| return await module | ||
| .readAndCompressImage(file, { | ||
| maxHeight: 224, | ||
| maxWidth: 224, | ||
| quality: 1, | ||
| }) | ||
| .then(async (el) => await file2base64(el as File)); | ||
| }) | ||
| const base64Files = await Promise.all( | ||
| (files ?? []).map((file) => | ||
| file2base64(file).then((value) => ({ type: "base64" as const, value, mime: file.type })) | ||
| ) | ||
| ); | ||
|
|
There was a problem hiding this comment.
Would be nice to be able to do the resizing in browser but it should be configurable indeed. Would save quite some bandwidth I think at huggingchat scale
Maybe model.multimodal could be true | { maxSize?: number, preferredMimeType?: string } ? That would give us a place to store multimodal specific settings, and the client has access to the model so we could do the resizing there.
There was a problem hiding this comment.
I created a more generic image processor in c8814f4 and put the multimodal options at endspoints[*].multimodal. What do you think?
There was a problem hiding this comment.
WRT to bandwidth, it might make sense to do some image processing on the server on upload to convert to a more suitable format (i.e. put everything in AVIF/WEBP?) but I found the existing upload limit to get in the way
|
And thanks for exposing the mime type in files 🔥 that's gonna be super handy down the road as we support more modalities |
|
@Ichigo3766 @Extremys @flexchar heads up that it was trivial so I added support for OpenAI in this PR as well |
Saghen
changed the title
|
@Saghen I will review it soon. Could you merge/rebase with the main so that the merge conflicts are gone ❤️ |
Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu>
Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu>
Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu>
Adds support for multimodal with Anthropic by increasing the maximum file size, adjusting the
message.filestype to support mime and removing the assumptions around TGI.string[]to{ type: 'hash' | 'base64', value: string, mime: string }prompting to TGI endpoint codeI'd like to move the file upload logic out of the UI code and begin uploading immediately upon selecting a file, but that's outside the scope of this PR. However, that should allow for processing files earlier, which could be particularly useful for non-images (i.e. making embeddings for PDFs).