Allow image based questions from /api/conversation/azure_byod

[adamdougal]

Required by #323

Description

When calling the /api/conversation/azure_byod endpoint, generate embeddings of the question using computer vision and query the image embeddings using the question embeddings. We should continue to create question embeddings using text-embeddings-ada-002 in addition to be used to search the image description.

Tasks

• Generate question embeddings using computer vision
• Query the search index using the generated embeddings
• Pass any returned images to gpt-4-vision along with the chat

[cecheta]

I'm not sure how you envision this @adamdougal , but I'm not sure it's possible, as Azure OpenAI on your data only supports vector search using Ada embeddings

[cecheta]

Is it that we use the user's question to find the closest images, and then we pass these image(s) to GPT-4?

If so, that seems like quite a lot of custom code for the BYOD endpoint, I had thought that the byod endpoint was essentially a wrapper for Azure OpenAI On Your Data without much customisation

[adamdougal]

I wasn't sure if it would be possible. If that's the case then we should ensure we document the limitation.

I don't think we should add any custom code to support this.