Images passed to Vision models with low quality parameter, even though high quality was chosen

[sigvardt]

Self Checks

• This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.6.7

Cloud or Self Hosted

Cloud

Steps to reproduce

1. Make a workflow with image capabilities and a LLM node with vision capabilities
2. Ensure LLM node is set to High Quality vision
3. Publish and run app
4. Provide image file to the workflow
5. The request sent to the LLM has the quality parameter set to low, even tough it is defined to be high
   Tested across models and providers - issue seems to be persistent

✔️ Expected Behavior

• Defined vision quality settings are reflected in the call the to the LLM endpoint

❌ Actual Behavior

• Discrepancy between settings and executed call to endpoint

[dosubot]

It appears there's a bug affecting the way the Dify platform handles the vision quality setting for LLM nodes, leading to a discrepancy between the selected 'High' quality in the UI and the 'low' quality parameter being sent in the request. This issue might stem from either the UI component responsible for setting the vision quality or the backend logic that constructs the request to the LLM endpoint.

The relevant code for handling vision quality settings is located in the VisionConfig class within the entities.py file, which includes a detail attribute for setting the quality parameter. The UI component for selecting the vision resolution, found in param-config-content.tsx, uses a RadioGroup component to update the visionConfig.detail state. The transition from UI selection to the request sent to the LLM endpoint is managed by the _generate method in the NVIDIALargeLanguageModel class, which constructs the request payload including the quality parameter.

To address this issue, it's essential to review the synchronization between the UI component's state and the backend request construction logic, particularly how the model_parameters are handled in the _generate method. Checking for any recent changes in these areas or known bugs could also provide insights.

For a more detailed investigation, sharing specific models, providers, and any error messages or logs you're encountering would be beneficial. This information could help pinpoint the exact cause of the discrepancy and facilitate a quicker resolution.

Details

Sources

• api/core/workflow/nodes/llm/entities.py
• sdks/nodejs-client/package.json
• web/package.json
• en/tutorials/model-configuration/xinference.md
• web/app/components/app/configuration/config-vision/param-config-content.tsx
• api/core/model_runtime/model_providers/ollama/llm/llm.py
• api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py
• api/core/model_runtime/model_providers/nvidia/llm/llm.py
• api/core/model_runtime/model_providers/xinference/xinference_helper.py
• web/app/components/workflow/nodes/llm/use-config.ts
• en/features/workflow/nodes/llm.md
• api/core/workflow/nodes/llm/llm_node.py
• web/app/components/app/configuration/index.tsx
• web/app/components/app/configuration/config-vision/index.tsx
• web/app/components/workflow/nodes/llm/panel.tsx

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.