Gradio components in `gr.Chatbot()` #8131

dawoodkhan82 · 2024-04-25T15:36:12Z

Description

Use Image, Audio, Video, Gallery, and Plot components in the chatbot.

Closes: #(issue)

🎯 PRs Should Target Issues

Before your create a PR, please check to see if there is an existing issue for this change. If not, please create an issue before you create this PR, unless the fix is very small.

Not adhering to this guideline will result in the PR being closed.

Tests

PRs will only be merged if tests pass on CI. To run the tests locally, please set up your Gradio environment locally and run the tests: bash scripts/run_all_tests.sh
You may need to run the linters: bash scripts/format_backend.sh and bash scripts/format_frontend.sh

gradio-pr-bot · 2024-04-29T15:12:45Z

🪼 branch checks and previews

•	Name	Status	URL
	Spaces	ready!	Spaces preview
	Website	ready!	Website preview
	Storybook	ready!	Storybook preview
🦄	Changes	detected!	Details

Install Gradio from this PR

pip install https://gradio-builds.s3.amazonaws.com/5f3321bacc9ecbd8b5fb4017a2d3810478662d4d/gradio-4.31.5-py3-none-any.whl

Install Gradio Python Client from this PR

pip install "gradio-client @ git+https://github.com/gradio-app/gradio@5f3321bacc9ecbd8b5fb4017a2d3810478662d4d#subdirectory=client/python"

gradio-pr-bot · 2024-04-29T15:12:59Z

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
`@gradio/chatbot`	`minor`
`@gradio/image`	`minor`
`@gradio/multimodaltextbox`	`minor`
`@gradio/plot`	`minor`
`gradio`	`minor`

Maintainers can select this checkbox to manually select packages to update.

With the following changelog entry.

Gradio components in gr.Chatbot()

Maintainers or the PR author can modify the PR title to modify this entry.

Something isn't right?

Maintainers can change the version label to modify the version bump.
If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

* Add `pull_request.branches.main` as a trigger of the `publish` workflow * [WIP] Comment out the publish steps * Package and upload the NPM package for debug * Skip the copy_frontend.py hook in the Lite build * add changeset * [WIP] Show gradio files * [WIP] Show gradio files * Comment out installing the gradio and gradio_client libraries * Restore installing gradio_client because it's used by `python js/_website/generate_jsons/generate.py` that follows * Restore installing gradio because it's used by `python js/_website/generate_jsons/generate.py` that follows * Add code * Revert "[WIP] Show gradio files" This reverts commit e15fef2. * Build the gradio wheel with the custom lite target * add changeset * Revert "[WIP] Show gradio files" This reverts commit aef053f. * Revert "Skip the copy_frontend.py hook in the Lite build" This reverts commit ca296d0. * Update .github/actions/install-frontend-deps/action.yml for hatch installation * [WIP] Fix test-functional.yml and .github/actions/install-all-deps/action.yml to call the setup actions in this branch * Revert "[WIP] Fix test-functional.yml and .github/actions/install-all-deps/action.yml to call the setup actions in this branch" This reverts commit 571823b. * Comment-in lines in publish.yml * Move Lite build from publish.yml to deploy-spaces.yml * Use the build_lite option of install-all-deps instead of running the build command * [TMP] Change the branch of action files * Fix the hatch Lite build setting * Return pnpm pack to deploy-space * Revert "[TMP] Change the branch of action files" This reverts commit fe4e1c8. * Remove dependencies for lite build * [TMP] Change the branch of action files * Revert "Remove dependencies for lite build" This reverts commit 856a858. * Install packaging>=23.2 * [TMP] Show packaging version * Fix pip install * Fix * Uninstall packaging once * Use `pip install -U` instead of uninstalling the exiting version * Revert "[TMP] Show packaging version" This reverts commit 910b6bb. * Add `-U` flag * Set packaging version as >=23.2 * Revert the changes on pip install * Set packaging version as >=23.2 in requirements.txt * Revert "Set packaging version as >=23.2" This reverts commit 8aa77c8. * Fix hook name * Revert "Set packaging version as >=23.2 in requirements.txt" This reverts commit fbd605c. * Revert "Revert the changes on pip install" This reverts commit 81ff38a. * Add comments * Revert "[TMP] Change the branch of action files" This reverts commit 0d6aa48. * Revert the trigger of .github/workflows/deploy-spaces.yml * Remove unused `node_auth_token` and `npm_token` inputs from the `install-all-deps` action * [TMP] Trigger CI based on this PR * Remove packging installation * Revert "Remove packging installation" This reverts commit 4a4f18d. * Revert "[TMP] Trigger CI based on this PR" This reverts commit 6cea830. --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com> Co-authored-by: freddyaboulton <alfonsoboulton@gmail.com>

* Add code * add changeset * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* add msw setup and initialisation tests * add changeset * add walk_and_store_blobs improvements and add tests * add changeset * api_info tests * add direct space URL link tests * fix tests * add view_api tests * add post_message test * tweak * add spaces tests * jwt and protocol tests * add post_data tests * test tweaks --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

…eeded (#8174) * Remove hatch installation in js/app/package.json which is no longer needed * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* Update test-functional.yml * Update test-functional.yml * tweaks --------- Co-authored-by: pngwn <hello@pngwn.io>

…8179) * rework upload to be a class method + pass client into each component * add changeset * Update client/js/src/utils/upload_files.ts * fix storybook * review comments * Apply suggestions from code review Co-authored-by: Hannah <hannahblair@users.noreply.github.com> * format * ts fix --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com> Co-authored-by: Hannah <hannahblair@users.noreply.github.com>

* chore(deps): update pnpm to v9 * update workflow --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: pngwn <hello@pngwn.io>

* workspace * add changeset * remove circular import from preview * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* get error message * Support multiple clients * add changeset * add changeset * add changeset * Display in UI * console.error the python traceback * lint --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* convert sse calls in client from async to sync * add changeset * more sync * lint * more sync * fix threadpool * fix timeouts * reuse executor * lint --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* run python reload only if python file changed * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com> Co-authored-by: Freddy Boulton <alfonsoboulton@gmail.com>

…_thread in blocks.py (#8158) * fix: handling SIGINT, single block_thread and fix popen * Use pass --------- Co-authored-by: freddyaboulton <alfonsoboulton@gmail.com>

* add msw setup and initialisation tests * add changeset * add eventsource polyfill for node and browser envs * add changeset * add changeset * config tweak * types * update eventsource usage * add changeset * add walk_and_store_blobs improvements and add tests * add changeset * api_info tests * add direct space URL link tests * fix tests * add view_api tests * add post_message test * tweak * add spaces tests * jwt and protocol tests * add post_data tests * test tweaks * dynamically import eventsource * revet eventsource imports * add node test * lockfile * add client test in root pkg file * lcokfile * remove eventsource from js/app * add changeset * remove ts ignore * move eventsource polyfill to eventsource factory * add changeset * tweak --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* add msw setup and initialisation tests * add changeset * add eventsource polyfill for node and browser envs * add changeset * add changeset * config tweak * types * update eventsource usage * add changeset * add walk_and_store_blobs improvements and add tests * add changeset * api_info tests * add direct space URL link tests * fix tests * add view_api tests * add post_message test * tweak * add spaces tests * jwt and protocol tests * add post_data tests * test tweaks * dynamically import eventsource * revet eventsource imports * add jwt param to sse requests * add stream test * add changeset * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* Add code * add changeset * Update fuzzy-mirrors-scream.md * Update fuzzy-mirrors-scream.md * Fix tests * Update .changeset/fuzzy-mirrors-scream.md Co-authored-by: Abubakar Abid <abubakar@huggingface.co> * Fix code --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com> Co-authored-by: Abubakar Abid <abubakar@huggingface.co>

* Analytics refactor * add changeset * add changeset * Fix wasm? * Fix python tests' * Revert changes chrome * use util function --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

… is not available on Pyodide yet (#8204) * Specify the fastapi version on Lite to avoid ujson installation which is not available on Pyodide yet * add changeset * Refactoring --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* Set the show_api flag on Lite * add changeset * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

…nes on Lite (#8052) * Extend Interface.from_pipeline() to support Transformers.js.py pipelines on Lite (wip: only object-detection in this commit) * add changeset * Add image-classification and image-segmentation * Add zero-shot-image-classification and zero-shot-object-detection * Add document-question-answering * Add feature-extraction and fill-mask * Add question-answering and summarization * Fix an error message * Add text2text-generation, text-classification, and text-generation * Add translation andtranslation_xx_to_yy * Add zero-shot-classification * Add postprocess_takes_inputs to control the args passed to the postprocess function of each pipeline * Add topk option to image-classification * format_backend * Add audio-classification, automatic-speech-recognition, and zero-shot-audio-classification * Add image-to-text * Add token-classification (with JSON component as an output. Is it correct?) * Ignore import type failure of transformers_js_py * Add image-feature-extraction * Add image-to-image * Add text-to-audio * Add depth-estimation * Remove `render=False` * Reorder the if-blocks following the Transformers.js doc * Update gradio/pipelines_utils.py Co-authored-by: Abubakar Abid <abubakar@huggingface.co> * Update gradio/pipelines_utils.py Co-authored-by: Abubakar Abid <abubakar@huggingface.co> * Fix feature-extraction demo * Fix demo title * Add guides/08_gradio-clients-and-lite/gradio-lite-and-transformers-js.md without contents * Rename guides/08_gradio-clients-and-lite/*.md to fix the order * Use pipeline.model.config._name_or_path for the demo title instead of pipeline.model.config.model_type * Fix normal Interface.from_pipeline to use pipeline.model.config.name_or_path as the demo title * Write an article about Gradio-Lite and Transformers.js * Update the doc * tweaks * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com> Co-authored-by: Abubakar Abid <abubakar@huggingface.co>

* remove redundant event source logic * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

* Add connect_heartbeat field * fix types * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

freddyaboulton

🔥 PR @dawoodkhan82 ! Made some suggestions to simplify the api in the backend

demo/chatbot_multimodal/run.py

gradio/components/chatbot.py

dawoodkhan82 · 2024-05-14T16:12:06Z

demo/chatbot_multimodal/run.py

+         ["Video", gr.Video(value="files/world.mp4", render=False)],
+         ["Audio", gr.Audio(value="files/cantina.wav", render=False)],
+         ["Plot", gr.Plot(value=fig, render=False)],
+         ["Gallery", gr.Gallery(value=["files/lion.jpg", "files/cheetah.jpg", "files/zebra.jpg"], render=False)]],


@freddyaboulton Any suggestions on how to simplify this so render=False is not necessary? I tried passing it as a constructor arg during postprocessing, but the component would still render.

Can you call unrender for each component in the initial value in the chatbot init?

Wondering if it would work.

andreped · 2024-05-15T20:50:55Z

@dawoodkhan82 Would be neat if there was added support for gradio.Dataframe, gradio.Code, and gradio.Button as well, as our application needs it :] As a minimum, gradio.Dataframe would be quite neat.

Could be done in a separate PR I suppose. Just throwing it out there, as you are already adding support for five of the most commonly used ones.

dawoodkhan82 · 2024-05-16T00:38:10Z

@andreped For sure, dataframe is already on the roadmap. Will look into adding additional components in a future PR.

aliabid94 · 2024-05-16T20:12:38Z

Sorry should've commented earlier, but this seems a lot of custom code for each component. We should have a general approach that works for all components, rather than having to support each component one by one. We should also only import components "dynamically" if needed, rather than making chatbot have every other type of component as a dependency.
Also think visually, Component outputs should not be in a bubble, but stand alone in their usual Block.
Happy to sync on this to design a solution.

dawoodkhan82 · 2024-05-22T23:04:51Z

Ok worked on refactoring the PR to generalize for all components, but reached some areas where I still can't get rid of some custom code. cc @aliabid94 lmk if you want to sync on a solution to see if its possible without having to change component code itself.

dawoodkhan82 · 2024-05-22T23:05:42Z

gradio/components/chatbot.py

+            else:
+                raise ValueError(
+                    f"Invalid component for Chatbot component: {chat_message.component}"
+                )


@aliabid94 here we still need some custom code for some components

dawoodkhan82 · 2024-05-22T23:06:07Z

js/chatbot/shared/ChatBot.svelte

-										/>
-									{:else if message !== null && message.file?.url !== null}
+									{:else if message !== null && "component" in message}
+										{#if message.component == "gallery"}


Similar thing here, accessing the value can be different per component.

dawoodkhan82 added 3 commits April 24, 2024 14:53

chatbot components

803a059

demoi

eaf3737

merge

0e161e7

add changeset

539ff05

dawoodkhan82 and others added 24 commits April 29, 2024 17:33

preprocess fix

fb2b107

Merge branch 'main' into chatbot-components

54af1b5

add changeset

d1b9604

Make guide for tailwind more verbose (#8152)

838965f

Add ETag to /custom_component route to control browser caching (#8170)

80b868e

* Add code * add changeset * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

Remove hatch installation in js/app/package.json which is no longer n…

e4bd0c4

…eeded (#8174) * Remove hatch installation in js/app/package.json which is no longer needed * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

Update test-functional.yml - Fix vulnerability code injection (#8145)

e94d7da

* Update test-functional.yml * Update test-functional.yml * tweaks --------- Co-authored-by: pngwn <hello@pngwn.io>

chore(deps): update pnpm to v9 (#8123)

7d1c87b

* chore(deps): update pnpm to v9 * update workflow --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: pngwn <hello@pngwn.io>

Use workspace version for code in _website (#8189)

1b3d2b6

* workspace * add changeset * remove circular import from preview * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

Pass Error status in /dev/reload stream (#8106)

1f0483f

* get error message * Support multiple clients * add changeset * add changeset * add changeset * Display in UI * console.error the python traceback * lint --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

run python reload only if python file changed (#8194)

0430f79

* run python reload only if python file changed * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com> Co-authored-by: Freddy Boulton <alfonsoboulton@gmail.com>

fix: handling SIGINT correctly in reload.py, single entrance of block…

8e0d09d

…_thread in blocks.py (#8158) * fix: handling SIGINT, single block_thread and fix popen * Use pass --------- Co-authored-by: freddyaboulton <alfonsoboulton@gmail.com>

Refactor analytics to not use api.gradio.app (#8180)

6b102f8

* Analytics refactor * add changeset * add changeset * Fix wasm? * Fix python tests' * Revert changes chrome * use util function --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

Set the show_api flag on Lite (#8205)

9533ea6

* Set the show_api flag on Lite * add changeset * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

merge

078df9c

hannahblair and others added 10 commits May 6, 2024 10:01

remove redundant event source logic (#8211)

4183dd8

* remove redundant event source logic * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

Only connect to heartbeat if needed (#8169)

c96ef0d

* Add connect_heartbeat field * fix types * add changeset --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com>

chore: update versions (#8172)

1b2fa59

merge

68c70df

fixes

190edf7

type fixes

2b4c079

Merge branch 'main' into chatbot-components

1a9318d

type fixes

ba056cb

Merge branch 'main' into chatbot-components

c3ba20a

notebook fix

5955494

dawoodkhan82 marked this pull request as ready for review May 13, 2024 15:32

type ignore

daeb642

freddyaboulton reviewed May 14, 2024

View reviewed changes

dawoodkhan82 commented May 14, 2024

View reviewed changes

dawoodkhan82 added 3 commits May 15, 2024 11:22

data object model

2fc0ba9

Merge branch 'main' into chatbot-components

d179f7e

remove component in tuple

649c566

dawoodkhan82 added 2 commits May 16, 2024 02:35

more fixes

42144cb

Merge branch 'main' into chatbot-components

688e576

dawoodkhan82 requested a review from pngwn May 16, 2024 18:31

abidlabs requested a review from aliabid94 May 16, 2024 18:37

dawoodkhan82 added 2 commits May 19, 2024 16:25

Merge branch 'main' into chatbot-components

a7ca154

extend components

411e736

dawoodkhan82 commented May 22, 2024

View reviewed changes

merge

5f3321b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gradio components in `gr.Chatbot()` #8131

Gradio components in `gr.Chatbot()` #8131

dawoodkhan82 commented Apr 25, 2024

gradio-pr-bot commented Apr 29, 2024 •

edited

gradio-pr-bot commented Apr 29, 2024 •

edited

Something isn't right?

freddyaboulton left a comment

dawoodkhan82 May 14, 2024

freddyaboulton May 15, 2024

freddyaboulton May 15, 2024

andreped commented May 15, 2024

dawoodkhan82 commented May 16, 2024

aliabid94 commented May 16, 2024

dawoodkhan82 commented May 22, 2024

dawoodkhan82 May 22, 2024

dawoodkhan82 May 22, 2024

Gradio components in gr.Chatbot() #8131

Are you sure you want to change the base?

Gradio components in gr.Chatbot() #8131

Conversation

dawoodkhan82 commented Apr 25, 2024

Description

🎯 PRs Should Target Issues

Tests

gradio-pr-bot commented Apr 29, 2024 • edited

🪼 branch checks and previews

gradio-pr-bot commented Apr 29, 2024 • edited

🦄 change detected

This Pull Request includes changes to the following packages.

With the following changelog entry.

Something isn't right?

freddyaboulton left a comment

Choose a reason for hiding this comment

dawoodkhan82 May 14, 2024

Choose a reason for hiding this comment

freddyaboulton May 15, 2024

Choose a reason for hiding this comment

freddyaboulton May 15, 2024

Choose a reason for hiding this comment

andreped commented May 15, 2024

dawoodkhan82 commented May 16, 2024

aliabid94 commented May 16, 2024

dawoodkhan82 commented May 22, 2024

dawoodkhan82 May 22, 2024

Choose a reason for hiding this comment

dawoodkhan82 May 22, 2024

Choose a reason for hiding this comment

Gradio components in `gr.Chatbot()` #8131

Gradio components in `gr.Chatbot()` #8131

gradio-pr-bot commented Apr 29, 2024 •

edited

gradio-pr-bot commented Apr 29, 2024 •

edited