[Feat]: view Live screen of devices in novnc of selenium grid 4 via novnc port

[krishtoautomate]

Is your feature request related to a problem? Please describe.
selenium grid 4 supports live view of browsers via no-vnc port but appium session cannot be viewed live though we use mjpegurl or port

Describe the solution you'd like
would like to see live view of devices screen wheres tests being executed on selenium grid with relay appium

Describe alternatives you've considered
is it possible to point appium mjpegurl to selenium grid sessions novnc port to see it live.
we have similar feature in appium inspector.

[KazuCocoa]

Didn't appium:mjpegServerPort in UIA2 and XCUITest driver help?

[mykola-mokhnach]

novnc uses RFB protocol to communicate (also wrapped into websockets), which is something completely different from what MJPEG client would expect.
In theory it would be possible to write a plugin, which may connect to the RFB server, periodically request screen buffers, convert them to JPEGs if necessary and then expose it on some other local port, which MJPEG client could consume. In order to do it one must have a deep knowledge of the Remote Frame Buffer protocol and binary data processing in node.js ;)

[krishtoautomate]

Didn't appium:mjpegServerPort in UIA2 and XCUITest driver help?

this doesn't show in same grid 4 UI. each URL needed to be opened in separate tab and also for large scale, this will not be ideal.

[krishtoautomate]

novnc uses RFB protocol to communicate (also wrapped into websockets), which is something completely different from what MJPEG client would expect. In theory it would be possible to write a plugin, which may connect to the RFB server, periodically request screen buffers, convert them to JPEGs if necessary and then expose it on some other local port, which MJPEG client could consume. In order to do it one must have a deep knowledge of the Remote Frame Buffer protocol and binary data processing in node.js ;)

ya something like this can be done :

`const WebSocket = require('ws');
const axios = require('axios');

// Fetch the MJPEG stream from appium-wda
const mjpegUrl = 'your_appium_wda_mjpeg_url';
const response = await axios.get(mjpegUrl, { responseType: 'stream' });

// Create a WebSocket connection to the noVNC server
const ws = new WebSocket('ws://your_novnc_server_address:your_novnc_port');

response.data.on('data', frameData => {
// Forward the frame data to the noVNC server
ws.send(frameData, { binary: true });
});

response.data.on('end', () => {
// Close the WebSocket connection when the stream ends
ws.close();
});`

[jlipps]

Seems like a great idea for a plugin @krishtoautomate! Let us know when you build it and we will link to it in the docs 😉

[krishtoautomate]

need help.

I buffered mjpeg video to nonvnc port : 7900

and specified below capabilities in selenium node stereotype configs

"se:noVncPort": 7900,
"se:vncEnabled": true,

and started node as:

java -jar selenium-server-4.12.1.jar node --config ./config/IOS-00008101-001C15A20AD2001E.toml --vnc-env-var START_XVFB

and failed to see vnc stream in grid while tests are running.