PV forecast in browser - pyodide

[zakwatts]

To investigate if pyodide (https://pyodide.org/en/stable/) can be used to run the a PV forecast in a browser application on WebAssembly.

[zakwatts]

@mrchrisadams I made this github issue for your comment. We can use this issue as place to discuss this.

[mrchrisadams]

Thanks @zakwatts!

I think for this to run in a browser, I think we'd need to check that the list of packages used by this quartz solar forecast would be compatible with Pyodide, and there is a list of packages pre-installed already in pyodide. you can see them, below, along with information about how to create pre-compiled wheels to make including new packages possible:

https://pyodide.org/en/stable/usage/packages-in-pyodide.html

Once something is in the browser, it should be possible to use the browser geolocation tools, to allow it to provide the latlngs. I think this would mean anyone in the world accessing the page in a modern browser would be able to perform a local solar forecast for where they are (as long as the data coverage for the necessary inputs is ok).

You can see an example of using the the browser's geolocation API in this notebook:

https://nextjournal.com/greenweb/experiments-with-the-free-marginal-carbon-intensity-from-wattime

[mrchrisadams]

I've added a post here on social media, to see if any further advice comes in

https://mastodon.social/@mrchrisadams/111900393206118727

[mrchrisadams]

Ok, I've had a bit more of a look.

The following packages are dependencies in this project:

pandas
xarray
pv-site-prediction
pydantic
huggingface_hub # only for evaluation
python-dotenv

If a package is python it works with pyodide "out of the box". If it uses some underlying C library like pandas, then there needs to be a step to compile it for WASM. Rust libraries that are consumed by python also work, like in pydantic's case, through the py03 library.

• pandas - has pyodide friendly version already
• xarray - is pure python
• pv-site-prediction - likely has a bunch of C/C++ deps, from the looks of it
• pydantic - appears to be python (I thought it used rust internally now though)
• huggingface_hub - I think it's pure python
• python-dotenv - pure python, i think

For this to be possible, you'd probably need to figure out how to make the pv-site-prediction library work in pyodide. that might be quite a mission. It's possible and the steps are outlined below for compiling in this way:

https://pyodide.org/en/stable/development/new-packages.html

However, this is definitely beyond my knowledge. If I bump into some WASM / pyodide ninja who can help, I'll update here, but I'm also aware I don't have that much time to commit to this idea, as cool as it might be.

I've made a note to close this issue at the end of the month if that helps keep this issue list easier to manage.

[zakwatts]

Hi @mrchrisadams! Thanks for looking into this and giving a detailed explanation of the issues with the dependencies.

for pv-site-prediction the packages used are (from the pyproject.toml:
python = "^3.10,<3.12"
scikit-learn = "^1.1.3,<1.2.0"
pvlib = "^0.9.3"
numpy = "1.23.5"
xarray = "^2022.12.0"
click = "^8.1.3"
google-cloud-storage = "^2.7.0"
zarr = "^2.13.3"
dask = "^2022.12.1"
h5netcdf = "^1.1.0"
pyproj = "^3.4.1"
fsspec = "^2022.11.0"
gcsfs = "^2022.11.0"
torch = {version="^1.13.1", optional=true}
tqdm = "^4.64.1"
ocf-blosc2 = "^0.0.2"
netcdf4 = "^1.6.3"
pyresample = "^1.27.1"

After doing a bit of research the issues will mostly likely be with the libraries torch, pyproj, h5netcdf, netcdf4 having heavy C or C++ dependencies (and potentially google-cloud-storage and gcsfs).

Using pytorch in pyodide has been looked into before but it seems the large amount of dependencies involved might make it a large task. Also looks like there is an open issue on pytorch for it, but looks outside the scope of the core dev team. And yes if you bump into an WASM ninjas do let us know! :)

[zakwatts]

Closing this issue for now

[zakwatts]

@all-contributors please add @mrchrisadams for ideas.

[allcontributors]

@zakwatts

I've put up a pull request to add @mrchrisadams! 🎉