Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

explore acceleration w WebAssembly #400

Closed
jywarren opened this issue Oct 11, 2018 · 51 comments · Fixed by #1093
Closed

explore acceleration w WebAssembly #400

jywarren opened this issue Oct 11, 2018 · 51 comments · Fixed by #1093
Assignees
@Divy123
Labels
enhancement help wanted performance project

Comments

@jywarren
Copy link
Member

WebAssembly seems like a way we might speed up pixel-by-pixel operations.

https://developer.mozilla.org/en-US/docs/WebAssembly

But would it suffer from the same issues as webworkers, i.e. a cost of passing the image data over?
@jywarren
Copy link
Member Author

Also note this way of writing WebAssembly:

https://github.com/ballercat/walt

@gitmate
Copy link

gitmate bot commented Oct 11, 2018

GitMate.io thinks the contributor most likely able to help you is @ccpandhare.

A possibly related issue is #237 (Explore OpenCV.js module).

@tech4GT
Copy link
Member

tech4GT commented Oct 11, 2018

@jywarren Even I was thinking we can write the pixel manipulation code in c and create a js binding to it, is that the same as web assembly? I mean the code shows up as native code.

@gitmate gitmate bot added the enhancement label Oct 11, 2018
@publiclab publiclab deleted a comment from gitmate bot Nov 1, 2018
@jywarren
Copy link
Member Author

jywarren commented Nov 1, 2018

Interesting -- well, the WebAssembly approach means it'd be runnable at (near?) native speed in the browser, too! I think it could get really fast -- maybe not GPU speed, but maybe closer to video speed. For a speed reference we'd like to get closer to, see the Infragram WebGL demo: https://publiclab.github.io/infragram/

@jywarren
Copy link
Member Author

jywarren commented Nov 1, 2018

Walt seems like a good way to easily write webassembly code!

Provide a thin layer of syntax sugar on top of .wat text format. Preferably porting as much of JavaScript syntax to WebAssembly as possible. This improved syntax should give direct control over the WebAssembly output. Meaning there should be minimal to none post optimization to be done to the wast code generated. The re-use of JavaScript semantics is intentional as I do not wish to create a brand new language.

@jywarren
Copy link
Member Author

jywarren commented Nov 8, 2018

Also noting imagemagick has been compiled for webassembly... Wow!

https://github.com/KnicKnic/WASM-ImageMagick/blob/master/Readme.md

@jywarren
Copy link
Member Author

jywarren commented Nov 8, 2018

I think we could write a version of Pixelmanipulation following this example!

https://github.com/ballercat/walt/wiki/Walt-In-5-Minutes

...but would we be able to run a JavaScript function inside? I'm not sure about that. Would the function have to be compiled as well?

@jywarren
Copy link
Member Author

jywarren commented Nov 8, 2018

Docs for reading an image into webassembly: https://developer.mozilla.org/en-US/docs/WebAssembly/existing_C_to_wasm

@jywarren
Copy link
Member Author

jywarren commented Nov 8, 2018

This is a great area for testing code snippets!

https://ballercat.github.io/walt/

@jywarren jywarren mentioned this issue Nov 11, 2018
Closed
@jywarren
Copy link
Member Author

Just asked over at the Walt project if their compiler runs in the browser. Otherwise we could look at precompiling these at build time.

@jywarren
Copy link
Member Author

jywarren commented Nov 11, 2018 •
edited

So I think if the simple pixel manipulation function we pass into the PixelManipulation module is close enough to Walt syntax, which is actually WebAssembly, but "looks like" javascript... so if the inside of a function like this could be compiled almost directly, that'd be interesting:

image-sequencer/src/modules/Brightness/Module.js

Lines 22 to 29 in 1311119

function changePixel(r, g, b, a){
var val = (options.brightness)/100.0
r = val*r<255?val*r:255
g = val*g<255?val*g:255
b = val*b<255?val*b:255
return [r , g, b, a]
}

By comparison, here is a section of walt code:

  for (i = 0; i < total; i += 1) {
    particle = i * particleByteSize;
    particle = {
      x: width / 2.0,
      y: height / 2.0,
      vx: ((i * 7 % 200) - 100) / 100.0,
      vy: ((i * 19 % 200) - 100) / 100.0
    };
  }
const x: i32 = 2;
export function echo(): i32 {
  const x: i32 = 42;
  return x;
}

@jywarren
Copy link
Member Author

Walt folks responded to say that walt does indeed compile in the browser, on the fly! Awesome.

@Mridul97
Copy link

Mridul97 commented Dec 31, 2018 •
edited

I was reading the above discussions and I think we can implement pixel manipulation code in c and then compile it into wasm file and then add it into the project. After that we can instantiate the WebAssembly module and use the functions written in pixel manipulation file, which is in c, in the js files!
cc @jywarren @tech4GT

@jywarren
Copy link
Member Author

jywarren commented Jan 2, 2019 via email

@Divy123
Copy link
Member

Divy123 commented Mar 1, 2019

Can I work on this issue?

@jywarren
Copy link
Member Author

jywarren commented Mar 1, 2019 via email

@harshkhandeparkar
Copy link
Member

harshkhandeparkar commented Mar 2, 2019 •
edited

@jywarren @publiclab/is-reviewers nectarJs claims that it can compile js to native OS binaries for all platforms. This can be used to compile the lib for CLI maybe? It says that it can also create webassembly from js code. (It is under development though)

NectarJs github page

webpage

@Divy123
Copy link
Member

Divy123 commented Mar 4, 2019

@jywarren your thoughts on nectarJs please?

@jywarren
Copy link
Member Author

jywarren commented Mar 4, 2019 via email

@Divy123
Copy link
Member

Divy123 commented Mar 4, 2019

@jywarren in that case may I proceed with nectarJs for applying the same to our PixelManipulation.js ?

@jywarren
Copy link
Member Author

jywarren commented Mar 4, 2019 via email

@Divy123
Copy link
Member

Divy123 commented Mar 4, 2019

Excited for this.
Thanks!

@jywarren
Copy link
Member Author

jywarren commented Mar 4, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented Mar 5, 2019

Really nice walkthrough of webassembly here! https://medium.freecodecamp.org/get-started-with-webassembly-using-only-14-lines-of-javascript-b37b6aaca1e4

@Divy123
Copy link
Member

Divy123 commented Mar 8, 2019

Thanks @harshkhandeparkar .

@Divy123 Divy123 self-assigned this Mar 12, 2019
@Divy123
Copy link
Member

Divy123 commented Mar 12, 2019 •
edited

@jywarren I looked into WASM, made these two little projects,
https://github.com/Divy123/canvas-wasmvas
https://github.com/Divy123/prime-with-wasm

I suggest we can proceed with writing native C code and compiling that with emscripten and utilizing the genrated glue JS files in our code which I feel can be quite handy.

Your opinion please!!

@jywarren
Copy link
Member Author

jywarren commented Mar 12, 2019 via email

@Divy123
Copy link
Member

Divy123 commented Mar 12, 2019

Ya I looked into but perhaps feel this can be more easy to go with as then we don't have the overhead of a library.
Yes we have to keep wasm files in src/webassembly and keep it in dist!!
What do you think?

@jywarren
Copy link
Member Author

jywarren commented Mar 12, 2019 via email

@Divy123
Copy link
Member

Divy123 commented Mar 15, 2019

Sure !!

@Divy123
Copy link
Member

Divy123 commented Mar 22, 2019

@jywarren just one thing that if you can suggest apart from PixelManipulation.js what are some other areas that we can apply wasm to include in our code base like can we add to different modules as well?
Thanks!!

@jywarren
Copy link
Member Author

jywarren commented Mar 22, 2019 via email

@tech4GT
Copy link
Member

tech4GT commented Mar 24, 2019

Yes, this is absolutely true, the heavy lifting is done by pixelManipulation for almost all modules, so that should give us a significant boost in performance!

@harshkhandeparkar
Copy link
Member

What abt gpu.js which I am trying to implement for PixelManipulation? It will also work in node and will process the pixels on gpu. Can that also be compiled to wasm? It uses a node library headless-gl(will use in the future). Currently it uses the canvas api.

@Divy123
Copy link
Member

Divy123 commented Mar 24, 2019

Hi Harsh!
Can you explain a bit of your work here with gpu.js?

@harshkhandeparkar
Copy link
Member

You can have a look at issue #849. Essentially it will manipulate all the pixels at the same time in parallel on a gpu.

@jywarren
Copy link
Member Author

jywarren commented Mar 24, 2019 via email

@Divy123
Copy link
Member

Divy123 commented May 27, 2019

@jywarren @tech4GT I have been through PixelManipulation.js file and what I am able to understand that the whole code here can not be converted as there are lots of things like get-pixels and save-pixels that have to be passed to our C code. So can I get some help regarding exactly what parts of the code do we want to convert to C code or the whole PixelManipulation.js code?

Thanks !!

@Divy123 Divy123 mentioned this issue May 27, 2019
Merged
@Divy123
Copy link
Member

Divy123 commented May 27, 2019

@jywarren can you please help on this?

@jywarren
Copy link
Member Author

Also noting @MargaretAN9 is interested in this!

Hmm. I think this may be a good question to ask @harshkhandeparkar for a bit more info here. I had hoped we could take everything /between/ get-pixels and save-pixels and run that in webassembly.

@Divy123
Copy link
Member

Divy123 commented May 27, 2019 •
edited

Ok sure!! Thanks!

@MargaretAN9 can we collaborate on this??
Also this is my this week's target.
@harshkhandeparkar any insights?

@jywarren this is my task for this week. So should I start working on this or wait for Harsh's reply?

@jywarren
Copy link
Member Author

jywarren commented May 28, 2019 via email

@Divy123
Copy link
Member

Divy123 commented May 28, 2019

@jywarren can I just send here a PR with a working demo and then proceed with everyone's suggestions. Meanwhile I can make preparations for the next weeks as well to speed up the things there??

@jywarren
Copy link
Member Author

jywarren commented May 28, 2019 via email

@Divy123
Copy link
Member

Divy123 commented May 28, 2019

Thanks a lot Jeff!! 😄

@Divy123
Copy link
Member

Divy123 commented May 29, 2019 •
edited

Screenshot from 2019-05-30 01-26-27
I have made a wasm file from wasm fiddle and here is my C code

int getPixels(int , int , int );

int setPixels(int, int, int, int);

int * changePixel(int, int, int, int, int, int);

void paceOp();

void consoleLog(int);

void manipulatePixel(int width, int height, int inBrowser, int test){
  consoleLog(100);
  for (int x = 0; x < width; x++) {
        for (int y = 0; y <height; y++) {
          int* pixel =changePixel(
            getPixels(x, y, 0),
            getPixels(x, y, 1),
             getPixels(x, y, 2),
             getPixels(x, y, 3),
            x,
            y
          );
          setPixels(x, y, 0, pixel[0]);
          setPixels(x, y, 1, pixel[1]);
          setPixels(x, y, 2, pixel[2]);
          setPixels(x, y, 3, pixel[3]);

          if (!inBrowser && !test) paceOp();
        }
      }
}

I was trying to convert this part of code for testing from PixelManipulation.js file


for (var x = 0; x < pixels.shape[0]; x++) {
      for (var y = 0; y < pixels.shape[1]; y++) {
       let pixel = options.changePixel(
         pixels.get(x, y, 0),
         pixels.get(x, y, 1),
          pixels.get(x, y, 2),
             pixels.get(x, y, 3),
           x,
           y
       );

        pixels.set(x, y, 0, pixel[0]);
        pixels.set(x, y, 1, pixel[1]);
          pixels.set(x, y, 2, pixel[2]);
          pixels.set(x, y, 3, pixel[3]);

          if (!options.inBrowser && !process.env.TEST) pace.op();
         }
       }

After adding the wasm file, the code looks like:

const imports = {
         env:{
           changePixel:options.changePixel,
           getPixels: pixels.get,
           setPixels: pixels.set,
           paceOp: require('pace').op,
           consoleLog:console.log // for testing 
         }
       }
      const inBrowser = (options.inBrowser)?1:0;
      const test = (process.env.TEST)?1:0
       WebAssembly.instantiateStreaming(fetch('./program.wasm'), imports)
           .then(wasm => {
      wasm.instance.exports.manipulatePixel(pixels.shape[0],pixels.shape[1],inBrowser, test)
           });

But I think the wasm file is not getting recognized, is it some bundler issue?
As of now I have kept it in src/no_module but is there something else I need to consider??
Am I required to include it in dist?If yes, then how to? I am not accustomed to it.

@jywarren @tech4GT @harshkhandeparkar need your help here!!
Its something urgent. Please help !

@jywarren jywarren mentioned this issue May 29, 2019
Closed
@jywarren
Copy link
Member Author

Hi @Divy123 -- can you share a PR link, so we can take a look at how you've gotten it set up? Help us reproduce your error so we can help you solve it! I've also pinged folks in the weekly check-in to see if anyone there can help.

@Divy123
Copy link
Member

Divy123 commented May 29, 2019

Thanks a lot @jywarren !!
Making a PR now

@Divy123 Divy123 mentioned this issue May 29, 2019
Merged
4 tasks
@Divy123
Copy link
Member

Divy123 commented May 29, 2019

@jywarren here is the link:
#1093

Please look into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Divy123 Divy123

Labels
enhancement help wanted performance project
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Using wasm to accelerate PixelManipulation.js Divy123/image-sequencer
5 participants
@jywarren @Mridul97 @tech4GT @Divy123 @harshkhandeparkar