-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
558 lines (473 loc) · 26 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Faust 2 Web Audio</title>
<meta name="description" content="Using Emscripten to compile Faust -> Javascript">
<meta name="author" content="Myles Borins">
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<link rel="stylesheet" href="css/reveal.min.css">
<link rel="stylesheet" href="css/theme/simple.css" id="theme">
<!-- For syntax highlighting -->
<link rel="stylesheet" href="lib/css/zenburn.css">
<!-- If the query includes 'print-pdf', use the PDF print sheet -->
<script>
document.write( '<link rel="stylesheet" href="css/print/' + ( window.location.search.match( /print-pdf/gi ) ? 'pdf' : 'paper' ) + '.css" type="text/css" media="print">' );
</script>
<!--[if lt IE 9]>
<script src="lib/js/html5shiv.js"></script>
<![endif]-->
</head>
<body>
<div class="reveal">
<!-- Any section element inside of this container is displayed as a slide -->
<div class="slides">
<section>
<h1>Faust2WebAudio</h1>
<h3>Digital Signal Processing in the Browser</h3>
<p>
<small>Created by <a href="http://github.com/thealphanerd">Myles Borins</a> / <a href="http://twitter.com/the_alpha_nerd">@the_alpha_nerd</a></small>
</p>
</section>
<section>
<section data-markdown>
<script type="text/template">
##What is Faust
</script>
</section>
<section data-markdown>
<script type="text/template">
[FAUST](http://faust.grame.fr/) (Functional Audio Stream) is a functional programming language specifically designed for real-time signal processing and synthesis. FAUST targets high-performance signal processing applications and audio plug-ins for a variety of platforms and standards.
</script>
</section>
<section data-markdown>
<script type="text/template">
Simply put, Faust lets one program dsp code once in a purely functional language, and compile it to various platforms including [max/msp](http://cycling74.com/products/max/), [supercollider](http://supercollider.sourceforge.net/), [audio unit](https://en.wikipedia.org/wiki/Audio_Units), [vst](https://en.wikipedia.org/wiki/Virtual_Studio_Technology), and more.
</script>
</section>
</section>
<section>
<section data-markdown>
##What is the Web Audio Api?
</section>
<section data-markdown>
The [Web Audio Api](https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html) is a high-level JavaScript API for processing and synthesizing audio in web applications. It is currently being spearheaded by Chris Rogers at Google.
</section>
<section data-markdown>
The Web Audio API comes with a number of natively compiled audio nodes capable of doing quite a bit of advanced synthesis.
You can check out Hongchan's [WAAX](https://github.com/hoch/waax) library for an example of extensive work being done with native nodes.
</section>
<section>
<section data-markdown>
But What if you want something more?
</section>
</section>
</section>
<section>
<section data-markdown>
I present to you
##The JavaScriptNode
</section>
<section data-markdown>
The JavaScriptNode allows individuals to create their own web audio nodes in pure JavaScript. This allows individuals to extend the Web Audio Api with custom nodes.
</section>
<section data-markdown>
Web Audio Libraries such as [Flocking](flockingjs.org) by Colin Clark and [Gibber](http://www.charlie-roberts.com/gibber/) by Charlie Roberts make extensive use of the JavaScriptNode.
</section>
<section data-markdown>
========
#WARNING
========
Currently native Web Audio nodes and JavaScriptNodes don't play so nicely together, most implementations of Web Audio tend to pick one or the other.
So what do you want, stability or the bleeding edge
</section>
</section>
<section>
<section data-markdown>
##What does faust look like?
Below is an example of Noise Written in Faust
```
random = +(12345)~*(1103515245);
noise = random/2147483647.0;
process = noise * vslider("Volume[style:knob]", 0, 0, 1, 0.1);
```
</section>
<section data-markdown>
##What does a WebAudioNode look like?
Below is an example of White Noise taken from Flocking
```
flock.ugen.whiteNoise = function (inputs, output, options) {
var that = flock.ugen(inputs, output, options);
that.gen = function (numSamps) {
var out = that.output,
i;
for (i = 0; i < numSamps; i++) {
out[i] = Math.random();
}
that.mulAdd(numSamps);
};
that.onInputChanged = function () {
flock.onMulAddInputChanged(that);
};
that.onInputChanged();
return that;
};
```
</section>
</section>
<section>
<section data-markdown>
##But Doesn't Faust Already compile to web audio?
</section>
<section data-markdown>
##Sure
</section>
<section data-markdown>
##But does it work?
[Current Faust2Webaudio Noise](examples/current-noise.html)
</section>
</section>
<section>
<section data-markdown>
##Why did that break?
</section>
<section data-markdown>
There is only one answer...
</section>
<section data-markdown>
#JavaScript
</section>
</section>
<section>
<section data-markdown>
##Before we start a flame war
</section>
<section data-markdown>
#I freaking love JavaScript
</section>
<section data-markdown>
##But there are some things it can't do...
</section>
<section data-markdown>
#Like Integer Arithmetic
</section>
</section>
<section>
<section data-markdown>
##asm.js to the rescue!
[](http://asmjs.org/)
</section>
<section data-markdown>
asm.js is a strict subset of JavaScript that can be used as a low-level, efficient target language for compilers. The asm.js language provides an abstraction similar to the C/C++ virtual machine: a large binary heap with efficient loads and stores, integer and floating-point arithmetic, first-order function definitions, and function pointers.
</section>
</section>
<section>
<section data-markdown>
##What does an asm.js
##WebAudioNode look like?
An example from the asmjs Flocking Branch
```
flock.ugen.asmSin.module = function (stdlib, foreign, heap) {
"use asm";
var sin = stdlib.Math.sin;
var pi = 3.14159;
var out = new stdlib.Float32Array(heap);
function gen (numSamps, freq, phaseOffset, mul, add, sampleRate, phase) {
numSamps = numSamps|0;
freq = +freq;
phaseOffset = +phaseOffset;
mul = +mul;
add = +add;
sampleRate = +sampleRate;
phase = +phase;
var i = 0;
for (; (i | 0) < (numSamps | 0); i = i + 1 | 0) {
out[i >> 2] = +(sin(phase + phaseOffset) * mul + add);
phase = +(phase + (freq / sampleRate * pi * 2.0));
}
return +phase;
}
return {
gen: gen
};
};
```
</section>
</section>
<section>
<section data-markdown>
##So this is all great, but I don't want to hand roll asm.js like a sucker...
</section>
</section>
<section>
<section data-markdown>
##Introducing
[](http://emscripten.org/)
</section>
<section data-markdown>
Emscripten is an LLVM to JavaScript compiler. It takes LLVM bitcode (which can be generated from C/C++ using Clang, or any other language that can be converted into LLVM bitcode) and compiles that into JavaScript, which can be run on the web (or anywhere else JavaScript can run).
</section>
</section>
<section>
<section data-markdown>
##Faust -> C++
So using the faust compiler (specifically the faust2-asmjs branch) we can compile from [faust](https://gist.github.com/TheAlphaNerd/5710239#file-noise-dsp) to [C++](https://gist.github.com/TheAlphaNerd/5710239#file-noise-emscripten-cpp) with the following command
```
faust -a minimal.cpp -i -uim -cn Noise \
dsp/noise.dsp -o cpp/faust-noise.cpp
```
</section>
<section data-markdown>
In order to get access to the verious parts of a C++ class via emscripten we need to write a simple wrapper on top of our Noise class.
```
// Adapted From https://gist.github.com/camupod/5640386
// compile using "C" linkage to avoid name obfuscation
extern "C" {
float** fInChannel;
float** fOutChannel;
int numInputs;
int numOutputs;
//constructor
void *NOISE_constructor(int samplingFreq, int bufferSize) {
// Make a new noise object
Noise* n = new Noise();
// Init it with samplingFreq supplied... should we give a sample size here too?
n->init(samplingFreq);
// Lets get this once so we don't need to keep calculating every call to compute
numInputs = n->getNumInputs();
numOutputs = n->getNumOutputs();
// This Needs to be dealt with... curently only able to return a single buffer... rather
// Way too mono...
// This is due to the Channels not being properly initialized... need to look at how
// Other architecture fiels do this a bit better
for (int i = 0; i < numInputs; i++) {
fInChannel[i] = new float[bufferSize];
}
for (int i = 0; i < numOutputs; i++) {
fOutChannel[i] = new float[bufferSize];
}
return n;
}
FAUSTFLOAT *NOISE_compute(Noise *n, int count) {
n->compute(count, fInChannel, fOutChannel);
// Returning due to problem as mentioend above...
return fOutChannel[0];
}
int NOISE_getNumInputs(Noise *n){
return n->getNumInputs();
}
int NOISE_getNumOutputs(Noise *n){
return n->getNumOutputs();
}
void NOISE_destructor(Noise *n) {
for (int i = 0; i < numInputs; i++) {
delete fInChannel[i];
}
for (int i = 0; i < numOutputs; i++) {
delete fOutChannel[i];
}
delete n;
}
}
// Number of channels gotten from getNumInputs / getNumOutputs
// fOutChannel[i] = (float*)ioData->mBuffers[i].mData;
```
</section>
<section data-markdown>
We can then compile the resulting C++ file to asm.js using emscripten with the following command
```
emcc cpp/faust-noise.cpp -o js/faust-noise-temp.js \
-s EXPORTED_FUNCTIONS="['_NOISE_constructor','_NOISE_destructor', \
'_NOISE_compute', '_NOISE_getNumInputs', '_NOISE_getNumOutputs']"
```
Which creates this [JavaScript file](https://gist.github.com/TheAlphaNerd/5710239#file-noise-emscripten-js)
</section>
<section data-markdown>
Finally we apply the following JavaScript wrapper in order to break out the functions we wrapped earlier in C++
```
// Adapted From https://gist.github.com/camupod/5640386
var faust = faust || {};
(function() {
faust.context = new webkitAudioContext();
var NOISE_constructor = Module.cwrap('NOISE_constructor', 'number', 'number');
var NOISE_destructor = Module.cwrap('NOISE_destructor', null, ['number']);
var NOISE_compute = Module.cwrap('NOISE_compute', ['number'], ['number', 'number']);
var NOISE_getNumInputs = Module.cwrap('NOISE_getNumInputs', 'number', []);
var NOISE_getNumOutputs = Module.cwrap('NOISE_getNumOutputs', 'number', []);
faust.noise = function () {
that = {};
that.ptr = NOISE_constructor(faust.context.sampleRate);
that.getNumInputs = function () {
return NOISE_getNumInputs();
}
that.getNumOutputs = function () {
return NOISE_getNumOutputs();
}
that.compute = function (count) {
return NOISE_compute(that.ptr, count);
}
that.destroy = function () {
NOISE_destructor(that.ptr);
}
return that;
}
}())
noise = faust.noise();
```
</section>
<section data-markdown>
#Phew...
</section>
</section>
<section>
<section data-markdown>
#But does it work?
</section>
<section data-markdown>
#Sure,
#[take a listen](demo/index.html)
</section>
<section data-markdown>
##But it is unnecessarily complicated
</section>
</section>
<section>
<section data-markdown>
###Managing memory in C++ while maintaining persistence when compiled to JavaScript can be tricky.
</section>
<section data-markdown>
###Specifically because faust represents inputs and outputs as a float **
</section>
<section data-markdown>
That means the input and output variables are in fact a pointer to an array of channels
</section>
<section data-markdown>
Each channel in the array is a pointer to a buffer
</section>
<section data-markdown>
Each buffer is an array consisting of the samples represented as a floating point number between -1 and 1.
</section>
<section data-markdown>
</section>
</section>
<section>
<section data-markdown>
##This seems unnecessarily complicated
</section>
<section data-markdown>
##Why should we be spending time managing these pointers?
</section>
<section data-markdown>
##How are we supposed to deal with the pointers in JavaScript land without creating lots of garbage?
</section>
</section>
<section>
<section data-markdown>
##But didn't I say Faust is Functional?
</section>
<section data-markdown>
##Indeed it is
</section>
<section data-markdown>
###So why are am I going from
Functional -> Object Oriented -> Functional
</section>
<section data-markdown>
</section>
</section>
<section>
<section data-markdown>
##Emscripten comes with its own virtual machine as well as a number of compilation optimizations
</section>
<section data-markdown>
##Simply put, a bunch of performance for free!
</section>
<section data-markdown>
##That made it a tempting starting point
</section>
<section data-markdown>
##As a primary research goal is to create very efficient web audio based unit generators
</section>>
</section>
<section>
<section data-markdown>
#Moving Forward
##Faust -> asm.js directly
</section>
<section data-markdown>
###Prior to being compiled to C++, faust is compiled to Faust Intermediate Representation (FIR)
It is in this form that Faust is compiled to specific language architectures such as C++ and java.
</section>
<section data-markdown>
##It is from FIR that we can compile directly to asm.js
</section>
</section>
<section>
<section data-markdown>
##While Faust -> JavaScript directly might be more elegant
</section>
<section data-markdown>
###There is no guarantee it will be faster
</section>
<section data-markdown>
###As such we need to continue to develop both and compare via benchmarks to see which method ends up being more efficient
</section>
</section>
<section>
<section data-markdown>
##In Conclusion
Faust in the browser would be awesome
</section>
<section data-markdown>
##But we are not quite there yet.
</section>
<section data-markdown>
###Even if we solve the compilation issues, we will still run into the ceiling of the browser due to all processes running in a single thread
</section>
<section data-markdown>
##But if WebAudio is going to have a future, it is important to be able to extend it.
</section>
</section>
<section data-markdown>
#Find the source on [Github](https://github.com/TheAlphaNerd/faust2webaudio)
</section>
<section>
<h1>Thank You</h1>
<h3>by Myles Borins</h3>
</section>
</div>
</div>
<script src="lib/js/head.min.js"></script>
<script src="js/reveal.min.js"></script>
<script>
// Full list of configuration options available here:
// https://github.com/hakimel/reveal.js#configuration
Reveal.initialize({
controls: true,
progress: true,
history: true,
center: true,
theme: Reveal.getQueryHash().theme, // available themes are in /css/theme
transition: Reveal.getQueryHash().transition || 'default', // default/cube/page/concave/zoom/linear/fade/none
// Optional libraries used to extend on reveal.js
dependencies: [
{ src: 'lib/js/classList.js', condition: function() { return !document.body.classList; } },
{ src: 'plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
{ src: 'plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
{ src: 'plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } },
{ src: 'plugin/zoom-js/zoom.js', async: true, condition: function() { return !!document.body.classList; } },
{ src: 'plugin/notes/notes.js', async: true, condition: function() { return !!document.body.classList; } }
// { src: 'plugin/search/search.js', async: true, condition: function() { return !!document.body.classList; } }
// { src: 'plugin/remotes/remotes.js', async: true, condition: function() { return !!document.body.classList; } }
]
});
</script>
</body>
</html>