Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FreeQueue + WASM example #255

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Prev Previous commit
Next Next commit
Add README
  • Loading branch information
DivyamAhuja committed Sep 11, 2022
commit d76236975793f68dab5b990d288195816155ad0b
8 changes: 4 additions & 4 deletions src/_data/audioworklet_data.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,10 @@
This example demonstrates how this library can be used as a channel
between a worker thread and AudioWorklet to send audio data.
href: free-queue/examples/simple-passthrough/
- title: Hello WebAssembly
description: Example of using FreeQueue with WebAssembly.
This example uses a mp3 decoding library to play load and play song
from WebAssembly.
- title: Using FreeQueue with WebAssembly
description: An example demonstrates the usage of FreeQueue with
WebAssembly. It includes an external C/C++ MP3 decoder library to
load and play music with FreeQueue on top of AudioWorklet.
href: free-queue/examples/hello-webassembly/

- title: Migration from ScriptProcessorNode
Expand Down
2 changes: 1 addition & 1 deletion src/_data/build_info.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"version":"3.0.2","revision":"d20a9e8","lastUpdated":"2022-09-04","copyrightYear":2022}
{"version":"3.0.2","revision":"a7e8c6d","lastUpdated":"2022-09-10","copyrightYear":2022}
35 changes: 35 additions & 0 deletions src/audio-worklet/free-queue/examples/hello-webassembly/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Using FreeQueue with WebAssembly

This example demonstrates how to use FreeQueue with WebAssembly.
It includes an external C/C++ MP3 decoder
([dr_mp3.h](https://github.com/mackron/dr_libs/blob/master/dr_mp3.h))
library to load and play music with FreeQueue on top of AudioWorklet.

## Building

Run `build.sh` for Linux and `build.cmd` for windows to build.
Refer to build scripts for building instructions.

### Emscripten Flags
```
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
// To tell emscriptenm to build for worker environment

// To build and load as an ES6 module
-s ENVIRONMENT=worker
-s MODULARIZE=1
-s EXPORT_NAME=ExampleModule
-s EXPORT_ES6=1

// To run main automatically after loading module.
-s INVOKE_RUN=1

// To enable pthread support
-pthread

-s EXPORTED_RUNTIME_METHODS="['callMain','ccall', 'cwrap']"
-o build/example.js

// Preload mp3 file
--preload-file moonlight.mp3

DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
```
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#include <emscripten.h>
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved

/**
* Include FreeQueue interface library and dr_mp3 library (For loading and
* decodin mp3 file).
*/
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
// Include FreeQueue interface library and dr_mp3 library (For loading and
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
// decodin mp3 file).

#define FREE_QUEUE_IMPL
#include "../../src/interface/free_queue.h"
#define DR_MP3_IMPLEMENTATION
Expand All @@ -12,7 +12,7 @@
struct FreeQueue *queue;
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
drmp3 mp3;

// Function we can call from JavaScript to get FreeQueue address in memory.
// Function to get FreeQueue address in memory from JavaScript.
EMSCRIPTEN_KEEPALIVE void *getFreeQueue()
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
{
return queue;
Expand All @@ -23,8 +23,9 @@ drmp3_uint64 framesRead = 0;

#define FRAME_SIZE 4096
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved

// buffer to store interleaved PCM frames from mp3
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
float buffer[2*FRAME_SIZE]; // 2 channels so 2 * FRAME_SIZE
// input buffer
// input buffer to store decoded mp3 channel data.
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
float* input[] = {
(float[FRAME_SIZE]){},
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
(float[FRAME_SIZE]){}
Expand All @@ -35,6 +36,7 @@ float* input[] = {
*/
EMSCRIPTEN_KEEPALIVE bool audio_loop()
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
{
// Here if the last FRAME read was pushed we read more frames from
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
if (lastFramePushed) {
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
framesRead = drmp3_read_pcm_frames_f32(&mp3, DRMP3_COUNTOF(buffer)/mp3.channels, buffer);
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
// Seperating PCM frames into 2 channels
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -44,6 +46,7 @@ EMSCRIPTEN_KEEPALIVE bool audio_loop()
}

}
// Push FRAME to queue.
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
lastFramePushed = FreeQueuePush(queue, input, framesRead);
return lastFramePushed;
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
eleventyNavigation:
key: hello-webassembly
title: Hello WebAssembly
title: Using FreeQueue with WebAssembly
parent: audio-worklet
to_root_dir: ../../../../
---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,20 @@ const RENDER_QUANTUM = 128
class SinkProcessor extends AudioWorkletProcessor {
constructor(options) {
super();
/**
* Initialize the Queue.
* We have to set prototype of queue, because queue object is passed
* using structured cloning algorithm.
* See -
* https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm
*/
// Initialize the Queue.
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
// We have to set prototype of queue, because queue object is passed
// using structured cloning algorithm.
// See -
// https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm
this.queue = options.processorOptions.queue;
Object.setPrototypeOf(this.queue, FreeQueue.prototype);
}

process(inputs, outputs) {
const output = outputs[0];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this means it only fills up a single channel (mono). Why can't we do the stereo?

Copy link
Contributor Author

@DivyamAhuja DivyamAhuja Sep 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This library just acts as queue to share data between 2 threads.
So, I think it's up to user how they use it.

inputs and outputs are array of arrays of arrays (3 dimensional arrays)

outputs: [
  output_0: [
    channel_0: [0, 0, 0.5, 0.6....],
    channel_1: [0, 0, 0.5, 0.6....]
  ],
  output_1: [
    channel_0: [0, 0, 0.5, 0.6....],
    channel_1: [0, 0, 0.5, 0.6....],
  ]
]

So, we are technically pushing data into 2 channels of 0th output in this case


/**
* Pull out render quantum frame from the queue into output.
* If failed, print "failed" to console.
*/
// Pull out render quantum frame from the queue into output.
DivyamAhuja marked this conversation as resolved.
Show resolved Hide resolved
// If failed, print "failed" to console.
const didPull = this.queue.pull(output, RENDER_QUANTUM);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps:

return this.queue.pull(output, RENDER_QUANTUM) ? true : false;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can return false here.

Then I think audio worklet will just stop 🤔

https://webaudio.github.io/web-audio-api/#callback-audioworketprocess-callback

So I think we should always return true, to keep worklet alive

didPull ? null : console.log("failed");

Expand Down