Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: fast ArrayBuffer encoding #566

Merged
merged 10 commits into from May 11, 2022
31 changes: 31 additions & 0 deletions packages/candid/src/idl.test.ts
Expand Up @@ -152,6 +152,37 @@ test('IDL encoding (tuple)', () => {
);
});

test('IDL encoding (arraybuffer)', () => {
test_(
IDL.Vec(IDL.Nat8),
new Uint8Array([0, 1, 2, 3]),
'4449444c016d7b01000400010203',
'Array of Nat8s',
);
test_(
IDL.Vec(IDL.Int8),
new Int8Array([0, 1, 2, 3]),
'4449444c016d7701000400010203',
'Array of Int8s',
);
test_(
IDL.Vec(IDL.Int16),
new Int16Array([0, 1, 2, 3, 32767, -1]),
'4449444c016d760100060000010002000300ff7fffff',
'Array of Int16s',
);
test_(
IDL.Vec(IDL.Nat64),
new BigUint64Array([BigInt(0), BigInt(1), BigInt(1) << BigInt(60), BigInt(13)]),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I didn't know BigUint64Array exists.

'4449444c016d780100040000000000000000010000000000000000000000000000100d00000000000000',
'Array of Nat64s',
);
IDL.encode([IDL.Vec(IDL.Nat8)], [new Uint8Array()]);
IDL.encode([IDL.Vec(IDL.Nat8)], [new Uint8Array(100).fill(42)]);
IDL.encode([IDL.Vec(IDL.Nat16)], [new Uint16Array(200).fill(42)]);
expect(() => IDL.encode([IDL.Vec(IDL.Int8)], [new Uint16Array(10).fill(420)])).toThrow(/Invalid vec int8 argument/);
});

test('IDL encoding (array)', () => {
// Array
test_(
Expand Down
45 changes: 41 additions & 4 deletions packages/candid/src/idl.ts
Expand Up @@ -621,7 +621,7 @@ export class FloatClass extends PrimitiveType<number> {
* Represents an IDL fixed-width Int(n)
*/
export class FixedIntClass extends PrimitiveType<bigint | number> {
constructor(private _bits: number) {
constructor(public _bits: number) {
super();
}

Expand Down Expand Up @@ -724,6 +724,10 @@ export class FixedNatClass extends PrimitiveType<bigint | number> {

/**
* Represents an IDL Array
*
* Arrays of fixed-sized nat/int type (e.g. nat8), are encoded from and decoded to TypedArrays (e.g. Uint8Array).
* Arrays of float or other non-primitive types are encoded/decoded as untyped array in Javascript.
*
* @param {Type} t
*/
export class VecClass<T> extends ConstructType<T[]> {
Expand All @@ -742,14 +746,20 @@ export class VecClass<T> extends ConstructType<T[]> {
}

public covariant(x: any): x is T[] {
return Array.isArray(x) && x.every(v => this._type.covariant(v));
// Special case for ArrayBuffer
const bits = this._type instanceof FixedNatClass ? this._type.bits : (this._type instanceof FixedIntClass ? this._type._bits : 0);
return (ArrayBuffer.isView(x) && bits == (x as any).BYTES_PER_ELEMENT * 8)
|| (Array.isArray(x) && x.every(v => this._type.covariant(v)));
}

public encodeValue(x: T[]) {
const len = lebEncode(x.length);
if (this._blobOptimization) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems _blobOptimization is not needed anymore?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is still needed if people pass plain array in.

If we remove this, then old code will probably get worse performance unless they also switch from Array to Uint8Array.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment in the code. I think eventually, people want to use typedArray for large vector anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

return concat(len, new Uint8Array(x as unknown as number[]));
}
if (ArrayBuffer.isView(x)) {
return concat(len, new Uint8Array(x.buffer));
}
const buf = new Pipe(new ArrayBuffer(len.byteLength + x.length), 0);
buf.write(len);
for (const d of x) {
Expand All @@ -773,8 +783,35 @@ export class VecClass<T> extends ConstructType<T[]> {
throw new Error('Not a vector type');
}
const len = Number(lebDecode(b));
if (this._blobOptimization) {
return [...new Uint8Array(b.read(len))] as unknown as T[];

if (this._type instanceof FixedNatClass) {
if (this._type.bits == 8) {
return new Uint8Array(b.read(len)) as unknown as T[];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a breaking change? If so, we need to document it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a doc string to VecClass.

Not sure how to add to CHANGELOG.md since there is none in this repo.

}
if (this._type.bits == 16) {
return new Uint16Array(b.read(len * 2)) as unknown as T[];
}
if (this._type.bits == 32) {
return new Uint32Array(b.read(len * 4)) as unknown as T[];
}
if (this._type.bits == 64) {
return new BigUint64Array(b.read(len * 8)) as unknown as T[];
}
}

if (this._type instanceof FixedIntClass) {
if (this._type._bits == 8) {
return new Int8Array(b.read(len)) as unknown as T[];
}
if (this._type._bits == 16) {
return new Int16Array(b.read(len * 2)) as unknown as T[];
}
if (this._type._bits == 32) {
return new Int32Array(b.read(len * 4)) as unknown as T[];
}
if (this._type._bits == 64) {
return new BigInt64Array(b.read(len * 8)) as unknown as T[];
}
}

const rets: T[] = [];
Expand Down