[AVRO-3902] 64bit long support for JavaScript

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.11.3
• Component/s: javascript, js
• Assignee: Unassigned
• Reporter: Philippe Ozil
• Created: 14/Nov/23 08:02
• Updated: 15/Nov/23 13:11

Description

The latest version of avro-js cannot deserialize large numeric values that fit in a 64 bit Long. It's limited to the values that fit in the maximum safe integer in JavaScript (2^53 – 1).

Attempting to read larger values such as 1698100138468892700 triggers a deserialization error:

 Error: invalid "long": 1698100138468892700

 

The avro-js documentation provides a solution for this: specifying a custom Long type however, this documentation page is not easy to find and can easily be missed.

I understand that you may not want to change the default behavior of the library for backward compatibility reasons but could you at least mention this important limitation in the main documentation page?

 

Also, there's now a native JS solution for supporting large values in long type: BigInt. I believe that this should be the preferred approach since it doesn't introduce extra dependencies.

This is the native long-safe implementation (inspired from avsc):

const longType = avro.types.LongType.using({
  fromBuffer: (buf) => buf.readBigInt64LE(),
  toBuffer: (n) => {
    const buf = Buffer.alloc(8);
    buf.writeBigInt64LE(n);
    return buf;
  },
  fromJSON: BigInt,
  toJSON: Number,
  isValid: (n) => typeof n === 'bigint',
  compare: (n1, n2) => { return n1 === n2 ? 0 : (n1 < n2 ? -1 : 1); }
});

 

This is an alternate implementation that I wrote. It uses regular Number whenever possible and falls back to BigInt when the value exceeds the max safe value for Number. This could be a good built-in alternative for avro-js in which we avoid throwing a deserialization error.

const CUSTOM_LONG_AVRO_TYPE = avro.types.LongType.using({
    fromBuffer: (buf) => {
        const big = buf.readBigInt64LE();
        if (big > Number.MAX_SAFE_INTEGER) {
            return big;
        }
        return Number(BigInt.asIntN(64, big));
    },
    toBuffer: (n) => {
        const buf = Buffer.alloc(8);
        if (n instanceof BigInt) {
            buf.writeBigInt64LE(n);
        } else {
            buf.writeBigInt64LE(BigInt(n));
        }
        return buf;
    },
    fromJSON: BigInt,
    toJSON: Number,
    isValid: (n) => {
        const type = typeof n;
        return type === 'bigint' || type === 'number';
    },
    compare: (n1, n2) => {
        return n1 === n2 ? 0 : n1 < n2 ? -1 : 1;
    }
});