Buffer includes issue with escaped characters [closed]

Question

In the context of a babel plugin, I reading a Buffer from a file to check some content.

I'm looking specifically for the following string ɵɵfoobar escaped as \u0275\u0275foobar.

When I printing my buffer with myBuffer.toString() I can see

...

[...\u0275\u0275foobar()]
...

But when I check the content, I never get a positive. I've tried following :

• myBuffer.includes('ɵɵfoobar')
• myBuffer.includes('\u0275\u0275foobar')
• myBuffer.toString().includes('ɵɵfoobar')

Also note that myBuffer.includes('foobar()') returns true.

Any idea what I'm doing wrong ?

Answer 1

The encoding for the character you are looking for (ɵ) is 0x0275 in UTF-16 encoding, while it is 0xC9 or 0xB5 in UTF-8 encoding. Since you're looking for it as 0x0275, we know this file you've read into the buffer is encoded in UTF-16.

Node.js's Buffer.toString() accepts an 'encoding' as its first parameter. The default value of this is 'utf8'. This means that you must provide 'utf16le' as the first argument when calling the toString method (eg buffer.toString('utf16le')) if you want to be able to match against the UTF-16 encoding as \u0275. Note that Node.js only supports the little-endian variant of UTF-16.

myBuffer.includes('foobar()') returns true because all of the characters you are searching for in the string 'foobar()' are represented the same in ASCII as they are in UTF-8 (eg: 'f' is encoded as 0x66), and ASCII is a proper subset of UTF-8 (remember, we just saw UTF-8 is the default encoding of the Node.js Buffer.toString method).

If you're curious, this post on ASCII vs Unicode has some great jumping off points for some differences and encoding concepts.