getUtf8Codec

function getUtf8Codec(config?): VariableSizeCodec<string>;

Returns a codec for encoding and decoding UTF-8 strings.

This codec serializes strings using UTF-8 encoding. The encoded output contains as many bytes as needed to represent the string.

Parameters

Returns

VariableSizeCodec<string>

A VariableSizeCodec<string> for encoding and decoding UTF-8 strings.

Examples

Encoding and decoding a UTF-8 string.

const codec = getUtf8Codec();
const bytes = codec.encode('hello'); // 0x68656c6c6f
const value = codec.decode(bytes);   // "hello"

Rejecting invalid UTF-8 instead of substituting the replacement character.

const codec = getUtf8Codec({ fatal: true });
codec.encode('\ud800');                // Throws: lone surrogate.
codec.decode(new Uint8Array([0xff]));  // Throws: invalid byte sequence.

Decoding losslessly, preserving null characters and a leading byte order mark.

const codec = getUtf8Codec({ ignoreBOM: true, removeNullCharacters: false });
codec.decode(new Uint8Array([0x61, 0x00, 0x62]));       // "a\u0000b"
codec.decode(new Uint8Array([0xef, 0xbb, 0xbf, 0x61])); // "\ufeffa"

Remarks

By default, invalid UTF-8 is replaced with the replacement character (U+FFFD), a leading byte order mark (U+FEFF) is stripped and null characters are stripped from decoded strings, since they are commonly used as padding in fixed-size strings. Use the fatal, ignoreBOM and removeNullCharacters options to change these behaviours. On platforms whose TextDecoder does not implement the fatal option, the bytes are validated by this package instead.

This codec does not enforce a size boundary. It will encode and decode all bytes necessary to represent the string.

If you need a fixed-size UTF-8 codec, consider using fixCodecSize.

const codec = fixCodecSize(getUtf8Codec(), 5);

If you need a size-prefixed UTF-8 codec, consider using addCodecSizePrefix.

const codec = addCodecSizePrefix(getUtf8Codec(), getU32Codec());

Separate getUtf8Encoder and getUtf8Decoder functions are available.

const bytes = getUtf8Encoder().encode('hello');
const value = getUtf8Decoder().decode(bytes);

See

• getUtf8Encoder
• getUtf8Decoder