Node.js Buffer (Buffer Zone)

JavaScript itself only has a string data type, not a binary data type.

However, when dealing with streams like TCP or file streams, binary data must be used. Therefore, in Node.js, a Buffer class is defined to create a dedicated area for storing binary data.

In Node.js, the Buffer class is a core library released with the Node kernel. The Buffer library brings a method for storing raw data to Node.js, allowing it to handle binary data. Whenever data needs to be processed in Node.js for I/O operations, the Buffer library may be used. Raw data is stored in instances of the Buffer class. A Buffer is similar to an array of integers, but it corresponds to a block of raw memory outside the V8 heap.

>

Before v6.0, Buffer objects were directly created using the new Buffer() constructor to create instance objects. However, Buffer has extensive memory access rights and can directly capture some sensitive information. Therefore, after v6.0, the official documentation recommends using the Buffer.from() interface to create Buffer objects.

Buffer and Character Encoding

Buffer instances are generally used to represent sequences of encoded characters, such as UTF-8, UCS2, Base64, or hexadecimal encoded data. By using explicit character encodings, conversions can be made between Buffer instances and ordinary JavaScript strings.

const buf = Buffer.from('tutorialpro', 'ascii');

// Outputs 72756e6f6f62
console.log(buf.toString('hex'));

// Outputs cnVub29i
console.log(buf.toString('base64'));

Node.js currently supports the following character encodings:

-

ascii - Supports only 7-bit ASCII data. This encoding is very fast if the high bit is set to be removed.

-

utf8 - Multi-byte encoded Unicode characters. Many web pages and other document formats use UTF-8.

-

utf16le - 2 or 4 bytes, little-endian encoded Unicode characters. Supports surrogate pairs (U+10000 to U+10FFFF).

-

ucs2 - Alias for utf16le.

-

base64 - Base64 encoding.

-

latin1 - A method of encoding Buffer into a one-byte encoded string.

-

binary - Alias for latin1.

-

hex - Encodes each byte as two hexadecimal characters.

Creating a Buffer Class

Buffer provides the following API to create Buffer classes:

-Buffer.alloc(size[, fill[, encoding]]): Returns a Buffer instance of the specified size. If fill is not set, it defaults to filling with 0.

-Buffer.allocUnsafe(size): Returns a Buffer instance of the specified size, but it is not initialized, so it may contain sensitive data.

-Buffer.allocUnsafeSlow(size)

-Buffer.from(array): Returns a new Buffer instance initialized with the values of the array (elements in the array can only be numbers, otherwise they will be automatically overwritten with 0).

-Buffer.from(arrayBuffer[, byteOffset[, length]]): Returns a new Buffer that shares the same memory as the given ArrayBuffer.

-Buffer.from(buffer): Copies the data from the passed Buffer instance and returns a new Buffer instance.

-Buffer.from(string[, encoding]): Returns a new Buffer instance initialized with the value of the string.

// Creates a Buffer of length 10, filled with 0.
const buf1 = Buffer.alloc(10);

// Creates a Buffer of length 10, filled with 0x1.
const buf2 = Buffer.alloc(10, 1);

// Creates a Buffer of length 10, uninitialized.
// This method is faster than calling Buffer.alloc(),
// but the returned Buffer instance may contain old data,
// so it needs to be overwritten with fill() or write().
const buf3 = Buffer.allocUnsafe(10);

// Creates a Buffer containing [0x1, 0x2, 0x3].
const buf4 = Buffer.from([1, 2, 3]);

// Create a Buffer containing the UTF-8 bytes [0x74, 0xc3, 0xa9, 0x73, 0x74].
const buf5 = Buffer.from('tést');

// Create a Buffer containing the Latin-1 bytes [0x74, 0xe9, 0x73, 0x74].
const buf6 = Buffer.from('tést', 'latin1');

Writing to Buffers

Syntax

The syntax for writing to a Node buffer is as follows:

buf.write(string[, offset[, length]][, encoding])

Parameters

The parameters are described as follows:

• string - The string to be written to the buffer.
• offset - The index at which to start writing in the buffer, defaults to 0.
• length - The number of bytes to write, defaults to buffer.length.
• encoding - The encoding to use. Defaults to 'utf8'.

The string is written to the buffer at the offset position according to the specified encoding. The length parameter is the number of bytes to write. If the buffer does not have enough space to hold the entire string, only a part of the string will be written. Partially decoded characters will not be written.

Return Value

Returns the actual number of bytes written. If the buffer does not have enough space, only a part of the string will be written.

Example

buf = Buffer.alloc(256);
len = buf.write("www.tutorialpro.org");

console.log("Bytes written : "+  len);

Executing the above code will output:

$ node main.js
Bytes written : 14

Reading from Buffers

Syntax

The syntax for reading data from a Node buffer is as follows:

buf.toString([encoding[, start[, end]]])

Parameters

The parameters are described as follows:

• encoding - The encoding to use. Defaults to 'utf8'.
• start - The index to start reading from, defaults to 0.
• end - The end position, defaults to the end of the buffer.

Return Value

Returns the buffer data decoded as a string using the specified encoding.

Example

buf = Buffer.alloc(26);
for (var i = 0 ; i < 26 ; i++) {
  buf[i] = i + 97;
}

console.log( buf.toString('ascii'));       // Output: abcdefghijklmnopqrstuvwxyz
console.log( buf.toString('ascii',0,5));   // Output: abcde using 'ascii' encoding
console.log( buf.toString('utf8',0,5));    // Output: abcde using 'utf8' encoding
console.log( buf.toString(undefined,0,5)); // Output: abcde using default 'utf8' encoding

Executing the above code will output:

$ node main.js
abcdefghijklmnopqrstuvwxyz
abcde
abcde
abcde

Converting Buffer to JSON Object

Syntax

The syntax for converting a Node Buffer to a JSON object is as follows:

buf.toJSON()

When stringifying a Buffer instance, JSON.stringify() implicitly calls this toJSON().

Return Value

Returns a JSON object.

Example

const buf = Buffer.from([0x1, 0x2, 0x3, 0x4, 0x5]);
const json = JSON.stringify(buf);

// Output: {"type":"Buffer","data":[1,2,3,4,5]}
console.log(json);

const copy = JSON.parse(json, (key, value) => {
  return value && value.type === 'Buffer' ?
    Buffer.from(value.data) :
    value;
});

// Output: <Buffer 01 02 03 04 05>
console.log(copy);

Executing the above code will output:

{"type":"Buffer","data":[1,2,3,4,5]}
<Buffer 01 02 03 04 05>

Buffer Concatenation

Syntax

Buffer.concat(list[, totalLength])

Parameters

The parameters are described as follows:

• list - Array list of Buffer objects to concatenate.
• totalLength - Specifies the total length of the concatenated Buffer object.

Return Value

Returns a new Buffer object that is the result of concatenating multiple members.

Example

var buffer1 = Buffer.from(('tutorialpro.org'));
var buffer2 = Buffer.from(('www.tutorialpro.org'));
var buffer3 = Buffer.concat([buffer1, buffer2]);
console.log("buffer3 content: " + buffer3.toString());

Executing the above code will output:

buffer3 content: tutorialpro.orgwww.tutorialpro.org

Buffer Comparison

Syntax

The function syntax for comparing Node buffers is as follows, introduced in Node.js v0.12.2:

buf.compare(otherBuffer);

Parameters

The parameters are described as follows:

• otherBuffer - Another Buffer object to compare with the buf object.

Return Value

Returns a number indicating whether buf comes before, after, or is the same as otherBuffer.

Example

var buffer1 = Buffer.from('ABC');
var buffer2 = Buffer.from('ABCD');
var result = buffer1.compare(buffer2);

if (result < 0) {
   console.log(buffer1 + " comes before " + buffer2);
} else if (result == 0) {
   console.log(buffer1 + " is the same as " + buffer2);
} else {
   console.log(buffer1 + " comes after " + buffer2);
}

Executing the above code will output:

ABC comes before ABCD

Copying Buffer

Syntax

The syntax for copying Node buffers is as follows:

buf.copy(targetBuffer[, targetStart[, sourceStart[, sourceEnd]]])

Parameters

The parameters are described as follows:

• targetBuffer - Buffer object to copy to.
• targetStart - Number, optional, default: 0
• sourceStart - Number, optional, default: 0
• sourceEnd - Number, optional, default: buffer.length

Return Value

No return value.

Example

var buf1 = Buffer.from('abcdefghijkl');
var buf2 = Buffer.from('tutorialpro');

// Insert buf2 into buf1 at the specified position
buf2.copy(buf1, 2);

console.log(buf1.toString());

Executing the above code will output:

abtutorialproijkl

Slicing Buffer

Syntax

The syntax for slicing Node buffers is as follows:

buf.slice([start[, end]])

Parameters

The parameters are described as follows:

• start - Number, optional, default: 0
• end - Number, optional, default: buffer.length

Return Value

Returns a new buffer that references the same memory as the old buffer, but cropped to the index start to end.

Example

var buffer1 = Buffer.from('tutorialpro');
// Slice the buffer
var buffer2 = buffer1.slice(0, 2);
console.log("buffer2 content: " + buffer2.toString());

Executing the above code will output:

buffer2 content: tu

Buffer Length

Syntax

The syntax for calculating the length of a Node buffer is as follows:

buf.length;

Return Value

Returns the memory length occupied by the Buffer object.

Example

var buffer1 = Buffer.from('tutorialpro');
console.log("Buffer length: " + buffer1.length);

Executing the above code will output:

Buffer length: 11

var buffer = Buffer.from('www.tutorialpro.org');
// Buffer length
console.log("buffer length: " + buffer.length);

Executing the above code will output:

buffer length: 14

Method Reference Manual

Below are listed commonly used methods of the Node.js Buffer module (note that some methods are not available in older versions):

Writes value to the buffer at the specified offset. Note that value must be a valid unsigned 8-bit integer.
If noAssert is set to true, it will not validate the offset. This means that value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Do not use this unless you are very sure about this parameter. The default is false.
| 36 | buf.writeUInt16LE(value, offset[, noAssert])
Writes value to the buffer at the specified offset and endian format. Note that value must be a valid unsigned 16-bit integer, little-endian.
If noAssert is set to true, it will not validate value and offset. This means that value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Do not use this unless you are very sure about this parameter. The default is false.
| 37 | buf.writeUInt16BE(value, offset[, noAssert])
Writes value to the buffer at the specified offset and endian format. Note that value must be a valid unsigned 16-bit integer, big-endian.
If noAssert is set to true, it will not validate value and offset. This means that value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Do not use this unless you are very sure about this parameter. The default is false.
| 38 | buf.writeUInt32LE(value, offset[, noAssert])
Writes value to the buffer at the specified offset and endian format (LITTLE-ENDIAN: little-endian). Note that value must be a valid unsigned 32-bit integer, little-endian.
If noAssert is set to true, it will not validate value and offset. This means that value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Do not use this unless you are very sure about this parameter. The default is false.
| 39 | buf.writeUInt32BE(value, offset[, noAssert])
Writes value to the buffer at the specified offset and endian format (Big-Endian: big-endian). Note that value must be a valid unsigned 32-bit integer.
If noAssert is set to true, it will not validate value and offset. This means that value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Do not use this unless you are very sure about this parameter. The default is false.
| 40 | buf.writeInt8(value, offset[, noAssert])
| 41 | buf.writeInt16LE(value, offset[, noAssert])
Writes value to the buffer at the specified offset and endian format. Note that value must be a valid signed 16-bit integer.
If noAssert is set to true, it will not validate value and offset. This means that value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Do not use this unless you are very sure about this parameter. The default is false. | 42 | buf.writeInt16BE(value, offset[, noAssert]) <br>Writes value to the buffer at the specified offset with specified endian format. Note: value must be a valid signed 16-bit integer. <br>If noAssert is true, it will not validate the value and offset parameters. This means value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Avoid using this unless you are very confident. Defaults to false. | | 43 | buf.writeInt32LE(value, offset[, noAssert]) <br>Writes value to the buffer at the specified offset with specified endian format. Note: value must be a valid signed 32-bit integer. <br>If noAssert is true, it will not validate the value and offset parameters. This means value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Avoid using this unless you are very confident. Defaults to false. | | 44 | buf.writeInt32BE(value, offset[, noAssert]) <br>Writes value to the buffer at the specified offset with specified endian format. Note: value must be a valid signed 32-bit integer. <br>If noAssert is true, it will not validate the value and offset parameters. This means value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Avoid using this unless you are very confident. Defaults to false. | | 45 | buf.writeFloatLE(value, offset[, noAssert]) <br>Writes value to the buffer at the specified offset with specified endian format. Note: The result is unpredictable when value is not a 32-bit float. <br>If noAssert is true, it will not validate the value and offset parameters. This means value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Avoid using this unless you are very confident. Defaults to false. | | 46 | buf.writeFloatBE(value, offset[, noAssert]) <br>Writes value to the buffer at the specified offset with specified endian format. Note: The result is unpredictable when value is not a 32-bit float. <br>If noAssert is true, it will not validate the value and offset parameters. This means value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Avoid using this unless you are very confident. Defaults to false. | | 47 | buf.writeDoubleLE(value, offset[, noAssert]) <br>Writes value to the buffer at the specified offset with specified endian format. Note: value must be a valid 64-bit double. <br>If noAssert is true, it will not validate the value and offset parameters. This means value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Avoid using this unless you are very confident. Defaults to false. | | 48 | buf.writeDoubleBE(value, offset[, noAssert]) <br>Writes value to the buffer at the specified offset with specified endian format. Note: value must be a valid 64-bit double. <br>If noAssert is true, it will not validate the value and offset parameters. This means value might be too large, or offset might exceed the end of the buffer, causing value to be discarded. Avoid using this unless you are very confident. Defaults to false. | | 49 | buf.fill(value[, offset][, end]) <br>Fills the buffer with the specified value. If offset is not provided (default is 0) and end is not provided (default is buffer.length), the entire buffer will be filled.