Node.js v5.12.0 Documentation

Table of Contents

Buffer#

Stability: 2 - Stable

Prior to the introduction of TypedArray in ECMAScript 2015 (ES6), the JavaScript language had no mechanism for reading or manipulating streams of binary data. The Buffer class was introduced as part of the Node.js API to make it possible to interact with octet streams in the context of things like TCP streams and file system operations.

Now that TypedArray has been added in ES6, the Buffer class implements the Uint8Array API in a manner that is more optimized and suitable for Node.js' use cases.

Instances of the Buffer class are similar to arrays of integers but correspond to fixed-sized, raw memory allocations outside the V8 heap. The size of the Buffer is established when it is created and cannot be resized.

The Buffer class is a global within Node.js, making it unlikely that one would need to ever use require('buffer'). 

const buf1 = Buffer.alloc(10);
  // Creates a zero-filled Buffer of length 10.

const buf2 = Buffer.alloc(10, 1);
  // Creates a Buffer of length 10, filled with 0x01.

const buf3 = Buffer.allocUnsafe(10);
  // Creates an uninitialized buffer of length 10.
  // This is faster than calling Buffer.alloc() but the returned
  // Buffer instance might contain old data that needs to be
  // overwritten using either fill() or write().

const buf4 = Buffer.from([1,2,3]);
  // Creates a Buffer containing [01, 02, 03].

const buf5 = Buffer.from('test');
  // Creates a Buffer containing ASCII bytes [74, 65, 73, 74].

const buf6 = Buffer.from('tést', 'utf8');
  // Creates a Buffer containing UTF8 bytes [74, c3, a9, 73, 74].

Buffer.from(), Buffer.alloc(), and Buffer.allocUnsafe()#

Historically, Buffer instances have been created using the Buffer constructor function, which allocates the returned Buffer differently based on what arguments are provided:

  • Passing a number as the first argument to Buffer() (e.g. new Buffer(10)), allocates a new Buffer object of the specified size. The memory allocated for such Buffer instances is not initialized and can contain sensitive data. Such Buffer objects must be initialized manually by using either buf.fill(0) or by writing to the Buffer completely. While this behavior is intentional to improve performance, development experience has demonstrated that a more explicit distinction is required between creating a fast-but-uninitialized Buffer versus creating a slower-but-safer Buffer.
  • Passing a string, array, or Buffer as the first argument copies the passed object's data into the Buffer.
  • Passing an ArrayBuffer returns a Buffer that shares allocated memory with the given ArrayBuffer.

Because the behavior of new Buffer() changes significantly based on the type of value passed as the first argument, applications that do not properly validate the input arguments passed to new Buffer(), or that fail to appropriately initialize newly allocated Buffer content, can inadvertently introduce security and reliability issues into their code.

To make the creation of Buffer objects more reliable and less error prone, new Buffer.from(), Buffer.alloc(), and Buffer.allocUnsafe() methods have been introduced as an alternative means of creating Buffer instances.

Developers should migrate all existing uses of the new Buffer() constructors to one of these new APIs.

Buffer instances returned by Buffer.allocUnsafe(size) may be allocated off a shared internal memory pool if size is less than or equal to half Buffer.poolSize. Instances returned by Buffer.allocUnsafeSlow(size) never use the shared internal memory pool.

What makes Buffer.allocUnsafe(size) and Buffer.allocUnsafeSlow(size) "unsafe"?#

When calling Buffer.allocUnsafe() (and Buffer.allocUnsafeSlow()), the segment of allocated memory is uninitialized (it is not zeroed-out). While this design makes the allocation of memory quite fast, the allocated segment of memory might contain old data that is potentially sensitive. Using a Buffer created by Buffer.allocUnsafe() without completely overwriting the memory can allow this old data to be leaked when the Buffer memory is read.

While there are clear performance advantages to using Buffer.allocUnsafe(), extra care must be taken in order to avoid introducing security vulnerabilities into an application.

Buffers and Character Encodings#

Buffers are commonly used to represent sequences of encoded characters such as UTF8, UCS2, Base64 or even Hex-encoded data. It is possible to convert back and forth between Buffers and ordinary JavaScript string objects by using an explicit encoding method. 

const buf = Buffer.from('hello world', 'ascii');
console.log(buf.toString('hex'));
  // prints: 68656c6c6f20776f726c64
console.log(buf.toString('base64'));
  // prints: aGVsbG8gd29ybGQ=

The character encodings currently supported by Node.js include:

  • 'ascii' - for 7-bit ASCII data only. This encoding method is very fast and will strip the high bit if set.

  • 'utf8' - Multibyte encoded Unicode characters. Many web pages and other document formats use UTF-8.

  • 'utf16le' - 2 or 4 bytes, little-endian encoded Unicode characters. Surrogate pairs (U+10000 to U+10FFFF) are supported.

  • 'ucs2' - Alias of 'utf16le'.

  • 'base64' - Base64 string encoding. When creating a buffer from a string, this encoding will also correctly accept "URL and Filename Safe Alphabet" as specified in RFC 4648, Section 5.

  • 'binary' - A way of encoding the buffer into a one-byte (latin-1) encoded string. The string 'latin-1' is not supported. Instead, pass 'binary' to use 'latin-1' encoding.

  • 'hex' - Encode each byte as two hexadecimal characters.

Buffers and TypedArray#

Buffers are also Uint8Array TypedArray instances. However, there are subtle incompatibilities with the TypedArray specification in ECMAScript 2015. For instance, while ArrayBuffer#slice() creates a copy of the slice, the implementation of Buffer#slice() creates a view over the existing Buffer without copying, making Buffer#slice() far more efficient.

It is also possible to create new TypedArray instances from a Buffer with the following caveats:

  1. The Buffer object's memory is copied to the TypedArray, not shared.

  2. The Buffer object's memory is interpreted as an array of distinct elements, and not as a byte array of the target type. That is, new Uint32Array(Buffer.from([1,2,3,4])) creates a 4-element Uint32Array with elements [1,2,3,4], not a Uint32Array with a single element [0x1020304] or [0x4030201].

It is possible to create a new Buffer that shares the same allocated memory as a TypedArray instance by using the TypeArray object's .buffer property: 

const arr = new Uint16Array(2);
arr[0] = 5000;
arr[1] = 4000;

const buf1 = Buffer.from(arr); // copies the buffer
const buf2 = Buffer.from(arr.buffer); // shares the memory with arr;

console.log(buf1);
  // Prints: <Buffer 88 a0>, copied buffer has only two elements
console.log(buf2);
  // Prints: <Buffer 88 13 a0 0f>

arr[1] = 6000;
console.log(buf1);
  // Prints: <Buffer 88 a0>
console.log(buf2);
  // Prints: <Buffer 88 13 70 17>

Note that when creating a Buffer using the TypedArray's .buffer, it is possible to use only a portion of the underlying ArrayBuffer by passing in byteOffset and length parameters: 

const arr = new Uint16Array(20);
const buf = Buffer.from(arr.buffer, 0, 16);
console.log(buf.length);
  // Prints: 16

The Buffer.from() and TypedArray.from() (e.g.Uint8Array.from()) have different signatures and implementations. Specifically, the TypedArray variants accept a second argument that is a mapping function that is invoked on every element of the typed array:

  • TypedArray.from(source[, mapFn[, thisArg]])

The Buffer.from() method, however, does not support the use of a mapping function:

Buffers and ES6 iteration#

Buffers can be iterated over using the ECMAScript 2015 (ES6) for..of syntax: 

const buf = Buffer.from([1, 2, 3]);

for (var b of buf)
  console.log(b)

// Prints:
//   1
//   2
//   3

Additionally, the buf.values(), buf.keys(), and buf.entries() methods can be used to create iterators.

The --zero-fill-buffers command line option#

Node.js can be started using the --zero-fill-buffers command line option to force all newly allocated Buffer and SlowBuffer instances created using either new Buffer(size), Buffer.allocUnsafe(size), Buffer.allocUnsafeSlow(size) or new SlowBuffer(size) to be automatically zero-filled upon creation. Use of this flag changes the default behavior of these methods and can have a significant impact on performance. Use of the --zero-fill-buffers option is recommended only when absolutely necessary to enforce that newly allocated Buffer instances cannot contain potentially sensitive data. 

$ node --zero-fill-buffers
> Buffer(5);
<Buffer 00 00 00 00 00>

Class: Buffer#

The Buffer class is a global type for dealing with binary data directly. It can be constructed in a variety of ways.

new Buffer(array)#

Allocates a new Buffer using an array of octets. 

const buf = new Buffer([0x62,0x75,0x66,0x66,0x65,0x72]);
  // creates a new Buffer containing ASCII bytes
  // ['b','u','f','f','e','r']

new Buffer(buffer)#

Copies the passed buffer data onto a new Buffer instance. 

const buf1 = new Buffer('buffer');
const buf2 = new Buffer(buf1);

buf1[0] = 0x61;
console.log(buf1.toString());
  // 'auffer'
console.log(buf2.toString());
  // 'buffer' (copy is not changed)

new Buffer(arrayBuffer[, byteOffset[, length]])#

  • arrayBuffer - The .buffer property of a TypedArray or a new ArrayBuffer()
  • byteOffset <Number> Default: 0
  • length <Number> Default: arrayBuffer.length - byteOffset

When passed a reference to the .buffer property of a TypedArray instance, the newly created Buffer will share the same allocated memory as the TypedArray.

The optional byteOffset and length arguments specify a memory range within the arrayBuffer that will be shared by the Buffer. 

const arr = new Uint16Array(2);
arr[0] = 5000;
arr[1] = 4000;

const buf = new Buffer(arr.buffer); // shares the memory with arr;

console.log(buf);
  // Prints: <Buffer 88 13 a0 0f>

// changing the TypdArray changes the Buffer also
arr[1] = 6000;

console.log(buf);
  // Prints: <Buffer 88 13 70 17>

new Buffer(size)#

Allocates a new Buffer of size bytes. The size must be less than or equal to the value of require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is (2^31)-1). Otherwise, a RangeError is thrown. If a size less than 0 is specified, a zero-length Buffer will be created.

Unlike ArrayBuffers, the underlying memory for Buffer instances created in this way is not initialized. The contents of a newly created Buffer are unknown and could contain sensitive data. Use buf.fill(0) to initialize a Buffer to zeroes. 

const buf = new Buffer(5);
console.log(buf);
  // <Buffer 78 e0 82 02 01>
  // (octets will be different, every time)
buf.fill(0);
console.log(buf);
  // <Buffer 00 00 00 00 00>

new Buffer(str[, encoding])#

Creates a new Buffer containing the given JavaScript string str. If provided, the encoding parameter identifies the strings character encoding. 

const buf1 = new Buffer('this is a tést');
console.log(buf1.toString());
  // prints: this is a tést
console.log(buf1.toString('ascii'));
  // prints: this is a tC)st

const buf2 = new Buffer('7468697320697320612074c3a97374', 'hex');
console.log(buf2.toString());
  // prints: this is a tést

Class Method: Buffer.alloc(size[, fill[, encoding]])#

Allocates a new Buffer of size bytes. If fill is undefined, the Buffer will be zero-filled. 

const buf = Buffer.alloc(5);
console.log(buf);
  // <Buffer 00 00 00 00 00>

The size must be less than or equal to the value of require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is (2^31)-1). Otherwise, a RangeError is thrown. If a size less than 0 is specified, a zero-length Buffer will be created.

If fill is specified, the allocated Buffer will be initialized by calling buf.fill(fill). See [buf.fill()][] for more information. 

const buf = Buffer.alloc(5, 'a');
console.log(buf);
  // <Buffer 61 61 61 61 61>

If both fill and encoding are specified, the allocated Buffer will be initialized by calling buf.fill(fill, encoding). For example: 

const buf = Buffer.alloc(11, 'aGVsbG8gd29ybGQ=', 'base64');
console.log(buf);
  // <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>

Calling Buffer.alloc(size) can be significantly slower than the alternative Buffer.allocUnsafe(size) but ensures that the newly created Buffer instance contents will never contain sensitive data.

A TypeError will be thrown if size is not a number.

Class Method: Buffer.allocUnsafe(size)#

Allocates a new non-zero-filled Buffer of size bytes. The size must be less than or equal to the value of require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is (2^31)-1). Otherwise, a RangeError is thrown. If a size less than 0 is specified, a zero-length Buffer will be created.

The underlying memory for Buffer instances created in this way is not initialized. The contents of the newly created Buffer are unknown and may contain sensitive data. Use buf.fill(0) to initialize such Buffer instances to zeroes. 

const buf = Buffer.allocUnsafe(5);
console.log(buf);
  // <Buffer 78 e0 82 02 01>
  // (octets will be different, every time)
buf.fill(0);
console.log(buf);
  // <Buffer 00 00 00 00 00>

A TypeError will be thrown if size is not a number.

Note that the Buffer module pre-allocates an internal Buffer instance of size Buffer.poolSize that is used as a pool for the fast allocation of new Buffer instances created using Buffer.allocUnsafe(size) (and the new Buffer(size) constructor) only when size is less than or equal to Buffer.poolSize >> 1 (floor of Buffer.poolSize divided by two). The default value of Buffer.poolSize is 8192 but can be modified.

Use of this pre-allocated internal memory pool is a key difference between calling Buffer.alloc(size, fill) vs. Buffer.allocUnsafe(size).fill(fill). Specifically, Buffer.alloc(size, fill) will never use the internal Buffer pool, while Buffer.allocUnsafe(size).fill(fill) will use the internal Buffer pool if size is less than or equal to half Buffer.poolSize. The difference is subtle but can be important when an application requires the additional performance that Buffer.allocUnsafe(size) provides.

Class Method: Buffer.allocUnsafeSlow(size)#

Allocates a new non-zero-filled and non-pooled Buffer of size bytes. The size must be less than or equal to the value of require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is (2^31)-1). Otherwise, a RangeError is thrown. If a size less than 0 is specified, a zero-length Buffer will be created.

The underlying memory for Buffer instances created in this way is not initialized. The contents of the newly created Buffer are unknown and may contain sensitive data. Use buf.fill(0) to initialize such