+ 
+
+
+ 
+ 
+ 
+ 
+ 
+
+
+ Consider supporting us on our Open Collective: +
+
+
+
+
+---
+
+**You might be looking for the [v2 docs](https://github.com/node-fetch/node-fetch/tree/2.x#readme)**
@@ -20,32 +27,53 @@ A light-weight module that brings `window.fetch` to Node.js
- [Difference from client-side fetch](#difference-from-client-side-fetch)
- [Installation](#installation)
- [Loading and configuring the module](#loading-and-configuring-the-module)
+- [Upgrading](#upgrading)
- [Common Usage](#common-usage)
- - [Plain text or HTML](#plain-text-or-html)
- - [JSON](#json)
- - [Simple Post](#simple-post)
- - [Post with JSON](#post-with-json)
- - [Post with form parameters](#post-with-form-parameters)
- - [Handling exceptions](#handling-exceptions)
- - [Handling client and server errors](#handling-client-and-server-errors)
+ - [Plain text or HTML](#plain-text-or-html)
+ - [JSON](#json)
+ - [Simple Post](#simple-post)
+ - [Post with JSON](#post-with-json)
+ - [Post with form parameters](#post-with-form-parameters)
+ - [Handling exceptions](#handling-exceptions)
+ - [Handling client and server errors](#handling-client-and-server-errors)
+ - [Handling cookies](#handling-cookies)
- [Advanced Usage](#advanced-usage)
- - [Streams](#streams)
- - [Buffer](#buffer)
- - [Accessing Headers and other Meta data](#accessing-headers-and-other-meta-data)
- - [Extract Set-Cookie Header](#extract-set-cookie-header)
- - [Post data using a file stream](#post-data-using-a-file-stream)
- - [Post with form-data (detect multipart)](#post-with-form-data-detect-multipart)
- - [Request cancellation with AbortSignal](#request-cancellation-with-abortsignal)
+ - [Streams](#streams)
+ - [Accessing Headers and other Metadata](#accessing-headers-and-other-metadata)
+ - [Extract Set-Cookie Header](#extract-set-cookie-header)
+ - [Post data using a file](#post-data-using-a-file)
+ - [Request cancellation with AbortSignal](#request-cancellation-with-abortsignal)
- [API](#api)
- - [fetch(url[, options])](#fetchurl-options)
- - [Options](#options)
- - [Class: Request](#class-request)
- - [Class: Response](#class-response)
- - [Class: Headers](#class-headers)
- - [Interface: Body](#interface-body)
- - [Class: FetchError](#class-fetcherror)
-- [License](#license)
+ - [fetch(url[, options])](#fetchurl-options)
+ - [Options](#options)
+ - [Default Headers](#default-headers)
+ - [Custom Agent](#custom-agent)
+ - [Custom highWaterMark](#custom-highwatermark)
+ - [Insecure HTTP Parser](#insecure-http-parser)
+ - [Class: Request](#class-request)
+ - [new Request(input[, options])](#new-requestinput-options)
+ - [Class: Response](#class-response)
+ - [new Response([body[, options]])](#new-responsebody-options)
+ - [response.ok](#responseok)
+ - [response.redirected](#responseredirected)
+ - [response.type](#responsetype)
+ - [Class: Headers](#class-headers)
+ - [new Headers([init])](#new-headersinit)
+ - [Interface: Body](#interface-body)
+ - [body.body](#bodybody)
+ - [body.bodyUsed](#bodybodyused)
+ - [body.arrayBuffer()](#bodyarraybuffer)
+ - [body.blob()](#bodyblob)
+ - [body.formData()](#formdata)
+ - [body.json()](#bodyjson)
+ - [body.text()](#bodytext)
+ - [Class: FetchError](#class-fetcherror)
+ - [Class: AbortError](#class-aborterror)
+- [TypeScript](#typescript)
- [Acknowledgement](#acknowledgement)
+- [Team](#team)
+ - [Former](#former)
+- [License](#license)
@@ -53,253 +81,418 @@ A light-weight module that brings `window.fetch` to Node.js
Instead of implementing `XMLHttpRequest` in Node.js to run browser-specific [Fetch polyfill](https://github.com/github/fetch), why not go from native `http` to `fetch` API directly? Hence, `node-fetch`, minimal code for a `window.fetch` compatible API on Node.js runtime.
-See Matt Andrews' [isomorphic-fetch](https://github.com/matthew-andrews/isomorphic-fetch) or Leonardo Quixada's [cross-fetch](https://github.com/lquixada/cross-fetch) for isomorphic usage (exports `node-fetch` for server-side, `whatwg-fetch` for client-side).
+See Jason Miller's [isomorphic-unfetch](https://www.npmjs.com/package/isomorphic-unfetch) or Leonardo Quixada's [cross-fetch](https://github.com/lquixada/cross-fetch) for isomorphic usage (exports `node-fetch` for server-side, `whatwg-fetch` for client-side).
## Features
- Stay consistent with `window.fetch` API.
- Make conscious trade-off when following [WHATWG fetch spec][whatwg-fetch] and [stream spec](https://streams.spec.whatwg.org/) implementation details, document known differences.
-- Use native promise but allow substituting it with [insert your favorite promise library].
-- Use native Node streams for body on both request and response.
-- Decode content encoding (gzip/deflate) properly and convert string output (such as `res.text()` and `res.json()`) to UTF-8 automatically.
-- Useful extensions such as timeout, redirect limit, response size limit, [explicit errors](ERROR-HANDLING.md) for troubleshooting.
+- Use native promise and async functions.
+- Use native Node streams for body, on both request and response.
+- Decode content encoding (gzip/deflate/brotli) properly, and convert string output (such as `res.text()` and `res.json()`) to UTF-8 automatically.
+- Useful extensions such as redirect limit, response size limit, [explicit errors][error-handling.md] for troubleshooting.
## Difference from client-side fetch
-- See [Known Differences](LIMITS.md) for details.
+- See known differences:
+ - [As of v3.x](docs/v3-LIMITS.md)
+ - [As of v2.x](docs/v2-LIMITS.md)
- If you happen to use a missing feature that `window.fetch` offers, feel free to open an issue.
- Pull requests are welcomed too!
## Installation
-Current stable release (`2.x`)
+Current stable release (`3.x`) requires at least Node.js 12.20.0.
```sh
-$ npm install node-fetch
+npm install node-fetch
```
## Loading and configuring the module
-We suggest you load the module via `require` until the stabilization of ES modules in node:
+
+### ES Modules (ESM)
+
```js
-const fetch = require('node-fetch');
+import fetch from 'node-fetch';
```
-If you are using a Promise library other than native, set it through `fetch.Promise`:
+### CommonJS
+
+`node-fetch` from v3 is an ESM-only module - you are not able to import it with `require()`.
+
+If you cannot switch to ESM, please use v2 which remains compatible with CommonJS. Critical bug fixes will continue to be published for v2.
+
+```sh
+npm install node-fetch@2
+```
+
+Alternatively, you can use the async `import()` function from CommonJS to load `node-fetch` asynchronously:
+
```js
-const Bluebird = require('bluebird');
+// mod.cjs
+const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));
+```
+
+### Providing global access
+
+To use `fetch()` without importing it, you can patch the `global` object in node:
-fetch.Promise = Bluebird;
+```js
+// fetch-polyfill.js
+import fetch, {
+ Blob,
+ blobFrom,
+ blobFromSync,
+ File,
+ fileFrom,
+ fileFromSync,
+ FormData,
+ Headers,
+ Request,
+ Response,
+} from 'node-fetch'
+
+if (!globalThis.fetch) {
+ globalThis.fetch = fetch
+ globalThis.Headers = Headers
+ globalThis.Request = Request
+ globalThis.Response = Response
+}
+
+// index.js
+import './fetch-polyfill'
+
+// ...
```
+## Upgrading
+
+Using an old version of node-fetch? Check out the following files:
+
+- [2.x to 3.x upgrade guide](docs/v3-UPGRADE-GUIDE.md)
+- [1.x to 2.x upgrade guide](docs/v2-UPGRADE-GUIDE.md)
+- [Changelog](https://github.com/node-fetch/node-fetch/releases)
+
## Common Usage
-NOTE: The documentation below is up-to-date with `2.x` releases; see the [`1.x` readme](https://github.com/bitinn/node-fetch/blob/1.x/README.md), [changelog](https://github.com/bitinn/node-fetch/blob/1.x/CHANGELOG.md) and [2.x upgrade guide](UPGRADE-GUIDE.md) for the differences.
+NOTE: The documentation below is up-to-date with `3.x` releases, if you are using an older version, please check how to [upgrade](#upgrading).
+
+### Plain text or HTML
-#### Plain text or HTML
```js
-fetch('https://github.com/')
- .then(res => res.text())
- .then(body => console.log(body));
+import fetch from 'node-fetch';
+
+const response = await fetch('https://github.com/');
+const body = await response.text();
+
+console.log(body);
```
-#### JSON
+### JSON
```js
+import fetch from 'node-fetch';
+
+const response = await fetch('https://api.github.com/users/github');
+const data = await response.json();
-fetch('https://api.github.com/users/github')
- .then(res => res.json())
- .then(json => console.log(json));
+console.log(data);
```
-#### Simple Post
+### Simple Post
+
```js
-fetch('https://httpbin.org/post', { method: 'POST', body: 'a=1' })
- .then(res => res.json()) // expecting a json response
- .then(json => console.log(json));
+import fetch from 'node-fetch';
+
+const response = await fetch('https://httpbin.org/post', {method: 'POST', body: 'a=1'});
+const data = await response.json();
+
+console.log(data);
```
-#### Post with JSON
+### Post with JSON
```js
-const body = { a: 1 };
-
-fetch('https://httpbin.org/post', {
- method: 'post',
- body: JSON.stringify(body),
- headers: { 'Content-Type': 'application/json' },
- })
- .then(res => res.json())
- .then(json => console.log(json));
+import fetch from 'node-fetch';
+
+const body = {a: 1};
+
+const response = await fetch('https://httpbin.org/post', {
+ method: 'post',
+ body: JSON.stringify(body),
+ headers: {'Content-Type': 'application/json'}
+});
+const data = await response.json();
+
+console.log(data);
```
-#### Post with form parameters
-`URLSearchParams` is available in Node.js as of v7.5.0. See [official documentation](https://nodejs.org/api/url.html#url_class_urlsearchparams) for more usage methods.
+### Post with form parameters
+
+`URLSearchParams` is available on the global object in Node.js as of v10.0.0. See [official documentation](https://nodejs.org/api/url.html#url_class_urlsearchparams) for more usage methods.
NOTE: The `Content-Type` header is only set automatically to `x-www-form-urlencoded` when an instance of `URLSearchParams` is given as such:
```js
-const { URLSearchParams } = require('url');
+import fetch from 'node-fetch';
const params = new URLSearchParams();
params.append('a', 1);
-fetch('https://httpbin.org/post', { method: 'POST', body: params })
- .then(res => res.json())
- .then(json => console.log(json));
+const response = await fetch('https://httpbin.org/post', {method: 'POST', body: params});
+const data = await response.json();
+
+console.log(data);
```
-#### Handling exceptions
-NOTE: 3xx-5xx responses are *NOT* exceptions and should be handled in `then()`; see the next section for more information.
+### Handling exceptions
+
+NOTE: 3xx-5xx responses are _NOT_ exceptions, and should be handled in `then()`, see the next section.
-Adding a catch to the fetch promise chain will catch *all* exceptions, such as errors originating from node core libraries, network errors and operational errors, which are instances of FetchError. See the [error handling document](ERROR-HANDLING.md) for more details.
+Wrapping the fetch function into a `try/catch` block will catch _all_ exceptions, such as errors originating from node core libraries, like network errors, and operational errors which are instances of FetchError. See the [error handling document][error-handling.md] for more details.
```js
-fetch('https://domain.invalid/')
- .catch(err => console.error(err));
+import fetch from 'node-fetch';
+
+try {
+ await fetch('https://domain.invalid/');
+} catch (error) {
+ console.log(error);
+}
```
-#### Handling client and server errors
+### Handling client and server errors
+
It is common to create a helper function to check that the response contains no client (4xx) or server (5xx) error responses:
```js
-function checkStatus(res) {
- if (res.ok) { // res.status >= 200 && res.status < 300
- return res;
- } else {
- throw MyCustomError(res.statusText);
- }
+import fetch from 'node-fetch';
+
+class HTTPResponseError extends Error {
+ constructor(response, ...args) {
+ super(`HTTP Error Response: ${response.status} ${response.statusText}`, ...args);
+ this.response = response;
+ }
}
-fetch('https://httpbin.org/status/400')
- .then(checkStatus)
- .then(res => console.log('will not get here...'))
+const checkStatus = response => {
+ if (response.ok) {
+ // response.status >= 200 && response.status < 300
+ return response;
+ } else {
+ throw new HTTPResponseError(response);
+ }
+}
+
+const response = await fetch('https://httpbin.org/status/400');
+
+try {
+ checkStatus(response);
+} catch (error) {
+ console.error(error);
+
+ const errorBody = await error.response.text();
+ console.error(`Error body: ${errorBody}`);
+}
```
+### Handling cookies
+
+Cookies are not stored by default. However, cookies can be extracted and passed by manipulating request and response headers. See [Extract Set-Cookie Header](#extract-set-cookie-header) for details.
+
## Advanced Usage
-#### Streams
-The "Node.js way" is to use streams when possible:
+### Streams
+
+The "Node.js way" is to use streams when possible. You can pipe `res.body` to another stream. This example uses [stream.pipeline](https://nodejs.org/api/stream.html#stream_stream_pipeline_streams_callback) to attach stream error handlers and wait for the download to complete.
```js
-fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
- .then(res => {
- const dest = fs.createWriteStream('./octocat.png');
- res.body.pipe(dest);
- });
+import {createWriteStream} from 'node:fs';
+import {pipeline} from 'node:stream';
+import {promisify} from 'node:util'
+import fetch from 'node-fetch';
+
+const streamPipeline = promisify(pipeline);
+
+const response = await fetch('https://github.githubassets.com/images/modules/logos_page/Octocat.png');
+
+if (!response.ok) throw new Error(`unexpected response ${response.statusText}`);
+
+await streamPipeline(response.body, createWriteStream('./octocat.png'));
```
-#### Buffer
-If you prefer to cache binary data in full, use buffer(). (NOTE: `buffer()` is a `node-fetch`-only API)
+In Node.js 14 you can also use async iterators to read `body`; however, be careful to catch
+errors -- the longer a response runs, the more likely it is to encounter an error.
```js
-const fileType = require('file-type');
+import fetch from 'node-fetch';
-fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
- .then(res => res.buffer())
- .then(buffer => fileType(buffer))
- .then(type => { /* ... */ });
+const response = await fetch('https://httpbin.org/stream/3');
+
+try {
+ for await (const chunk of response.body) {
+ console.dir(JSON.parse(chunk.toString()));
+ }
+} catch (err) {
+ console.error(err.stack);
+}
```
-#### Accessing Headers and other Meta data
+In Node.js 12 you can also use async iterators to read `body`; however, async iterators with streams
+did not mature until Node.js 14, so you need to do some extra work to ensure you handle errors
+directly from the stream and wait on it response to fully close.
+
```js
-fetch('https://github.com/')
- .then(res => {
- console.log(res.ok);
- console.log(res.status);
- console.log(res.statusText);
- console.log(res.headers.raw());
- console.log(res.headers.get('content-type'));
- });
-```
+import fetch from 'node-fetch';
+
+const read = async body => {
+ let error;
+ body.on('error', err => {
+ error = err;
+ });
+
+ for await (const chunk of body) {
+ console.dir(JSON.parse(chunk.toString()));
+ }
+
+ return new Promise((resolve, reject) => {
+ body.on('close', () => {
+ error ? reject(error) : resolve();
+ });
+ });
+};
-#### Extract Set-Cookie Header
+try {
+ const response = await fetch('https://httpbin.org/stream/3');
+ await read(response.body);
+} catch (err) {
+ console.error(err.stack);
+}
+```
-Unlike browsers, you can access raw `Set-Cookie` headers manually using `Headers.raw()`. This is a `node-fetch` only API.
+### Accessing Headers and other Metadata
```js
-fetch(url).then(res => {
- // returns an array of values, instead of a string of comma-separated values
- console.log(res.headers.raw()['set-cookie']);
-});
+import fetch from 'node-fetch';
+
+const response = await fetch('https://github.com/');
+
+console.log(response.ok);
+console.log(response.status);
+console.log(response.statusText);
+console.log(response.headers.raw());
+console.log(response.headers.get('content-type'));
```
-#### Post data using a file stream
+### Extract Set-Cookie Header
+
+Unlike browsers, you can access raw `Set-Cookie` headers manually using `Headers.raw()`. This is a `node-fetch` only API.
```js
-const { createReadStream } = require('fs');
+import fetch from 'node-fetch';
-const stream = createReadStream('input.txt');
+const response = await fetch('https://example.com');
-fetch('https://httpbin.org/post', { method: 'POST', body: stream })
- .then(res => res.json())
- .then(json => console.log(json));
+// Returns an array of values, instead of a string of comma-separated values
+console.log(response.headers.raw()['set-cookie']);
```
-#### Post with form-data (detect multipart)
+### Post data using a file
```js
-const FormData = require('form-data');
+import fetch {
+ Blob,
+ blobFrom,
+ blobFromSync,
+ File,
+ fileFrom,
+ fileFromSync,
+} from 'node-fetch'
+
+const mimetype = 'text/plain'
+const blob = fileFromSync('./input.txt', mimetype)
+const url = 'https://httpbin.org/post'
+
+const response = await fetch(url, { method: 'POST', body: blob })
+const data = await response.json()
+
+console.log(data)
+```
-const form = new FormData();
-form.append('a', 1);
+node-fetch comes with a spec-compliant [FormData] implementations for posting
+multipart/form-data payloads
-fetch('https://httpbin.org/post', { method: 'POST', body: form })
- .then(res => res.json())
- .then(json => console.log(json));
+```js
+import fetch, { FormData, File, fileFrom } from 'node-fetch'
-// OR, using custom headers
-// NOTE: getHeaders() is non-standard API
+const httpbin = 'https://httpbin.org/post'
+const formData = new FormData()
+const binary = new Uint8Array([ 97, 98, 99 ])
+const abc = new File([binary], 'abc.txt', { type: 'text/plain' })
-const form = new FormData();
-form.append('a', 1);
+formData.set('greeting', 'Hello, world!')
+formData.set('file-upload', abc, 'new name.txt')
-const options = {
- method: 'POST',
- body: form,
- headers: form.getHeaders()
-}
+const response = await fetch(httpbin, { method: 'POST', body: formData })
+const data = await response.json()
-fetch('https://httpbin.org/post', options)
- .then(res => res.json())
- .then(json => console.log(json));
+console.log(data)
```
-#### Request cancellation with AbortSignal
+If you for some reason need to post a stream coming from any arbitrary place,
+then you can append a [Blob] or a [File] look-a-like item.
+
+The minimum requirement is that it has:
+1. A `Symbol.toStringTag` getter or property that is either `Blob` or `File`
+2. A known size.
+3. And either a `stream()` method or a `arrayBuffer()` method that returns a ArrayBuffer.
-> NOTE: You may cancel streamed requests only on Node >= v8.0.0
+The `stream()` must return any async iterable object as long as it yields Uint8Array (or Buffer)
+so Node.Readable streams and whatwg streams works just fine.
+
+```js
+formData.append('upload', {
+ [Symbol.toStringTag]: 'Blob',
+ size: 3,
+ *stream() {
+ yield new Uint8Array([97, 98, 99])
+ },
+ arrayBuffer() {
+ return new Uint8Array([97, 98, 99]).buffer
+ }
+}, 'abc.txt')
+```
+
+### Request cancellation with AbortSignal
You may cancel requests with `AbortController`. A suggested implementation is [`abort-controller`](https://www.npmjs.com/package/abort-controller).
An example of timing out a request after 150ms could be achieved as the following:
```js
-import AbortController from 'abort-controller';
+import fetch, { AbortError } from 'node-fetch';
+
+// AbortController was added in node v14.17.0 globally
+const AbortController = globalThis.AbortController || await import('abort-controller')
const controller = new AbortController();
-const timeout = setTimeout(
- () => { controller.abort(); },
- 150,
-);
-
-fetch(url, { signal: controller.signal })
- .then(res => res.json())
- .then(
- data => {
- useData(data)
- },
- err => {
- if (err.name === 'AbortError') {
- // request was aborted
- }
- },
- )
- .finally(() => {
- clearTimeout(timeout);
- });
+const timeout = setTimeout(() => {
+ controller.abort();
+}, 150);
+
+try {
+ const response = await fetch('https://example.com', {signal: controller.signal});
+ const data = await response.json();
+} catch (error) {
+ if (error instanceof AbortError) {
+ console.log('request was aborted');
+ }
+} finally {
+ clearTimeout(timeout);
+}
```
-See [test cases](https://github.com/bitinn/node-fetch/blob/master/test/test.js) for more examples.
-
+See [test cases](https://github.com/node-fetch/node-fetch/blob/master/test/) for more examples.
## API
@@ -311,47 +504,51 @@ See [test cases](https://github.com/bitinn/node-fetch/blob/master/test/test.js)
Perform an HTTP(S) fetch.
-`url` should be an absolute url, such as `https://example.com/`. A path-relative URL (`/file/under/root`) or protocol-relative URL (`//can-be-http-or-https.com/`) will result in a rejected `Promise`.
+`url` should be an absolute URL, such as `https://example.com/`. A path-relative URL (`/file/under/root`) or protocol-relative URL (`//can-be-http-or-https.com/`) will result in a rejected `Promise`.
+
### Options
The default values are shown after each option key.
```js
{
- // These properties are part of the Fetch Standard
- method: 'GET',
- headers: {}, // request headers. format is the identical to that accepted by the Headers constructor (see below)
- body: null, // request body. can be null, a string, a Buffer, a Blob, or a Node.js Readable stream
- redirect: 'follow', // set to `manual` to extract redirect headers, `error` to reject redirect
- signal: null, // pass an instance of AbortSignal to optionally abort requests
-
- // The following properties are node-fetch extensions
- follow: 20, // maximum redirect count. 0 to not follow redirect
- timeout: 0, // req/res timeout in ms, it resets on redirect. 0 to disable (OS limit applies). Signal is recommended instead.
- compress: true, // support gzip/deflate content encoding. false to disable
- size: 0, // maximum response body size in bytes. 0 to disable
- agent: null // http(s).Agent instance or function that returns an instance (see below)
+ // These properties are part of the Fetch Standard
+ method: 'GET',
+ headers: {}, // Request headers. format is the identical to that accepted by the Headers constructor (see below)
+ body: null, // Request body. can be null, or a Node.js Readable stream
+ redirect: 'follow', // Set to `manual` to extract redirect headers, `error` to reject redirect
+ signal: null, // Pass an instance of AbortSignal to optionally abort requests
+
+ // The following properties are node-fetch extensions
+ follow: 20, // maximum redirect count. 0 to not follow redirect
+ compress: true, // support gzip/deflate content encoding. false to disable
+ size: 0, // maximum response body size in bytes. 0 to disable
+ agent: null, // http(s).Agent instance or function that returns an instance (see below)
+ highWaterMark: 16384, // the maximum number of bytes to store in the internal buffer before ceasing to read from the underlying resource.
+ insecureHTTPParser: false // Use an insecure HTTP parser that accepts invalid HTTP headers when `true`.
}
```
-##### Default Headers
+#### Default Headers
If no values are set, the following request headers will be sent automatically:
-Header | Value
-------------------- | --------------------------------------------------------
-`Accept-Encoding` | `gzip,deflate` _(when `options.compress === true`)_
-`Accept` | `*/*`
-`Connection` | `close` _(when no `options.agent` is present)_
-`Content-Length` | _(automatically calculated, if possible)_
-`Transfer-Encoding` | `chunked` _(when `req.body` is a stream)_
-`User-Agent` | `node-fetch/1.0 (+https://github.com/bitinn/node-fetch)`
+| Header | Value |
+| ------------------- | ------------------------------------------------------ |
+| `Accept-Encoding` | `gzip, deflate, br` (when `options.compress === true`) |
+| `Accept` | `*/*` |
+| `Connection` | `close` _(when no `options.agent` is present)_ |
+| `Content-Length` | _(automatically calculated, if possible)_ |
+| `Host` | _(host and port information from the target URI)_ |
+| `Transfer-Encoding` | `chunked` _(when `req.body` is a stream)_ |
+| `User-Agent` | `node-fetch` |
+
Note: when `body` is a `Stream`, `Content-Length` is not set automatically.
-##### Custom Agent
+#### Custom Agent
The `agent` option allows you to specify networking related options which are out of the scope of Fetch, including and not limited to the following:
@@ -364,25 +561,85 @@ See [`http.Agent`](https://nodejs.org/api/http.html#http_new_agent_options) for
In addition, the `agent` option accepts a function that returns `http`(s)`.Agent` instance given current [URL](https://nodejs.org/api/url.html), this is useful during a redirection chain across HTTP and HTTPS protocol.
```js
+import http from 'node:http';
+import https from 'node:https';
+
const httpAgent = new http.Agent({
- keepAlive: true
+ keepAlive: true
});
const httpsAgent = new https.Agent({
- keepAlive: true
+ keepAlive: true
});
const options = {
- agent: function (_parsedURL) {
- if (_parsedURL.protocol == 'http:') {
- return httpAgent;
- } else {
- return httpsAgent;
- }
- }
+ agent: function(_parsedURL) {
+ if (_parsedURL.protocol == 'http:') {
+ return httpAgent;
+ } else {
+ return httpsAgent;
+ }
+ }
+};
+```
+
+
+
+#### Custom highWaterMark
+
+Stream on Node.js have a smaller internal buffer size (16kB, aka `highWaterMark`) from client-side browsers (>1MB, not consistent across browsers). Because of that, when you are writing an isomorphic app and using `res.clone()`, it will hang with large response in Node.
+
+The recommended way to fix this problem is to resolve cloned response in parallel:
+
+```js
+import fetch from 'node-fetch';
+
+const response = await fetch('https://example.com');
+const r1 = response.clone();
+
+const results = await Promise.all([response.json(), r1.text()]);
+
+console.log(results[0]);
+console.log(results[1]);
+```
+
+If for some reason you don't like the solution above, since `3.x` you are able to modify the `highWaterMark` option:
+
+```js
+import fetch from 'node-fetch';
+
+const response = await fetch('https://example.com', {
+ // About 1MB
+ highWaterMark: 1024 * 1024
+});
+
+const result = await res.clone().arrayBuffer();
+console.dir(result);
+```
+
+#### Insecure HTTP Parser
+
+Passed through to the `insecureHTTPParser` option on http(s).request. See [`http.request`](https://nodejs.org/api/http.html#http_http_request_url_options_callback) for more information.
+
+#### Manual Redirect
+
+The `redirect: 'manual'` option for node-fetch is different from the browser & specification, which
+results in an [opaque-redirect filtered response](https://fetch.spec.whatwg.org/#concept-filtered-response-opaque-redirect).
+node-fetch gives you the typical [basic filtered response](https://fetch.spec.whatwg.org/#concept-filtered-response-basic) instead.
+
+```js
+const fetch = require('node-fetch');
+
+const response = await fetch('https://httpbin.org/status/301', { redirect: 'manual' });
+
+if (response.status === 301 || response.status === 302) {
+ const locationURL = new URL(response.headers.get('location'), response.url);
+ const response2 = await fetch(locationURL, { redirect: 'manual' });
+ console.dir(response2);
}
```
+
### Class: Request
An HTTP(S) request containing information about URL, method, headers, and the body. This class implements the [Body](#iface-body) interface.
@@ -391,8 +648,6 @@ Due to the nature of Node.js, the following properties are not implemented at th
- `type`
- `destination`
-- `referrer`
-- `referrerPolicy`
- `mode`
- `credentials`
- `cache`
@@ -405,35 +660,34 @@ The following node-fetch extension properties are provided:
- `compress`
- `counter`
- `agent`
+- `highWaterMark`
See [options](#fetch-options) for exact meaning of these extensions.
#### new Request(input[, options])
-*(spec-compliant)*
+_(spec-compliant)_
- `input` A string representing a URL, or another `Request` (which will be cloned)
-- `options` [Options][#fetch-options] for the HTTP(S) request
+- `options` [Options](#fetch-options) for the HTTP(S) request
Constructs a new `Request` object. The constructor is identical to that in the [browser](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request).
In most cases, directly `fetch(url, options)` is simpler than creating a `Request` object.
+
### Class: Response
An HTTP(S) response. This class implements the [Body](#iface-body) interface.
The following properties are not implemented in node-fetch at this moment:
-- `Response.error()`
-- `Response.redirect()`
-- `type`
- `trailer`
#### new Response([body[, options]])
-*(spec-compliant)*
+_(spec-compliant)_
- `body` A `String` or [`Readable` stream][node-readable]
- `options` A [`ResponseInit`][response-init] options dictionary
@@ -444,24 +698,31 @@ Because Node.js does not implement service workers (for which this class was des
#### response.ok
-*(spec-compliant)*
+_(spec-compliant)_
Convenience property representing if the request ended normally. Will evaluate to true if the response status was greater than or equal to 200 but smaller than 300.
#### response.redirected
-*(spec-compliant)*
+_(spec-compliant)_
Convenience property representing if the request has been redirected at least once. Will evaluate to true if the internal redirect counter is greater than 0.
+#### response.type
+
+_(deviation from spec)_
+
+Convenience property representing the response's type. node-fetch only supports `'default'` and `'error'` and does not make use of [filtered responses](https://fetch.spec.whatwg.org/#concept-filtered-response).
+
+
### Class: Headers
This class allows manipulating and iterating over a set of HTTP headers. All methods specified in the [Fetch Standard][whatwg-fetch] are implemented.
#### new Headers([init])
-*(spec-compliant)*
+_(spec-compliant)_
- `init` Optional argument to pre-fill the `Headers` object
@@ -469,122 +730,142 @@ Construct a new `Headers` object. `init` can be either `null`, a `Headers` objec
```js
// Example adapted from https://fetch.spec.whatwg.org/#example-headers-class
+import {Headers} from 'node-fetch';
const meta = {
- 'Content-Type': 'text/xml',
- 'Breaking-Bad': '<3'
+ 'Content-Type': 'text/xml'
};
const headers = new Headers(meta);
// The above is equivalent to
-const meta = [
- [ 'Content-Type', 'text/xml' ],
- [ 'Breaking-Bad', '<3' ]
-];
+const meta = [['Content-Type', 'text/xml']];
const headers = new Headers(meta);
// You can in fact use any iterable objects, like a Map or even another Headers
const meta = new Map();
meta.set('Content-Type', 'text/xml');
-meta.set('Breaking-Bad', '<3');
const headers = new Headers(meta);
const copyOfHeaders = new Headers(headers);
```
+
### Interface: Body
`Body` is an abstract interface with methods that are applicable to both `Request` and `Response` classes.
-The following methods are not yet implemented in node-fetch at this moment:
-
-- `formData()`
-
#### body.body
-*(deviation from spec)*
+_(deviation from spec)_
-* Node.js [`Readable` stream][node-readable]
+- Node.js [`Readable` stream][node-readable]
Data are encapsulated in the `Body` object. Note that while the [Fetch Standard][whatwg-fetch] requires the property to always be a WHATWG `ReadableStream`, in node-fetch it is a Node.js [`Readable` stream][node-readable].
#### body.bodyUsed
-*(spec-compliant)*
+_(spec-compliant)_
-* `Boolean`
+- `Boolean`
A boolean property for if this body has been consumed. Per the specs, a consumed body cannot be used again.
#### body.arrayBuffer()
-#### body.blob()
-#### body.json()
-#### body.text()
-
-*(spec-compliant)*
-
-* Returns:
+ +
A light-weight module that brings Fetch API to Node.js.
+
+
+ +
+ Consider supporting us on our Open Collective: +
+
+
+Promise
-
-Consume the body and return a promise that will resolve to one of these formats.
-#### body.buffer()
+#### body.formData()
-*(node-fetch extension)*
-
-* Returns: Promise<Buffer>
+#### body.blob()
-Consume the body and return a promise that will resolve to a Buffer.
+#### body.json()
-#### body.textConverted()
+#### body.text()
-*(node-fetch extension)*
+`fetch` comes with methods to parse `multipart/form-data` payloads as well as
+`x-www-form-urlencoded` bodies using `.formData()` this comes from the idea that
+Service Worker can intercept such messages before it's sent to the server to
+alter them. This is useful for anybody building a server so you can use it to
+parse & consume payloads.
-* Returns: Promise<String>
+
+
+
### Class: FetchError
-*(node-fetch extension)*
+_(node-fetch extension)_
An operational error in the fetching process. See [ERROR-HANDLING.md][] for more info.
+
### Class: AbortError
-*(node-fetch extension)*
+_(node-fetch extension)_
An Error thrown when the request is aborted in response to an `AbortSignal`'s `abort` event. It has a `name` property of `AbortError`. See [ERROR-HANDLING.MD][] for more info.
+## TypeScript
+
+**Since `3.x` types are bundled with `node-fetch`, so you don't need to install any additional packages.**
+
+For older versions please use the type definitions from [DefinitelyTyped](https://github.com/DefinitelyTyped/DefinitelyTyped):
+
+```sh
+npm install --save-dev @types/node-fetch@2.x
+```
+
## Acknowledgement
Thanks to [github/fetch](https://github.com/github/fetch) for providing a solid implementation reference.
-`node-fetch` v1 was maintained by [@bitinn](https://github.com/bitinn); v2 was maintained by [@TimothyGu](https://github.com/timothygu), [@bitinn](https://github.com/bitinn) and [@jimmywarting](https://github.com/jimmywarting); v2 readme is written by [@jkantr](https://github.com/jkantr).
+## Team
+
+| [](https://github.com/bitinn) | [](https://github.com/jimmywarting) | [](https://github.com/xxczaki) | [](https://github.com/Richienb) | [](https://github.com/gr2m) |
+| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- |
+| [David Frank](https://bitinn.net/) | [Jimmy Wärting](https://jimmy.warting.se/) | [Antoni Kepinski](https://kepinski.ch) | [Richie Bendall](https://www.richie-bendall.ml/) | [Gregor Martynus](https://twitter.com/gr2m) |
+
+###### Former
+
+- [Timothy Gu](https://github.com/timothygu)
+- [Jared Kantrowitz](https://github.com/jkantr)
## License
-MIT
-
-[npm-image]: https://flat.badgen.net/npm/v/node-fetch
-[npm-url]: https://www.npmjs.com/package/node-fetch
-[travis-image]: https://flat.badgen.net/travis/bitinn/node-fetch
-[travis-url]: https://travis-ci.org/bitinn/node-fetch
-[codecov-image]: https://flat.badgen.net/codecov/c/github/bitinn/node-fetch/master
-[codecov-url]: https://codecov.io/gh/bitinn/node-fetch
-[install-size-image]: https://flat.badgen.net/packagephobia/install/node-fetch
-[install-size-url]: https://packagephobia.now.sh/result?p=node-fetch
-[discord-image]: https://img.shields.io/discord/619915844268326952?color=%237289DA&label=Discord&style=flat-square
-[discord-url]: https://discord.gg/Zxbndcm
-[opencollective-image]: https://opencollective.com/node-fetch/backers.svg
-[opencollective-url]: https://opencollective.com/node-fetch
+[MIT](LICENSE.md)
+
[whatwg-fetch]: https://fetch.spec.whatwg.org/
[response-init]: https://fetch.spec.whatwg.org/#responseinit
[node-readable]: https://nodejs.org/api/stream.html#stream_readable_streams
[mdn-headers]: https://developer.mozilla.org/en-US/docs/Web/API/Headers
-[LIMITS.md]: https://github.com/bitinn/node-fetch/blob/master/LIMITS.md
-[ERROR-HANDLING.md]: https://github.com/bitinn/node-fetch/blob/master/ERROR-HANDLING.md
-[UPGRADE-GUIDE.md]: https://github.com/bitinn/node-fetch/blob/master/UPGRADE-GUIDE.md
+[error-handling.md]: https://github.com/node-fetch/node-fetch/blob/master/docs/ERROR-HANDLING.md
+[FormData]: https://developer.mozilla.org/en-US/docs/Web/API/FormData
+[Blob]: https://developer.mozilla.org/en-US/docs/Web/API/Blob
+[File]: https://developer.mozilla.org/en-US/docs/Web/API/File
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 000000000..e60fc6870
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,5 @@
+# Security Policy
+
+## Reporting a Vulnerability
+
+Please report security issues to `jimmy@warting.se`
\ No newline at end of file
diff --git a/browser.js b/browser.js
deleted file mode 100644
index 83c54c584..000000000
--- a/browser.js
+++ /dev/null
@@ -1,25 +0,0 @@
-"use strict";
-
-// ref: https://github.com/tc39/proposal-global
-var getGlobal = function () {
- // the only reliable means to get the global object is
- // `Function('return this')()`
- // However, this causes CSP violations in Chrome apps.
- if (typeof self !== 'undefined') { return self; }
- if (typeof window !== 'undefined') { return window; }
- if (typeof global !== 'undefined') { return global; }
- throw new Error('unable to locate global object');
-}
-
-var global = getGlobal();
-
-module.exports = exports = global.fetch;
-
-// Needed for TypeScript and Webpack.
-if (global.fetch) {
- exports.default = global.fetch.bind(global);
-}
-
-exports.Headers = global.Headers;
-exports.Request = global.Request;
-exports.Response = global.Response;
\ No newline at end of file
diff --git a/build/babel-plugin.js b/build/babel-plugin.js
deleted file mode 100644
index 8cddae954..000000000
--- a/build/babel-plugin.js
+++ /dev/null
@@ -1,61 +0,0 @@
-// This Babel plugin makes it possible to do CommonJS-style function exports
-
-const walked = Symbol('walked');
-
-module.exports = ({ types: t }) => ({
- visitor: {
- Program: {
- exit(program) {
- if (program[walked]) {
- return;
- }
-
- for (let path of program.get('body')) {
- if (path.isExpressionStatement()) {
- const expr = path.get('expression');
- if (expr.isAssignmentExpression() &&
- expr.get('left').matchesPattern('exports.*')) {
- const prop = expr.get('left').get('property');
- if (prop.isIdentifier({ name: 'default' })) {
- program.unshiftContainer('body', [
- t.expressionStatement(
- t.assignmentExpression('=',
- t.identifier('exports'),
- t.assignmentExpression('=',
- t.memberExpression(
- t.identifier('module'), t.identifier('exports')
- ),
- expr.node.right
- )
- )
- ),
- t.expressionStatement(
- t.callExpression(
- t.memberExpression(
- t.identifier('Object'), t.identifier('defineProperty')),
- [
- t.identifier('exports'),
- t.stringLiteral('__esModule'),
- t.objectExpression([
- t.objectProperty(t.identifier('value'), t.booleanLiteral(true))
- ])
- ]
- )
- ),
- t.expressionStatement(
- t.assignmentExpression('=',
- expr.node.left, t.identifier('exports')
- )
- )
- ]);
- path.remove();
- }
- }
- }
- }
-
- program[walked] = true;
- }
- }
- }
-});
diff --git a/build/rollup-plugin.js b/build/rollup-plugin.js
deleted file mode 100644
index 36ebdc804..000000000
--- a/build/rollup-plugin.js
+++ /dev/null
@@ -1,18 +0,0 @@
-export default function tweakDefault() {
- return {
- transformBundle: function (source) {
- var lines = source.split('\n');
- for (var i = 0; i < lines.length; i++) {
- var line = lines[i];
- var matches = /^(exports(?:\['default']|\.default)) = (.*);$/.exec(line);
- if (matches) {
- lines[i] = 'module.exports = exports = ' + matches[2] + ';\n' +
- 'Object.defineProperty(exports, "__esModule", { value: true });\n' +
- matches[1] + ' = exports;';
- break;
- }
- }
- return lines.join('\n');
- }
- };
-}
diff --git a/codecov.yml b/codecov.yml
deleted file mode 100644
index b4e9d3fcd..000000000
--- a/codecov.yml
+++ /dev/null
@@ -1,3 +0,0 @@
-parsers:
- javascript:
- enable_partials: yes
diff --git a/docs/ERROR-HANDLING.md b/docs/ERROR-HANDLING.md
new file mode 100644
index 000000000..85b1f0de8
--- /dev/null
+++ b/docs/ERROR-HANDLING.md
@@ -0,0 +1,39 @@
+
+Error handling with node-fetch
+==============================
+
+Because `window.fetch` isn't designed to be transparent about the cause of request errors, we have to come up with our own solutions.
+
+The basics:
+
+- A cancelled request is rejected with an [`AbortError`](https://github.com/node-fetch/node-fetch/blob/master/README.md#class-aborterror). You can check if the reason for rejection was that the request was aborted by checking the `Error`'s `name` is `AbortError`.
+
+```js
+const fetch = require('node-fetch');
+
+(async () => {
+ try {
+ await fetch(url, {signal});
+ } catch (error) {
+ if (error.name === 'AbortError') {
+ console.log('request was aborted');
+ }
+ }
+})();
+```
+
+- All [operational errors][joyent-guide] *other than aborted requests* are rejected with a [FetchError](https://github.com/node-fetch/node-fetch/blob/master/README.md#class-fetcherror). You can handle them all through the `try/catch` block or promise `catch` clause.
+
+- All errors come with an `error.message` detailing the cause of errors.
+
+- All errors originating from `node-fetch` are marked with a custom `err.type`.
+
+- All errors originating from Node.js core are marked with `error.type = 'system'`, and in addition contain an `error.code` and an `error.errno` for error handling. These are aliases for error codes thrown by Node.js core.
+
+- [Programmer errors][joyent-guide] are either thrown as soon as possible, or rejected with default `Error` with `error.message` for ease of troubleshooting.
+
+List of error types:
+
+- Because we maintain 100% coverage, see [test/main.js](https://github.com/node-fetch/node-fetch/blob/master/test/main.js) for a full list of custom `FetchError` types, as well as some of the common errors from Node.js
+
+[joyent-guide]: https://www.joyent.com/node-js/production/design/errors#operational-errors-vs-programmer-errors
diff --git a/docs/media/Banner.svg b/docs/media/Banner.svg
new file mode 100644
index 000000000..b9c079783
--- /dev/null
+++ b/docs/media/Banner.svg
@@ -0,0 +1,21 @@
+
+
\ No newline at end of file
diff --git a/docs/media/Logo.svg b/docs/media/Logo.svg
new file mode 100644
index 000000000..8d1a2c9e8
--- /dev/null
+++ b/docs/media/Logo.svg
@@ -0,0 +1,21 @@
+
+
\ No newline at end of file
diff --git a/docs/media/NodeFetch.sketch b/docs/media/NodeFetch.sketch
new file mode 100644
index 000000000..ad858e7bf
Binary files /dev/null and b/docs/media/NodeFetch.sketch differ
diff --git a/LIMITS.md b/docs/v2-LIMITS.md
similarity index 90%
rename from LIMITS.md
rename to docs/v2-LIMITS.md
index 9c4b8c0c8..849a15533 100644
--- a/LIMITS.md
+++ b/docs/v2-LIMITS.md
@@ -26,7 +26,7 @@ Known differences
- If you are using `res.clone()` and writing an isomorphic app, note that stream on Node.js have a smaller internal buffer size (16Kb, aka `highWaterMark`) from client-side browsers (>1Mb, not consistent across browsers).
-- Because node.js stream doesn't expose a [*disturbed*](https://fetch.spec.whatwg.org/#concept-readablestream-disturbed) property like Stream spec, using a consumed stream for `new Response(body)` will not set `bodyUsed` flag correctly.
+- Because Node.js stream doesn't expose a [*disturbed*](https://fetch.spec.whatwg.org/#concept-readablestream-disturbed) property like Stream spec, using a consumed stream for `new Response(body)` will not set `bodyUsed` flag correctly.
[readable-stream]: https://nodejs.org/api/stream.html#stream_readable_streams
-[ERROR-HANDLING.md]: https://github.com/bitinn/node-fetch/blob/master/ERROR-HANDLING.md
+[ERROR-HANDLING.md]: https://github.com/node-fetch/node-fetch/blob/master/docs/ERROR-HANDLING.md
diff --git a/UPGRADE-GUIDE.md b/docs/v2-UPGRADE-GUIDE.md
similarity index 95%
rename from UPGRADE-GUIDE.md
rename to docs/v2-UPGRADE-GUIDE.md
index 22aab748b..3660dfb3a 100644
--- a/UPGRADE-GUIDE.md
+++ b/docs/v2-UPGRADE-GUIDE.md
@@ -45,7 +45,7 @@ spec-compliant. These changes are done in conjunction with GitHub's
const headers = new Headers({
'Abc': 'string',
- 'Multi': [ 'header1', 'header2' ]
+ 'Multi': ['header1', 'header2']
});
// before after
@@ -63,14 +63,14 @@ headers.get('Multi') => headers.get('Multi') =>
const headers = new Headers({
'Abc': 'string',
- 'Multi': [ 'header1', 'header2' ]
+ 'Multi': ['header1', 'header2']
});
// before after
headers.getAll('Multi') => headers.getAll('Multi') =>
[ 'header1', 'header2' ]; throws ReferenceError
headers.get('Multi').split(',') =>
- [ 'header1', 'header2' ];
+ ['header1', 'header2'];
//////////////////////////////////////////////////////////////////////////////
@@ -91,7 +91,7 @@ headers.get(undefined) headers.get(undefined)
const headers = new Headers();
headers.set('Héy', 'ok'); // now throws
headers.get('Héy'); // now throws
-new Headers({ 'Héy': 'ok' }); // now throws
+new Headers({'Héy': 'ok'}); // now throws
```
## Node.js v0.x support dropped
diff --git a/docs/v3-LIMITS.md b/docs/v3-LIMITS.md
new file mode 100644
index 000000000..a53202e64
--- /dev/null
+++ b/docs/v3-LIMITS.md
@@ -0,0 +1,30 @@
+Known differences
+=================
+
+*As of 3.x release*
+
+- Topics such as Cross-Origin, Content Security Policy, Mixed Content, Service Workers are ignored, given our server-side context.
+
+- On the upside, there are no forbidden headers.
+
+- `res.url` contains the final url when following redirects.
+
+- For convenience, `res.body` is a Node.js [Readable stream][readable-stream], so decoding can be handled independently.
+
+- Similarly, `req.body` can either be `null`, a buffer or a Readable stream.
+
+- Also, you can handle rejected fetch requests through checking `err.type` and `err.code`. See [ERROR-HANDLING.md][] for more info.
+
+- Only support `res.text()`, `res.json()`, `res.blob()`, `res.arraybuffer()`, `res.buffer()`
+
+- There is currently no built-in caching, as server-side caching varies by use-cases.
+
+- Current implementation lacks server-side cookie store, you will need to extract `Set-Cookie` headers manually.
+
+- If you are using `res.clone()` and writing an isomorphic app, note that stream on Node.js has a smaller internal buffer size (16Kb, aka `highWaterMark`) from client-side browsers (>1Mb, not consistent across browsers). Learn [how to get around this][highwatermark-fix].
+
+- Because Node.js stream doesn't expose a [*disturbed*](https://fetch.spec.whatwg.org/#concept-readablestream-disturbed) property like Stream spec, using a consumed stream for `new Response(body)` will not set `bodyUsed` flag correctly.
+
+[readable-stream]: https://nodejs.org/api/stream.html#stream_readable_streams
+[ERROR-HANDLING.md]: https://github.com/node-fetch/node-fetch/blob/master/docs/ERROR-HANDLING.md
+[highwatermark-fix]: https://github.com/node-fetch/node-fetch/blob/master/README.md#custom-highwatermark
diff --git a/docs/v3-UPGRADE-GUIDE.md b/docs/v3-UPGRADE-GUIDE.md
new file mode 100644
index 000000000..4e9eada0f
--- /dev/null
+++ b/docs/v3-UPGRADE-GUIDE.md
@@ -0,0 +1,144 @@
+# Upgrade to node-fetch v3.x
+
+node-fetch v3.x brings about many changes that increase the compliance of
+WHATWG's [Fetch Standard][whatwg-fetch]. However, many of these changes mean
+that apps written for node-fetch v2.x needs to be updated to work with
+node-fetch v3.x and be conformant with the Fetch Standard. This document helps
+you make this transition.
+
+Note that this document is not an exhaustive list of all changes made in v3.x,
+but rather that of the most important breaking changes. See our [changelog] for
+other comparatively minor modifications.
+
+- [Breaking Changes](#breaking)
+- [Enhancements](#enhancements)
+
+---
+
+
+
+# Breaking Changes
+
+## Minimum supported Node.js version is now 12.20
+
+Since Node.js 10 has been deprecated since May 2020, we have decided that node-fetch v3 will drop support for Node.js 4, 6, 8, and 10 (which were previously supported). We strongly encourage you to upgrade if you still haven't done so. Check out the Node.js official [LTS plan] for more information.
+
+## Converted to ES Module
+
+This module was converted to be a ESM only package in version `3.0.0-beta.10`.
+`node-fetch` is an ESM-only module - you are not able to import it with `require`. We recommend you stay on v2 which is built with CommonJS unless you use ESM yourself. We will continue to publish critical bug fixes for it.
+
+Alternatively, you can use the async `import()` function from CommonJS to load `node-fetch` asynchronously:
+
+```js
+// mod.cjs
+const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));
+```
+
+## The `timeout` option was removed.
+
+Since this was never part of the fetch specification, it was removed. AbortSignal offers more fine grained control of request timeouts, and is standardized in the Fetch spec. For convenience, you can use [timeout-signal](https://github.com/node-fetch/timeout-signal) as a workaround:
+
+```js
+import timeoutSignal from 'timeout-signal';
+import fetch from 'node-fetch';
+
+const {AbortError} = fetch
+
+fetch('https://www.google.com', { signal: timeoutSignal(5000) })
+ .then(response => {
+ // Handle response
+ })
+ .catch(error => {
+ if (error instanceof AbortError) {
+ // Handle timeout
+ }
+ })
+```
+
+## `Response.statusText` no longer sets a default message derived from the HTTP status code
+
+If the server didn't respond with status text, node-fetch would set a default message derived from the HTTP status code. This behavior was not spec-compliant and now the `statusText` will remain blank instead.
+
+## Dropped the `browser` field in package.json
+
+Prior to v3.x, we included a `browser` field in the package.json file. Since node-fetch is intended to be used on the server, we have removed this field. If you are using node-fetch client-side, consider switching to something like [cross-fetch].
+
+## Dropped the `res.textConverted()` function
+
+If you want charset encoding detection, please use the [fetch-charset-detection] package ([documentation][fetch-charset-detection-docs]).
+
+```js
+import fetch from 'node-fetch';
+import convertBody from 'fetch-charset-detection';
+
+fetch('https://somewebsite.com').then(async res => {
+ const buf = await res.arrayBuffer();
+ const text = convertBody(buf, res.headers);
+});
+```
+
+## JSON parsing errors from `res.json()` are of type `SyntaxError` instead of `FetchError`
+
+When attempting to parse invalid json via `res.json()`, a `SyntaxError` will now be thrown instead of a `FetchError` to align better with the spec.
+
+```js
+import fetch from 'node-fetch';
+
+fetch('https://somewebsitereturninginvalidjson.com').then(res => res.json())
+// Throws 'Uncaught SyntaxError: Unexpected end of JSON input' or similar.
+```
+
+## A stream pipeline is now used to forward errors
+
+If you are listening for errors via `res.body.on('error', () => ...)`, replace it with `res.body.once('error', () => ...)` so that your callback is not [fired twice](https://github.com/node-fetch/node-fetch/issues/668#issuecomment-569386115) in NodeJS >=13.5.
+
+## `req.body` can no longer be a string
+
+We are working towards changing body to become either null or a stream.
+
+## Changed default user agent
+
+The default user agent has been changed from `node-fetch/1.0 (+https://github.com/node-fetch/node-fetch)` to `node-fetch (+https://github.com/node-fetch/node-fetch)`.
+
+## Arbitrary URLs are no longer supported
+
+Since in 3.x we are using the WHATWG's `new URL()`, arbitrary URL parsing will fail due to lack of base.
+
+# Enhancements
+
+## Data URI support
+
+Previously, node-fetch only supported http url scheme. However, the Fetch Standard recently introduced the `data:` URI support. Following the specification, we implemented this feature in v3.x. Read more about `data:` URLs [here][data-url].
+
+## New & exposed Blob implementation
+
+Blob implementation is now [fetch-blob] and hence is exposed, unlikely previously, where Blob type was only internal and not exported.
+
+## Better UTF-8 URL handling
+
+We now use the new Node.js [WHATWG-compliant URL API][whatwg-nodejs-url], so UTF-8 URLs are handled properly.
+
+## Request errors are now piped using `stream.pipeline`
+
+Since the v3.x requires at least Node.js 12.20.0, we can utilise the new API.
+
+## Creating Request/Response objects with relative URLs is no longer supported
+
+We introduced Node.js `new URL()` API in 3.x, because it offers better UTF-8 support and is WHATWG URL compatible. The drawback is, given current limit of the API (nodejs/node#12682), it's not possible to support relative URL parsing without hacks.
+Due to the lack of a browsing context in Node.js, we opted to drop support for relative URLs on Request/Response object, and it will now throw errors if you do so.
+The main `fetch()` function will support absolute URLs and data url.
+
+## Bundled TypeScript types
+
+Since v3.x you no longer need to install `@types/node-fetch` package in order to use `node-fetch` with TypeScript.
+
+[whatwg-fetch]: https://fetch.spec.whatwg.org/
+[data-url]: https://fetch.spec.whatwg.org/#data-url-processor
+[LTS plan]: https://github.com/nodejs/LTS#lts-plan
+[cross-fetch]: https://github.com/lquixada/cross-fetch
+[fetch-charset-detection]: https://github.com/Richienb/fetch-charset-detection
+[fetch-charset-detection-docs]: https://richienb.github.io/fetch-charset-detection/globals.html#convertbody
+[fetch-blob]: https://github.com/node-fetch/fetch-blob#readme
+[whatwg-nodejs-url]: https://nodejs.org/api/url.html#url_the_whatwg_url_api
+[changelog]: CHANGELOG.md
diff --git a/example.js b/example.js
new file mode 100644
index 000000000..f19ff803f
--- /dev/null
+++ b/example.js
@@ -0,0 +1,37 @@
+/*
+ Here are some example ways in which you can use node-fetch. Test each code fragment separately so that you don't get errors related to constant reassigning, etc.
+
+ Top-level `await` support is required.
+*/
+
+import fetch from 'node-fetch';
+
+// Plain text or HTML
+const response = await fetch('https://github.com/');
+const body = await response.text();
+
+console.log(body);
+
+// JSON
+const response = await fetch('https://github.com/');
+const json = await response.json();
+
+console.log(json);
+
+// Simple Post
+const response = await fetch('https://httpbin.org/post', {method: 'POST', body: 'a=1'});
+const json = await response.json();
+
+console.log(json);
+
+// Post with JSON
+const body = {a: 1};
+
+const response = await fetch('https://httpbin.org/post', {
+ method: 'post',
+ body: JSON.stringify(body),
+ headers: {'Content-Type': 'application/json'}
+});
+const json = await response.json();
+
+console.log(json);
diff --git a/package.json b/package.json
index 8e5c883b2..ae5e19650 100644
--- a/package.json
+++ b/package.json
@@ -1,66 +1,131 @@
{
"name": "node-fetch",
- "version": "2.6.0",
- "description": "A light-weight module that brings window.fetch to node.js",
- "main": "lib/index",
- "browser": "./browser.js",
- "module": "lib/index.mjs",
+ "version": "3.1.1",
+ "description": "A light-weight module that brings Fetch API to node.js",
+ "main": "./src/index.js",
+ "sideEffects": false,
+ "type": "module",
"files": [
- "lib/index.js",
- "lib/index.mjs",
- "lib/index.es.js",
- "browser.js"
+ "src",
+ "@types/index.d.ts"
],
+ "types": "./@types/index.d.ts",
"engines": {
- "node": "4.x || >=6.0.0"
+ "node": "^12.20.0 || ^14.13.1 || >=16.0.0"
},
"scripts": {
- "build": "cross-env BABEL_ENV=rollup rollup -c",
- "prepare": "npm run build",
- "test": "cross-env BABEL_ENV=test mocha --require babel-register --throw-deprecation test/test.js",
- "report": "cross-env BABEL_ENV=coverage nyc --reporter lcov --reporter text mocha -R spec test/test.js",
- "coverage": "cross-env BABEL_ENV=coverage nyc --reporter json --reporter text mocha -R spec test/test.js && codecov -f coverage/coverage-final.json"
+ "test": "mocha",
+ "coverage": "c8 report --reporter=text-lcov | coveralls",
+ "test-types": "tsd",
+ "lint": "xo"
},
"repository": {
"type": "git",
- "url": "https://github.com/bitinn/node-fetch.git"
+ "url": "https://github.com/node-fetch/node-fetch.git"
},
"keywords": [
"fetch",
"http",
- "promise"
+ "promise",
+ "request",
+ "curl",
+ "wget",
+ "xhr",
+ "whatwg"
],
"author": "David Frank",
"license": "MIT",
"bugs": {
- "url": "https://github.com/bitinn/node-fetch/issues"
+ "url": "https://github.com/node-fetch/node-fetch/issues"
+ },
+ "homepage": "https://github.com/node-fetch/node-fetch",
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/node-fetch"
},
- "homepage": "https://github.com/bitinn/node-fetch",
"devDependencies": {
- "@ungap/url-search-params": "^0.1.2",
- "abort-controller": "^1.1.0",
- "abortcontroller-polyfill": "^1.3.0",
- "babel-core": "^6.26.3",
- "babel-plugin-istanbul": "^4.1.6",
- "babel-preset-env": "^1.6.1",
- "babel-register": "^6.16.3",
- "chai": "^3.5.0",
+ "abort-controller": "^3.0.0",
+ "abortcontroller-polyfill": "^1.7.1",
+ "busboy": "^1.4.0",
+ "c8": "^7.7.2",
+ "chai": "^4.3.4",
"chai-as-promised": "^7.1.1",
- "chai-iterator": "^1.1.1",
- "chai-string": "~1.3.0",
- "codecov": "^3.3.0",
- "cross-env": "^5.2.0",
- "form-data": "^2.3.3",
- "is-builtin-module": "^1.0.0",
- "mocha": "^5.0.0",
- "nyc": "11.9.0",
- "parted": "^0.1.1",
- "promise": "^8.0.3",
- "resumer": "0.0.0",
- "rollup": "^0.63.4",
- "rollup-plugin-babel": "^3.0.7",
- "string-to-arraybuffer": "^1.0.2",
- "whatwg-url": "^5.0.0"
+ "chai-iterator": "^3.0.2",
+ "chai-string": "^1.5.0",
+ "coveralls": "^3.1.0",
+ "form-data": "^4.0.0",
+ "formdata-node": "^4.2.4",
+ "mocha": "^9.1.3",
+ "p-timeout": "^5.0.0",
+ "stream-consumers": "^1.0.1",
+ "tsd": "^0.14.0",
+ "xo": "^0.39.1"
+ },
+ "dependencies": {
+ "data-uri-to-buffer": "^4.0.0",
+ "fetch-blob": "^3.1.4",
+ "formdata-polyfill": "^4.0.10"
+ },
+ "tsd": {
+ "cwd": "@types",
+ "compilerOptions": {
+ "esModuleInterop": true
+ }
+ },
+ "xo": {
+ "envs": [
+ "node",
+ "browser"
+ ],
+ "ignores": [
+ "example.js"
+ ],
+ "rules": {
+ "complexity": 0,
+ "import/extensions": 0,
+ "import/no-useless-path-segments": 0,
+ "import/no-anonymous-default-export": 0,
+ "import/no-named-as-default": 0,
+ "unicorn/import-index": 0,
+ "unicorn/no-array-reduce": 0,
+ "unicorn/prefer-node-protocol": 0,
+ "unicorn/numeric-separators-style": 0,
+ "unicorn/explicit-length-check": 0,
+ "capitalized-comments": 0,
+ "node/no-unsupported-features/es-syntax": 0,
+ "@typescript-eslint/member-ordering": 0
+ },
+ "overrides": [
+ {
+ "files": "test/**/*.js",
+ "envs": [
+ "node",
+ "mocha"
+ ],
+ "rules": {
+ "max-nested-callbacks": 0,
+ "no-unused-expressions": 0,
+ "no-warning-comments": 0,
+ "new-cap": 0,
+ "guard-for-in": 0,
+ "unicorn/no-array-for-each": 0,
+ "unicorn/prevent-abbreviations": 0,
+ "promise/prefer-await-to-then": 0,
+ "ava/no-import-test-files": 0
+ }
+ }
+ ]
},
- "dependencies": {}
+ "runkitExampleFilename": "example.js",
+ "release": {
+ "branches": [
+ "+([0-9]).x",
+ "main",
+ "next",
+ {
+ "name": "beta",
+ "prerelease": true
+ }
+ ]
+ }
}
diff --git a/rollup.config.js b/rollup.config.js
deleted file mode 100644
index a201ee455..000000000
--- a/rollup.config.js
+++ /dev/null
@@ -1,27 +0,0 @@
-import isBuiltin from 'is-builtin-module';
-import babel from 'rollup-plugin-babel';
-import tweakDefault from './build/rollup-plugin';
-
-process.env.BABEL_ENV = 'rollup';
-
-export default {
- input: 'src/index.js',
- output: [
- { file: 'lib/index.js', format: 'cjs', exports: 'named' },
- { file: 'lib/index.es.js', format: 'es', exports: 'named', intro: 'process.emitWarning("The .es.js file is deprecated. Use .mjs instead.");' },
- { file: 'lib/index.mjs', format: 'es', exports: 'named' },
- ],
- plugins: [
- babel({
- runtimeHelpers: true
- }),
- tweakDefault()
- ],
- external: function (id) {
- if (isBuiltin(id)) {
- return true;
- }
- id = id.split('/').slice(0, id[0] === '@' ? 2 : 1).join('/');
- return !!require('./package.json').dependencies[id];
- }
-};
diff --git a/src/abort-error.js b/src/abort-error.js
deleted file mode 100644
index cbb13caba..000000000
--- a/src/abort-error.js
+++ /dev/null
@@ -1,25 +0,0 @@
-/**
- * abort-error.js
- *
- * AbortError interface for cancelled requests
- */
-
-/**
- * Create AbortError instance
- *
- * @param String message Error message for human
- * @return AbortError
- */
-export default function AbortError(message) {
- Error.call(this, message);
-
- this.type = 'aborted';
- this.message = message;
-
- // hide custom error implementation details from end-users
- Error.captureStackTrace(this, this.constructor);
-}
-
-AbortError.prototype = Object.create(Error.prototype);
-AbortError.prototype.constructor = AbortError;
-AbortError.prototype.name = 'AbortError';
diff --git a/src/blob.js b/src/blob.js
deleted file mode 100644
index e1151a955..000000000
--- a/src/blob.js
+++ /dev/null
@@ -1,119 +0,0 @@
-// Based on https://github.com/tmpvar/jsdom/blob/aa85b2abf07766ff7bf5c1f6daafb3726f2f2db5/lib/jsdom/living/blob.js
-// (MIT licensed)
-
-import Stream from 'stream';
-
-// fix for "Readable" isn't a named export issue
-const Readable = Stream.Readable;
-
-export const BUFFER = Symbol('buffer');
-const TYPE = Symbol('type');
-
-export default class Blob {
- constructor() {
- this[TYPE] = '';
-
- const blobParts = arguments[0];
- const options = arguments[1];
-
- const buffers = [];
- let size = 0;
-
- if (blobParts) {
- const a = blobParts;
- const length = Number(a.length);
- for (let i = 0; i < length; i++) {
- const element = a[i];
- let buffer;
- if (element instanceof Buffer) {
- buffer = element;
- } else if (ArrayBuffer.isView(element)) {
- buffer = Buffer.from(element.buffer, element.byteOffset, element.byteLength);
- } else if (element instanceof ArrayBuffer) {
- buffer = Buffer.from(element);
- } else if (element instanceof Blob) {
- buffer = element[BUFFER];
- } else {
- buffer = Buffer.from(typeof element === 'string' ? element : String(element));
- }
- size += buffer.length;
- buffers.push(buffer);
- }
- }
-
- this[BUFFER] = Buffer.concat(buffers);
-
- let type = options && options.type !== undefined && String(options.type).toLowerCase();
- if (type && !/[^\u0020-\u007E]/.test(type)) {
- this[TYPE] = type;
- }
- }
- get size() {
- return this[BUFFER].length;
- }
- get type() {
- return this[TYPE];
- }
- text() {
- return Promise.resolve(this[BUFFER].toString())
- }
- arrayBuffer() {
- const buf = this[BUFFER];
- const ab = buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength);
- return Promise.resolve(ab);
- }
- stream() {
- const readable = new Readable();
- readable._read = () => {};
- readable.push(this[BUFFER]);
- readable.push(null);
- return readable;
- }
- toString() {
- return '[object Blob]'
- }
- slice() {
- const size = this.size;
-
- const start = arguments[0];
- const end = arguments[1];
- let relativeStart, relativeEnd;
- if (start === undefined) {
- relativeStart = 0;
- } else if (start < 0) {
- relativeStart = Math.max(size + start, 0);
- } else {
- relativeStart = Math.min(start, size);
- }
- if (end === undefined) {
- relativeEnd = size;
- } else if (end < 0) {
- relativeEnd = Math.max(size + end, 0);
- } else {
- relativeEnd = Math.min(end, size);
- }
- const span = Math.max(relativeEnd - relativeStart, 0);
-
- const buffer = this[BUFFER];
- const slicedBuffer = buffer.slice(
- relativeStart,
- relativeStart + span
- );
- const blob = new Blob([], { type: arguments[2] });
- blob[BUFFER] = slicedBuffer;
- return blob;
- }
-}
-
-Object.defineProperties(Blob.prototype, {
- size: { enumerable: true },
- type: { enumerable: true },
- slice: { enumerable: true }
-});
-
-Object.defineProperty(Blob.prototype, Symbol.toStringTag, {
- value: 'Blob',
- writable: false,
- enumerable: false,
- configurable: true
-});
diff --git a/src/body.js b/src/body.js
index a9d2e7973..714e27ec4 100644
--- a/src/body.js
+++ b/src/body.js
@@ -1,23 +1,24 @@
/**
- * body.js
+ * Body.js
*
* Body interface provides common methods for Request and Response
*/
-import Stream from 'stream';
+import Stream, {PassThrough} from 'node:stream';
+import {types, deprecate, promisify} from 'node:util';
+import {Buffer} from 'node:buffer';
-import Blob, { BUFFER } from './blob.js';
-import FetchError from './fetch-error.js';
+import Blob from 'fetch-blob';
+import {FormData, formDataToBlob} from 'formdata-polyfill/esm.min.js';
-let convert;
-try { convert = require('encoding').convert; } catch(e) {}
+import {FetchError} from './errors/fetch-error.js';
+import {FetchBaseError} from './errors/base.js';
+import {isBlob, isURLSearchParameters} from './utils/is.js';
+const pipeline = promisify(Stream.pipeline);
const INTERNALS = Symbol('Body internals');
-// fix an issue where "PassThrough" isn't a named export for node <10
-const PassThrough = Stream.PassThrough;
-
/**
* Body mixin
*
@@ -27,110 +28,136 @@ const PassThrough = Stream.PassThrough;
* @param Object opts Response options
* @return Void
*/
-export default function Body(body, {
- size = 0,
- timeout = 0
-} = {}) {
- if (body == null) {
- // body is undefined or null
- body = null;
- } else if (isURLSearchParams(body)) {
- // body is a URLSearchParams
- body = Buffer.from(body.toString());
- } else if (isBlob(body)) {
- // body is blob
- } else if (Buffer.isBuffer(body)) {
- // body is Buffer
- } else if (Object.prototype.toString.call(body) === '[object ArrayBuffer]') {
- // body is ArrayBuffer
- body = Buffer.from(body);
- } else if (ArrayBuffer.isView(body)) {
- // body is ArrayBufferView
- body = Buffer.from(body.buffer, body.byteOffset, body.byteLength);
- } else if (body instanceof Stream) {
- // body is stream
- } else {
- // none of the above
- // coerce to string then buffer
- body = Buffer.from(String(body));
- }
- this[INTERNALS] = {
- body,
- disturbed: false,
- error: null
- };
- this.size = size;
- this.timeout = timeout;
+export default class Body {
+ constructor(body, {
+ size = 0
+ } = {}) {
+ let boundary = null;
+
+ if (body === null) {
+ // Body is undefined or null
+ body = null;
+ } else if (isURLSearchParameters(body)) {
+ // Body is a URLSearchParams
+ body = Buffer.from(body.toString());
+ } else if (isBlob(body)) {
+ // Body is blob
+ } else if (Buffer.isBuffer(body)) {
+ // Body is Buffer
+ } else if (types.isAnyArrayBuffer(body)) {
+ // Body is ArrayBuffer
+ body = Buffer.from(body);
+ } else if (ArrayBuffer.isView(body)) {
+ // Body is ArrayBufferView
+ body = Buffer.from(body.buffer, body.byteOffset, body.byteLength);
+ } else if (body instanceof Stream) {
+ // Body is stream
+ } else if (body instanceof FormData) {
+ // Body is FormData
+ body = formDataToBlob(body);
+ boundary = body.type.split('=')[1];
+ } else {
+ // None of the above
+ // coerce to string then buffer
+ body = Buffer.from(String(body));
+ }
- if (body instanceof Stream) {
- body.on('error', err => {
- const error = err.name === 'AbortError'
- ? err
- : new FetchError(`Invalid response body while trying to fetch ${this.url}: ${err.message}`, 'system', err);
- this[INTERNALS].error = error;
- });
+ let stream = body;
+
+ if (Buffer.isBuffer(body)) {
+ stream = Stream.Readable.from(body);
+ } else if (isBlob(body)) {
+ stream = Stream.Readable.from(body.stream());
+ }
+
+ this[INTERNALS] = {
+ body,
+ stream,
+ boundary,
+ disturbed: false,
+ error: null
+ };
+ this.size = size;
+
+ if (body instanceof Stream) {
+ body.on('error', error_ => {
+ const error = error_ instanceof FetchBaseError ?
+ error_ :
+ new FetchError(`Invalid response body while trying to fetch ${this.url}: ${error_.message}`, 'system', error_);
+ this[INTERNALS].error = error;
+ });
+ }
}
-}
-Body.prototype = {
get body() {
- return this[INTERNALS].body;
- },
+ return this[INTERNALS].stream;
+ }
get bodyUsed() {
return this[INTERNALS].disturbed;
- },
+ }
/**
* Decode response as ArrayBuffer
*
* @return Promise
*/
- arrayBuffer() {
- return consumeBody.call(this).then(buf => buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength));
- },
+ async arrayBuffer() {
+ const {buffer, byteOffset, byteLength} = await consumeBody(this);
+ return buffer.slice(byteOffset, byteOffset + byteLength);
+ }
+
+ async formData() {
+ const ct = this.headers.get('content-type');
+
+ if (ct.startsWith('application/x-www-form-urlencoded')) {
+ const formData = new FormData();
+ const parameters = new URLSearchParams(await this.text());
+
+ for (const [name, value] of parameters) {
+ formData.append(name, value);
+ }
+
+ return formData;
+ }
+
+ const {toFormData} = await import('./utils/multipart-parser.js');
+ return toFormData(this.body, ct);
+ }
/**
* Return raw response as Blob
*
* @return Promise
*/
- blob() {
- let ct = this.headers && this.headers.get('content-type') || '';
- return consumeBody.call(this).then(buf => Object.assign(
- // Prevent copying
- new Blob([], {
- type: ct.toLowerCase()
- }),
- {
- [BUFFER]: buf
- }
- ));
- },
+ async blob() {
+ const ct = (this.headers && this.headers.get('content-type')) || (this[INTERNALS].body && this[INTERNALS].body.type) || '';
+ const buf = await this.arrayBuffer();
+
+ return new Blob([buf], {
+ type: ct
+ });
+ }
/**
* Decode response as json
*
* @return Promise
*/
- json() {
- return consumeBody.call(this).then((buffer) => {
- try {
- return JSON.parse(buffer.toString());
- } catch (err) {
- return Body.Promise.reject(new FetchError(`invalid json response body at ${this.url} reason: ${err.message}`, 'invalid-json'));
- }
- })
- },
+ async json() {
+ const text = await this.text();
+ return JSON.parse(text);
+ }
/**
* Decode response as text
*
* @return Promise
*/
- text() {
- return consumeBody.call(this).then(buffer => buffer.toString());
- },
+ async text() {
+ const buffer = await consumeBody(this);
+ return new TextDecoder().decode(buffer);
+ }
/**
* Decode response as buffer (non-spec api)
@@ -138,281 +165,129 @@ Body.prototype = {
* @return Promise
*/
buffer() {
- return consumeBody.call(this);
- },
-
- /**
- * Decode response as text, while automatically detecting the encoding and
- * trying to decode to UTF-8 (non-spec api)
- *
- * @return Promise
- */
- textConverted() {
- return consumeBody.call(this).then(buffer => convertBody(buffer, this.headers));
+ return consumeBody(this);
}
-};
+}
+
+Body.prototype.buffer = deprecate(Body.prototype.buffer, 'Please use \'response.arrayBuffer()\' instead of \'response.buffer()\'', 'node-fetch#buffer');
// In browsers, all properties are enumerable.
Object.defineProperties(Body.prototype, {
- body: { enumerable: true },
- bodyUsed: { enumerable: true },
- arrayBuffer: { enumerable: true },
- blob: { enumerable: true },
- json: { enumerable: true },
- text: { enumerable: true }
+ body: {enumerable: true},
+ bodyUsed: {enumerable: true},
+ arrayBuffer: {enumerable: true},
+ blob: {enumerable: true},
+ json: {enumerable: true},
+ text: {enumerable: true},
+ data: {get: deprecate(() => {},
+ 'data doesn\'t exist, use json(), text(), arrayBuffer(), or body instead',
+ 'https://github.com/node-fetch/node-fetch/issues/1000 (response)')}
});
-Body.mixIn = function (proto) {
- for (const name of Object.getOwnPropertyNames(Body.prototype)) {
- // istanbul ignore else: future proof
- if (!(name in proto)) {
- const desc = Object.getOwnPropertyDescriptor(Body.prototype, name);
- Object.defineProperty(proto, name, desc);
- }
- }
-};
-
/**
* Consume and convert an entire Body to a Buffer.
*
* Ref: https://fetch.spec.whatwg.org/#concept-body-consume-body
*
- * @return Promise
+ * @return Promise
*/
-function consumeBody() {
- if (this[INTERNALS].disturbed) {
- return Body.Promise.reject(new TypeError(`body used already for: ${this.url}`));
+async function consumeBody(data) {
+ if (data[INTERNALS].disturbed) {
+ throw new TypeError(`body used already for: ${data.url}`);
}
- this[INTERNALS].disturbed = true;
+ data[INTERNALS].disturbed = true;
- if (this[INTERNALS].error) {
- return Body.Promise.reject(this[INTERNALS].error);
+ if (data[INTERNALS].error) {
+ throw data[INTERNALS].error;
}
- let body = this.body;
+ const {body} = data;
- // body is null
+ // Body is null
if (body === null) {
- return Body.Promise.resolve(Buffer.alloc(0));
- }
-
- // body is blob
- if (isBlob(body)) {
- body = body.stream();
+ return Buffer.alloc(0);
}
- // body is buffer
- if (Buffer.isBuffer(body)) {
- return Body.Promise.resolve(body);
- }
-
- // istanbul ignore if: should never happen
+ /* c8 ignore next 3 */
if (!(body instanceof Stream)) {
- return Body.Promise.resolve(Buffer.alloc(0));
+ return Buffer.alloc(0);
}
- // body is stream
+ // Body is stream
// get ready to actually consume the body
- let accum = [];
+ const accum = [];
let accumBytes = 0;
- let abort = false;
-
- return new Body.Promise((resolve, reject) => {
- let resTimeout;
-
- // allow timeout on slow response body
- if (this.timeout) {
- resTimeout = setTimeout(() => {
- abort = true;
- reject(new FetchError(`Response timeout while trying to fetch ${this.url} (over ${this.timeout}ms)`, 'body-timeout'));
- }, this.timeout);
- }
-
- // handle stream errors
- body.on('error', err => {
- if (err.name === 'AbortError') {
- // if the request was aborted, reject with this Error
- abort = true;
- reject(err);
- } else {
- // other errors, such as incorrect content-encoding
- reject(new FetchError(`Invalid response body while trying to fetch ${this.url}: ${err.message}`, 'system', err));
- }
- });
- body.on('data', chunk => {
- if (abort || chunk === null) {
- return;
- }
-
- if (this.size && accumBytes + chunk.length > this.size) {
- abort = true;
- reject(new FetchError(`content size at ${this.url} over limit: ${this.size}`, 'max-size'));
- return;
+ try {
+ for await (const chunk of body) {
+ if (data.size > 0 && accumBytes + chunk.length > data.size) {
+ const error = new FetchError(`content size at ${data.url} over limit: ${data.size}`, 'max-size');
+ body.destroy(error);
+ throw error;
}
accumBytes += chunk.length;
accum.push(chunk);
- });
-
- body.on('end', () => {
- if (abort) {
- return;
- }
-
- clearTimeout(resTimeout);
-
- try {
- resolve(Buffer.concat(accum, accumBytes));
- } catch (err) {
- // handle streams that have accumulated too much data (issue #414)
- reject(new FetchError(`Could not create Buffer from response body for ${this.url}: ${err.message}`, 'system', err));
- }
- });
- });
-}
-
-/**
- * Detect buffer encoding and convert to target encoding
- * ref: http://www.w3.org/TR/2011/WD-html5-20110113/parsing.html#determining-the-character-encoding
- *
- * @param Buffer buffer Incoming buffer
- * @param String encoding Target encoding
- * @return String
- */
-function convertBody(buffer, headers) {
- if (typeof convert !== 'function') {
- throw new Error('The package `encoding` must be installed to use the textConverted() function');
- }
-
- const ct = headers.get('content-type');
- let charset = 'utf-8';
- let res, str;
-
- // header
- if (ct) {
- res = /charset=([^;]*)/i.exec(ct);
- }
-
- // no charset in content type, peek at response body for at most 1024 bytes
- str = buffer.slice(0, 1024).toString();
-
- // html5
- if (!res && str) {
- res = /Code example
-Identical to `body.text()`, except instead of always converting to UTF-8, encoding sniffing will be performed and text converted to UTF-8 if possible. +```js +import http from 'node:http' +import { Response } from 'node-fetch' + +http.createServer(async function (req, res) { + const formData = await new Response(req, { + headers: req.headers // Pass along the boundary value + }).formData() + const allFields = [...formData] + + const file = formData.get('uploaded-files') + const arrayBuffer = await file.arrayBuffer() + const text = await file.text() + const whatwgReadableStream = file.stream() + + // other was to consume the request could be to do: + const json = await new Response(req).json() + const text = await new Response(req).text() + const arrayBuffer = await new Response(req).arrayBuffer() + const blob = await new Response(req, { + headers: req.headers // So that `type` inherits `Content-Type` + }.blob() +}) +``` -(This API requires an optional dependency of the npm package [encoding](https://www.npmjs.com/package/encoding), which you need to install manually. `webpack` users may see [a warning message](https://github.com/bitinn/node-fetch/issues/412#issuecomment-379007792) due to this optional dependency.) +日本語
');
- });
- });
- });
-
- it('should support encoding decode, html5 detect', function() {
- const url = `${base}encoding/gbk`;
- return fetch(url).then(res => {
- expect(res.status).to.equal(200);
- return res.textConverted().then(result => {
- expect(result).to.equal('中文
');
- });
- });
- });
-
- it('should support encoding decode, html4 detect', function() {
- const url = `${base}encoding/gb2312`;
- return fetch(url).then(res => {
- expect(res.status).to.equal(200);
- return res.textConverted().then(result => {
- expect(result).to.equal('中文
');
- });
- });
- });
-
- it('should support encoding decode, html4 detect reverse http-equiv', function() {
- const url = `${base}encoding/gb2312-reverse`;
- return fetch(url).then(res => {
- expect(res.status).to.equal(200);
- return res.textConverted().then(result => {
- expect(result).to.equal('中文
');
- });
- });
- });
-
- it('should default to utf8 encoding', function() {
- const url = `${base}encoding/utf8`;
- return fetch(url).then(res => {
- expect(res.status).to.equal(200);
- expect(res.headers.get('content-type')).to.be.null;
- return res.textConverted().then(result => {
- expect(result).to.equal('中文');
- });
- });
- });
-
- it('should support uncommon content-type order, charset in front', function() {
- const url = `${base}encoding/order1`;
- return fetch(url).then(res => {
- expect(res.status).to.equal(200);
- return res.textConverted().then(result => {
- expect(result).to.equal('中文');
- });
- });
- });
-
- it('should support uncommon content-type order, end with qs', function() {
- const url = `${base}encoding/order2`;
- return fetch(url).then(res => {
- expect(res.status).to.equal(200);
- return res.textConverted().then(result => {
- expect(result).to.equal('中文');
- });
- });
- });
-
- it('should support chunked encoding, html4 detect', function() {
- const url = `${base}encoding/chunked`;
- return fetch(url).then(res => {
- expect(res.status).to.equal(200);
- const padding = 'a'.repeat(10);
- return res.textConverted().then(result => {
- expect(result).to.equal(`${padding}日本語
`);
- });
- });
- });
-
- it('should only do encoding detection up to 1024 bytes', function() {
- const url = `${base}encoding/invalid`;
- return fetch(url).then(res => {
- expect(res.status).to.equal(200);
- const padding = 'a'.repeat(1200);
- return res.textConverted().then(result => {
- expect(result).to.not.equal(`${padding}中文`);
- });
- });
- });
- });
-
- describe('without optional `encoding`', function() {
- before(function() {
- if (hasEncoding) this.skip()
- });
-
- it('should throw a FetchError if res.textConverted() is called without `encoding` in require cache', () => {
- const url = `${base}hello`;
- return fetch(url).then((res) => {
- return expect(res.textConverted()).to.eventually.be.rejected
- .and.have.property('message').which.includes('encoding')
- });
- });
- });
-
- describe('data uri', function() {
- const dataUrl = 'data:image/gif;base64,R0lGODlhAQABAIAAAAUEBAAAACwAAAAAAQABAAACAkQBADs=';
-
- const invalidDataUrl = 'data:@@@@';
-
- it('should accept data uri', function() {
- return fetch(dataUrl).then(r => {
- console.assert(r.status == 200);
- console.assert(r.headers.get('Content-Type') == 'image/gif');
-
- return r.buffer().then(b => {
- console.assert(b instanceof Buffer);
- });
- });
- });
-
- it('should reject invalid data uri', function() {
- return fetch(invalidDataUrl)
- .catch(e => {
- console.assert(e);
- console.assert(e.message.includes('invalid URL'));
- });
- });
- });
-});
diff --git a/test/utils/chai-timeout.js b/test/utils/chai-timeout.js
new file mode 100644
index 000000000..6838da347
--- /dev/null
+++ b/test/utils/chai-timeout.js
@@ -0,0 +1,15 @@
+import pTimeout from 'p-timeout';
+
+export default ({Assertion}, utils) => {
+ utils.addProperty(Assertion.prototype, 'timeout', async function () {
+ let timeouted = false;
+ await pTimeout(this._obj, 150, () => {
+ timeouted = true;
+ });
+ return this.assert(
+ timeouted,
+ 'expected promise to timeout but it was resolved',
+ 'expected promise not to timeout but it timed out'
+ );
+ });
+};
diff --git a/test/dummy.txt b/test/utils/dummy.txt
similarity index 100%
rename from test/dummy.txt
rename to test/utils/dummy.txt
diff --git a/test/server.js b/test/utils/server.js
similarity index 52%
rename from test/server.js
rename to test/utils/server.js
index 06c715d65..6ee3ef33b 100644
--- a/test/server.js
+++ b/test/utils/server.js
@@ -1,38 +1,60 @@
-import * as http from 'http';
-import { parse } from 'url';
-import * as zlib from 'zlib';
-import * as stream from 'stream';
-import { multipart as Multipart } from 'parted';
-
-let convert;
-try { convert = require('encoding').convert; } catch(e) {}
+import http from 'node:http';
+import zlib from 'node:zlib';
+import {once} from 'node:events';
+import busboy from 'busboy';
export default class TestServer {
- constructor() {
+ constructor(hostname) {
this.server = http.createServer(this.router);
- this.port = 30001;
- this.hostname = 'localhost';
- // node 8 default keepalive timeout is 5000ms
+ // Node 8 default keepalive timeout is 5000ms
// make it shorter here as we want to close server quickly at the end of tests
this.server.keepAliveTimeout = 1000;
- this.server.on('error', function(err) {
+ this.server.on('error', err => {
console.log(err.stack);
});
- this.server.on('connection', function(socket) {
+ this.server.on('connection', socket => {
socket.setTimeout(1500);
});
+ this.hostname = hostname || 'localhost';
+ }
+
+ async start() {
+ let host = this.hostname;
+ if (host.startsWith('[')) {
+ // If we're trying to listen on an IPv6 literal hostname, strip the
+ // square brackets before binding to the IPv6 address
+ host = host.slice(1, -1);
+ }
+
+ this.server.listen(0, host);
+ return once(this.server, 'listening');
+ }
+
+ async stop() {
+ this.server.close();
+ return once(this.server, 'close');
}
- start(cb) {
- this.server.listen(this.port, this.hostname, cb);
+ get port() {
+ return this.server.address().port;
}
- stop(cb) {
- this.server.close(cb);
+ mockResponse(responseHandler) {
+ this.server.nextResponseHandler = responseHandler;
+ return `http://${this.hostname}:${this.port}/mocked`;
}
- router(req, res) {
- let p = parse(req.url).pathname;
+ router(request, res) {
+ const p = request.url;
+
+ if (p === '/mocked') {
+ if (this.nextResponseHandler) {
+ this.nextResponseHandler(res);
+ this.nextResponseHandler = undefined;
+ } else {
+ throw new Error('No mocked response. Use ’TestServer.mockResponse()’.');
+ }
+ }
if (p === '/hello') {
res.statusCode = 200;
@@ -40,12 +62,22 @@ export default class TestServer {
res.end('world');
}
+ if (p.includes('question')) {
+ res.statusCode = 200;
+ res.setHeader('Content-Type', 'text/plain');
+ res.end('ok');
+ }
+
if (p === '/plain') {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.end('text');
}
+ if (p === '/no-status-text') {
+ res.writeHead(200, '', {}).end();
+ }
+
if (p === '/options') {
res.statusCode = 200;
res.setHeader('Allow', 'GET, HEAD, OPTIONS');
@@ -70,7 +102,11 @@ export default class TestServer {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.setHeader('Content-Encoding', 'gzip');
- zlib.gzip('hello world', function(err, buffer) {
+ zlib.gzip('hello world', (err, buffer) => {
+ if (err) {
+ throw err;
+ }
+
res.end(buffer);
});
}
@@ -79,9 +115,26 @@ export default class TestServer {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.setHeader('Content-Encoding', 'gzip');
- zlib.gzip('hello world', function(err, buffer) {
- // truncate the CRC checksum and size check at the end of the stream
- res.end(buffer.slice(0, buffer.length - 8));
+ zlib.gzip('hello world', (err, buffer) => {
+ if (err) {
+ throw err;
+ }
+
+ // Truncate the CRC checksum and size check at the end of the stream
+ res.end(buffer.slice(0, -8));
+ });
+ }
+
+ if (p === '/gzip-capital') {
+ res.statusCode = 200;
+ res.setHeader('Content-Type', 'text/plain');
+ res.setHeader('Content-Encoding', 'GZip');
+ zlib.gzip('hello world', (err, buffer) => {
+ if (err) {
+ throw err;
+ }
+
+ res.end(buffer);
});
}
@@ -89,7 +142,11 @@ export default class TestServer {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.setHeader('Content-Encoding', 'deflate');
- zlib.deflate('hello world', function(err, buffer) {
+ zlib.deflate('hello world', (err, buffer) => {
+ if (err) {
+ throw err;
+ }
+
res.end(buffer);
});
}
@@ -99,22 +156,36 @@ export default class TestServer {
res.setHeader('Content-Type', 'text/plain');
if (typeof zlib.createBrotliDecompress === 'function') {
res.setHeader('Content-Encoding', 'br');
- zlib.brotliCompress('hello world', function (err, buffer) {
+ zlib.brotliCompress('hello world', (err, buffer) => {
+ if (err) {
+ throw err;
+ }
+
res.end(buffer);
});
}
}
-
if (p === '/deflate-raw') {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.setHeader('Content-Encoding', 'deflate');
- zlib.deflateRaw('hello world', function(err, buffer) {
+ zlib.deflateRaw('hello world', (err, buffer) => {
+ if (err) {
+ throw err;
+ }
+
res.end(buffer);
});
}
+ if (p === '/empty/deflate') {
+ res.statusCode = 200;
+ res.setHeader('Content-Type', 'text/plain');
+ res.setHeader('Content-Encoding', 'deflate');
+ res.end();
+ }
+
if (p === '/sdch') {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
@@ -130,7 +201,7 @@ export default class TestServer {
}
if (p === '/timeout') {
- setTimeout(function() {
+ setTimeout(() => {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.end('text');
@@ -141,7 +212,7 @@ export default class TestServer {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.write('test');
- setTimeout(function() {
+ setTimeout(() => {
res.end('test');
}, 1000);
}
@@ -155,10 +226,10 @@ export default class TestServer {
if (p === '/size/chunk') {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
- setTimeout(function() {
+ setTimeout(() => {
res.write('test');
}, 10);
- setTimeout(function() {
+ setTimeout(() => {
res.end('test');
}, 20);
}
@@ -169,72 +240,27 @@ export default class TestServer {
res.end('testtest');
}
- if (p === '/encoding/gbk') {
- res.statusCode = 200;
- res.setHeader('Content-Type', 'text/html');
- res.end(convert('中文
', 'gbk'));
- }
-
- if (p === '/encoding/gb2312') {
- res.statusCode = 200;
- res.setHeader('Content-Type', 'text/html');
- res.end(convert('中文
', 'gb2312'));
- }
-
- if (p === '/encoding/gb2312-reverse') {
- res.statusCode = 200;
- res.setHeader('Content-Type', 'text/html');
- res.end(convert('中文
', 'gb2312'));
- }
-
- if (p === '/encoding/shift-jis') {
- res.statusCode = 200;
- res.setHeader('Content-Type', 'text/html; charset=Shift-JIS');
- res.end(convert('日本語
', 'Shift_JIS'));
- }
-
- if (p === '/encoding/euc-jp') {
- res.statusCode = 200;
- res.setHeader('Content-Type', 'text/xml');
- res.end(convert('日本語
', 'Shift_JIS'));
+ if (p === '/redirect/301/invalid') {
+ res.statusCode = 301;
+ res.setHeader('Location', '//super:invalid:url%/');
+ res.end();
}
- if (p === '/encoding/invalid') {
- res.statusCode = 200;
- res.setHeader('Content-Type', 'text/html');
- res.setHeader('Transfer-Encoding', 'chunked');
- res.write('a'.repeat(1200));
- res.end(convert('中文', 'gbk'));
+ if (p.startsWith('/redirect-to/3')) {
+ res.statusCode = p.slice(13, 16);
+ res.setHeader('Location', p.slice(17));
+ res.end();
}
- if (p === '/redirect/301') {
+ if (p === '/redirect/301/otherhost') {
res.statusCode = 301;
- res.setHeader('Location', '/inspect');
+ res.setHeader('Location', 'https://github.com/node-fetch');
res.end();
}
@@ -276,7 +302,7 @@ export default class TestServer {
if (p === '/redirect/slow') {
res.statusCode = 301;
res.setHeader('Location', '/redirect/301');
- setTimeout(function() {
+ setTimeout(() => {
res.end();
}, 1000);
}
@@ -284,7 +310,7 @@ export default class TestServer {
if (p === '/redirect/slow-chain') {
res.statusCode = 301;
res.setHeader('Location', '/redirect/slow');
- setTimeout(function() {
+ setTimeout(() => {
res.end();
}, 10);
}
@@ -295,6 +321,33 @@ export default class TestServer {
res.end();
}
+ if (p === '/redirect/bad-location') {
+ res.socket.write('HTTP/1.1 301\r\nLocation: <>\r\nContent-Length: 0\r\n');
+ res.socket.end('\r\n');
+ }
+
+ if (p === '/redirect/referrer-policy') {
+ res.statusCode = 301;
+ res.setHeader('Location', '/inspect');
+ res.setHeader('Referrer-Policy', 'foo unsafe-url bar');
+ res.end();
+ }
+
+ if (p === '/redirect/referrer-policy/same-origin') {
+ res.statusCode = 301;
+ res.setHeader('Location', '/inspect');
+ res.setHeader('Referrer-Policy', 'foo unsafe-url same-origin bar');
+ res.end();
+ }
+
+ if (p === '/redirect/chunked') {
+ res.writeHead(301, {
+ Location: '/inspect',
+ 'Transfer-Encoding': 'chunked'
+ });
+ setTimeout(() => res.end(), 10);
+ }
+
if (p === '/error/400') {
res.statusCode = 400;
res.setHeader('Content-Type', 'text/plain');
@@ -317,6 +370,49 @@ export default class TestServer {
res.destroy();
}
+ if (p === '/error/premature') {
+ res.writeHead(200, {'content-length': 50});
+ res.write('foo');
+ setTimeout(() => {
+ res.destroy();
+ }, 100);
+ }
+
+ if (p === '/error/premature/chunked') {
+ res.writeHead(200, {
+ 'Content-Type': 'application/json',
+ 'Transfer-Encoding': 'chunked'
+ });
+
+ res.write(`${JSON.stringify({data: 'hi'})}\n`);
+
+ setTimeout(() => {
+ res.write(`${JSON.stringify({data: 'bye'})}\n`);
+ }, 200);
+
+ setTimeout(() => {
+ res.destroy();
+ }, 400);
+ }
+
+ if (p === '/chunked/split-ending') {
+ res.socket.write('HTTP/1.1 200\r\nTransfer-Encoding: chunked\r\n\r\n');
+ res.socket.write('3\r\nfoo\r\n3\r\nbar\r\n');
+
+ setTimeout(() => {
+ res.socket.write('0\r\n');
+ }, 10);
+
+ setTimeout(() => {
+ res.socket.end('\r\n');
+ }, 20);
+ }
+
+ if (p === '/chunked/multiple-ending') {
+ res.socket.write('HTTP/1.1 200\r\nTransfer-Encoding: chunked\r\n\r\n');
+ res.socket.write('3\r\nfoo\r\n3\r\nbar\r\n0\r\n\r\n');
+ }
+
if (p === '/error/json') {
res.statusCode = 200;
res.setHeader('Content-Type', 'application/json');
@@ -355,41 +451,48 @@ export default class TestServer {
res.statusCode = 200;
res.setHeader('Content-Type', 'application/json');
let body = '';
- req.on('data', function(c) { body += c });
- req.on('end', function() {
+ request.on('data', c => {
+ body += c;
+ });
+ request.on('end', () => {
res.end(JSON.stringify({
- method: req.method,
- url: req.url,
- headers: req.headers,
+ method: request.method,
+ url: request.url,
+ headers: request.headers,
body
}));
});
}
if (p === '/multipart') {
+ let body = '';
res.statusCode = 200;
res.setHeader('Content-Type', 'application/json');
- const parser = new Multipart(req.headers['content-type']);
- let body = '';
- parser.on('part', function(field, part) {
- body += field + '=' + part;
+ const bb = busboy({headers: request.headers});
+ bb.on('file', async (fieldName, file, info) => {
+ body += `${fieldName}=${info.filename}`;
+ // consume file data
+ // eslint-disable-next-line no-empty, no-unused-vars
+ for await (const c of file) {}
+ });
+ bb.on('field', (fieldName, value) => {
+ body += `${fieldName}=${value}`;
});
- parser.on('end', function() {
+ bb.on('close', () => {
res.end(JSON.stringify({
- method: req.method,
- url: req.url,
- headers: req.headers,
- body: body
+ method: request.method,
+ url: request.url,
+ headers: request.headers,
+ body
}));
});
- req.pipe(parser);
+ request.pipe(bb);
}
- }
-}
-if (require.main === module) {
- const server = new TestServer;
- server.start(() => {
- console.log(`Server started listening at port ${server.port}`);
- });
+ if (p === '/m%C3%B6bius') {
+ res.statusCode = 200;
+ res.setHeader('Content-Type', 'text/plain');
+ res.end('ok');
+ }
+ }
}