Why does Membrane API Gateway return 504 Gateway Timeout with a connect timeout?

Membrane could not set up a connection to the backend within the connect timeout, so no request was sent. The backend host is down or unreachable, a firewall drops the packets, or the handshake with an outgoing proxy or the TLS handshake took too long.

How do I change the connect timeout in Membrane?

Set timeout in httpClientConfig, connection, of the configuration document. The default is 10000 milliseconds.

How do I hide the details in Membrane error responses?

Enable production mode by adding a configuration document with production: true to your apis.yaml. Membrane then leaves out detailed information from error responses.

504 Gateway Timeout: Connection to Backend Timed Out

Membrane API Gateway returns 504 Gateway Timeout with this page when the HTTP client that forwards to the target cannot set up a connection to the backend in time. No request was sent to the backend. The backend, the network, or a timeout that is too short is to blame.

The Error Response

The response contains the status 504 Gateway Timeout, the title Gateway Timeout, the problem type https://membrane-api.io/problems/internal and a link to this page. The type internal means that the gateway could not complete the processing of the request. In production mode, the title is replaced by Internal Server Error and the details are left out.

If the connection was set up but the backend did not answer in time, see socket timeout instead.

Common Causes

The Backend Does Not Answer the Connection Attempt

The backend host does not answer the TCP connection attempt within the connect timeout (timeout, default 10000 ms), e.g. because a firewall drops packets, the host is down, or the IP address is wrong.

A Handshake Takes Too Long

The connection was open, but the CONNECT handshake with an outgoing HTTP proxy or the TLS handshake with the backend did not finish within the connect timeout.

The Retries Are Used Up

The error is reported after the configured retries are used up. By default, connect timeouts are retried for every HTTP method, so the client may wait several times the connect timeout before it gets this response.

How to Fix It

Operator: check that the backend is running and reachable from the gateway host on the configured port.
Operator: check firewalls, security groups and DNS resolution between the gateway and the backend.
Operator: adjust the connect timeout and the retries in httpClientConfig to your network.
Client: retry later, or raise the client's own timeout if it gives up before the gateway does.

configuration:
  httpClientConfig:
    connection:
      timeout: 5000
    retries:
      retries: 2
      delay: 100
      retryOnConnectTimeout: true

configuration:
  httpClientConfig:
    connection:
      timeout: 5000
    retries:
      retries: 2
      delay: 100
      retryOnConnectTimeout: true

Set retryOnConnectTimeout: false to stop retrying connect timeouts, so that the client gets the error sooner.

Hide Details in Production

By default, Membrane runs in development mode and adds details to error responses that help you find the problem. These details may reveal information about your setup. In production, turn them off:

configuration:
  production: true

configuration:
  production: true

Add the configuration document to your apis.yaml and separate it from the APIs with ---.