Why does Membrane API Gateway return 500 with Error sending request to backend?

The HTTP client that forwards the request to the target failed with an error that has no more specific description. Common causes are a backend that closes the connection without answering, a failed TLS handshake, an expression in the target URL that cannot be evaluated, or a client that disconnects during the upload.

What does Network error mean in a Membrane 500 response?

The outgoing HTTP proxy configured in httpClientConfig did not accept the CONNECT request for an HTTPS target, for example because it answered 407 Proxy Authentication Required or 403. The detail shows the reply of the proxy.

How do I hide the details in Membrane error responses?

Enable production mode by adding a configuration document with production: true to your apis.yaml. Membrane then leaves out detailed information from error responses.

500 Internal Server Error: Backend Request Failed

Membrane API Gateway returns 500 Internal Server Error with this page when the HTTP client that forwards the request to the target fails with an error that has no more specific description. Depending on the cause, the backend, the outgoing proxy, the client or the gateway configuration is to blame.

The Error Response

The response contains the status 500 Internal Server Error, the problem type https://membrane-api.io/problems/internal and a link to this page. Two different errors share this URL:

Title or Detail	Meaning
Title Network error	The outgoing HTTP proxy refused to open a tunnel to the target.
Detail Error sending request to backend.	Any other failure while sending the request or reading the response.

The type internal means that the gateway could not complete the processing of the request. In production mode, the title is replaced by Internal Server Error and the details are left out. More specific failures have their own pages, e.g. connect timeouts, read timeouts, closed connections and network errors.

Common Causes

Network Error: The Outgoing Proxy Rejected the Tunnel

For an HTTPS target, Membrane asks the outgoing HTTP proxy configured in httpClientConfig (proxy) to open a tunnel with a CONNECT request. If the proxy does not answer with 200, e.g. with 407 Proxy Authentication Required or 403, the request fails with the title Network error. The detail shows the reply of the proxy.

The Backend Closed the Connection Without Answering

The backend accepted the connection but closed it without sending a single response byte. This is reported after the configured retries are used up. A common case is a kept-alive connection that the backend had already closed when Membrane reused it. Other cases are a crashing or overloaded backend, or a backend that expects TLS while the target uses http://.

The TLS Handshake with the Backend Failed

The backend's certificate is not trusted, has expired or does not match the host name.

An Expression in the Target URL Failed

The target url contains a ${...} expression that could not be evaluated for this request.

The Request Body Could Not Be Read

The client's request body could not be read while it was being streamed to the backend, e.g. because the client disconnected during the upload.

An Unexpected Exception in the HTTP Client

Other unexpected exceptions in the HTTP client land here too and are logged at error level. An exception that the HTTP client does not turn into an error response itself is reported under the same URL by Membrane's generic error handler, with the detail Aborting! Exception caused in … flow. Unless the exception points to your configuration or to the request, this usually indicates a bug in Membrane.

How to Fix It

Outgoing proxy: check host, port, authentication, username and password of the proxy inside httpClientConfig, and that the proxy allows CONNECT to the target host and port.
Backend: check the backend's log at the time of the failure. If the backend closes idle connections before Membrane does, set keepAliveTimeout below the backend's keep-alive timeout. Also check that the target scheme matches what the backend serves.
TLS: configure the trust store in the client's ssl settings so that the backend's certificate chain is accepted.
Expression: check the ${...} template in the target url against the requests it will see.

The outgoing proxy and the keep-alive timeout are set in the httpClientConfig of the configuration:

configuration:
  httpClientConfig:
    proxy:
      host: proxy.example.com
      port: 3128
      authentication: true
      username: alice
      password: secret
    connection:
      keepAliveTimeout: 4000

configuration:
  httpClientConfig:
    proxy:
      host: proxy.example.com
      port: 3128
      authentication: true
      username: alice
      password: secret
    connection:
      keepAliveTimeout: 4000

Enable development mode, or in production look up the log key in the gateway log, to see the exception. If it looks like an internal error, report it with the stack trace on GitHub Issues.

Hide Details in Production

By default, Membrane runs in development mode and adds details to error responses that help you find the problem. These details may reveal information about your setup. In production, turn them off:

configuration:
  production: true

configuration:
  production: true

Add the configuration document to your apis.yaml and separate it from the APIs with ---. The response then only contains a log key. Search the gateway log for this key to find the exception.