Why does the Membrane balancer return 503 Service unavailable?

The balancer plugin could not choose a backend node for the request. Usually no node in the cluster is UP, no nodes are configured, or the cluster is not named Default.

How do I fix a 503 from the Membrane load balancer?

Bring at least one node back UP, check that the clusters configuration contains nodes and a cluster named Default, and with byThreadStrategy raise the thread limits or add nodes. Clients can retry later.

How do I hide the details in Membrane error responses?

Enable production mode by adding a configuration document with production: true to your apis.yaml. Membrane then leaves out detailed information from error responses.

503 Service Unavailable: No Backend Node Available

Membrane API Gateway returns 503 Service Unavailable when the balancer plugin cannot choose a backend node for the request. No backend is contacted. Usually no node is available, which is a backend or configuration problem. Less often, the dispatching strategy or the session id extraction failed.

The Error Response

The response contains the status 503 Service Unavailable, the title Service unavailable, the problem type https://membrane-api.io/problems/internal and a link to this page. The type internal means that the gateway failed while processing the exchange. Two cases share this link.

No Backend Node Is Ready

The detail reads No backend node found that is ready to handle this request. The cluster has no node with status UP.

Could Not Get Dispatched Node

The detail reads Could not get dispatched node! The strategy or a session id extractor failed while selecting the node.

How the Balancer Chooses a Node

The balancer plugin distributes requests over the nodes of a cluster. It only considers nodes with status UP. With several clusters, it uses the cluster named Default:

api:
  port: 2000
  flow:
    - balancer:
        clusters:
          - name: Default
            nodes:
              - host: node1.example.com
                port: 8080
              - host: node2.example.com
                port: 8080

api:
  port: 2000
  flow:
    - balancer:
        clusters:
          - name: Default
            nodes:
              - host: node1.example.com
                port: 8080
              - host: node2.example.com
                port: 8080

Common Causes

All Nodes Are Down

No node in the cluster has status UP. Nodes are taken out of rotation by the balancerHealthMonitor, by the admin console or by clusterNotification messages, or they were never added.

No Nodes Are Configured

The cluster exists, but contains no nodes.

No Cluster Is Named Default

More than one cluster is configured under clusters and none of them is named Default. The balancer looks up the cluster named Default, which is then created empty.

All Nodes Are Busy

With byThreadStrategy, every node already handles maxNumberOfThreadsPerEndpoint requests, and none became free after five waits of retryTimeOnBusy milliseconds. The exception message is All available servers are busy.

The Session Id Extractor Failed

A configured session id extractor (xmlSessionIdExtractor, jSessionIdExtractor or sessionIdExtractor) failed on the request, for example because of a malformed XML body or an expression error.

How to Fix It

Check the health of the backends and the node status in the admin console or in the log output of the balancerHealthMonitor. Bring at least one node back UP.
Check the clusters configuration: add nodes, and name the cluster Default if you use several clusters with a single balancer.
With byThreadStrategy, raise maxNumberOfThreadsPerEndpoint or retryTimeOnBusy, or add nodes.
If none of this applies, look up the exception in the gateway log and report the problem with the stack trace on GitHub Issues.

Clients can retry later.

Hide Details in Production

By default, Membrane runs in development mode and adds details to error responses that help you find the problem, including the link to this page. These details may reveal information about your setup. In production, turn them off:

configuration:
  production: true

configuration:
  production: true

Add the configuration document to your apis.yaml and separate it from the APIs with ---.

In production mode, the title of this error is replaced by Internal Server Error, and the response only says Internal details are hidden. See server log (key: ...). Search the gateway log for that key to find the full error message.