← Tech

Marathon Load Balancer

To help with service to service HTTP calls, we’ve rolled out a dynamic load balancer using Envoy and some glue banno-marathon-envoy-ds. It handles discovery as Marathon app instances can scale up and down on different hosts and ports. There is also a DNS component that we’ve deployed to our DNS servers to help as apps begin their transition to Kubernetes.

Calling a service:

To call a service, you need to only know the the name of the service. The service is requestable via the URL: https://${serviceName}.banno.svc.cluster.local

There is still a way to use the team as a “namespace” instead of banno, but it is deprecated and the banno “namespace” should be used instead.

The hostname will resolve to local marathon-envoy instances in the same Marathon cluster, and the marathon-envoy will match on the requested host and load balance across upstream service instances for that serviceName. By default, it uses the second configured port which Marathon and mesos supplies as PORT1. Typically, this is our applications’ http port and PORT0 is the admin/metrics/health port.

If there is a service in a different cluster that you need to connect to i.e. production-aws-2, the URL becomes https://${serviceName}.banno.svc.cluster.production-aws-2. This is true for cross-environment requests too: a service in uat-2 can be requested from a production environment using https://${serviceName}.banno.svc.cluster.uat-2

Additionally, http can be used instead of https.

Customization via Marathon labels:

There are a few Marathon labels which can be configured:

  • marathon-envoy-http-port: Mentioned above, marathon-envoy will use the second port: PORT1 as the http port. There are some services which do not use mesos allocation for ports and listen on a non-allocatable port i.e. 9092. This can be overridden with the marathon-envoy-http-port label.
  • marathon-envoy-ping-path: marathon-envoy is configured to assess upstream health based on a 200 OK http response on the /ping endpoint. This can be overridden with the marathon-envoy-ping-path label to a different path i.e. /a/sentry/api/ping.
  • marathon-envoy-upstream-http2: if configured to any non-empty string (usually just "true"), marathon-envoy will use the http2 protocol to connect to the upstream. This is always used in grpc services.
  • marathon-envoy-upstream-timeout-seconds: (default: 120) if configured, marathon-envoy will set the timeout on connections for the upstream service to the desired integer in seconds. If there is a problem with the value, it will send a warn log output with the error message and default back to 120 seconds.
  • marathon-envoy-upstream-tls: if configured to any non-empty string (usually just "true"), marathon-envoy will use TLS to connect to the upstream. This is typically used in grpc services that use TLS.
  • marathon-envoy-additional-subdomains: configures marathon-envoy to also route for the subdomains listed in addition to the app name.
  • marathon-envoy-additional-namespaces: (deprecated) if configured, marathon-envoy will be configured to also route for these additional namespaces. It is a comma-delimited list and will still include the team with this set.
  • marathon-envoy-port-2-subdomain: if configured and there is at least 3 ports defined, marathon-envoy will be configured to route the subdomain given in the label (eg my-app-other-port) to the 3rd port (Marathon sets it as the PORT2 environment variable). From the example, my-app-other-port.banno.svc.cluster.local would route to PORT2 of the service. Note that Envoy will do health checks against this port on /ping as well.

These are usable in the jsonnet templates with the labels_lib (with underscores instead of dashes) and have conversions for types i.e. labels_lib.marathon_envoy_additional_namespaces(["espresso", "jabberwocky"]) and labels_lib.marathon_envoy_ping_path("/a/sentry/api/ping")

Tips

Health of marathon-envoy and upstream clusters:

Even though marathon-envoy dynamically gets updated as tasks live and die, things can go wrong: upstream instances can still misbehave, marathon-envoy could lose connection to its discovery service, and so on. Envoy takes a stance of eventually consistent service discovery in which even though service discovery can lets it know that an upstream is gone, Envoy will still route to it until is the upstream is unhealthy.

There are two different ways that marathon-envoy assesses upstream health:

  • Active health checks from marathon-envoy to the upstream http port on /ping. This is a different check than what Marathon does to know if a task needs killed. marathon-envoy checks every second.
  • If a request to an upstream is detected as unable to connect, that upstream is taking out of the cycle and retried again later.

Envoy exposes a good amount of statistices and we grab the higher level ones as Prometheus metrics.

These are exposed on the grfana dashboard: https://infra.banno-production.com/grafana/d/000000003/marathon-lb-envoy?refresh=30s&orgId=1&from=now-1h&to=now&var-cluster=%2Fapi&var-instance=All

Nuts and bolts:

Envoy is neat. It’s different than other load balancers in that the principal method of configuration is not configuration files, but an external discovery service which serves as an API. So we only configure marathon-envoy with a bootstrap configuration to find banno-marathon-envoy-ds which then marathon-envoy calls to get configuration and then streams down updates as they happen. banno-marathon-envoy-ds uses Marathon server sent events to get notified of Marathon application events: task and application changes. banno-marathon-envoy-ds converts these to the appropriate Envoy resources which get streamed down. go-control-plane and go-marathon do most of the heavy lifting for us.

Canary Support

The banno-marathon-envoy-ds service that marathon-envoy uses is configured to set up weighted clusters for the service and a canary counterpart. The url created for a service (https://${serviceName}.banno.svc.cluster.local) by default will route 100% of the traffic to the named service (eg https://template-service.banno.svc.cluster.local to the template-service Marathon apps) and nothing to the canary app (in this example template-service-canary). Runtime configuration for Envoy can be updated to change the weights for these clusters. The paths for each cluster are routing.traffic_shift./${serviceName}./${serviceName} and routing.traffic_shift./${serviceName}./${serviceName}-canary. These paths can be used on the /runtime_modify endpoint of Envoy to update to the values given. There is a script provided in the environments repository to help with this in a given environment:

$ scripts/update-envoy-canary-routing.sh
Usage: update-envoy-canary-routing.sh [-e <ENVIRONMENT>] <MARATHON APP ID> <PRIMARY CLUSTER PERCENT> <CANARY CLUSTER PERCENT>

MARATHON APP ID - Marathon app id, this includes the /, eg /template-service
PRIMARY CLUSTER PERCENT - Percent of traffic to send to the primary upstream service, this must add up to 100 with the canary cluster percent
CANARY CLUSTER PERCENT - Percent of traffic to send to the canary upstream service, this must add up to 100 with the primary cluster percent

Options:
 -e  Environment to update envoy

For example, to update https://template-service.banno.svc.cluster.staging-2 (alongside https://template-service.banno.svc.cluster.local for things in the same environment) to route 90% of traffic at the normal service, template-service, and 10% to the canary service, template-service-canary, the command to run would be:

$ scripts/update-envoy-canary-routing.sh -e staging-2 /template-service 90 10

Envoy requires giving a “total” weight to upstream clusters, and the default is 100. The weights of all of the clusters added together needs to equal the total weight, otherwise it throws a warning. This script will check that the weights given add up to 100.

Note:If no ${serviceName}-canary is available, this means that the percent of traffic given to the canary cluster will result in a 503 status.