Go 1.19 introduced the ability to tune the way the garbage collector works by setting a soft memory limit. One recommendation is to use this new memory limit when deploying in a container environment. This post looks at using the Kubernetes Downward API to set this soft limit.
Do I need a Soft Memory Limit
Do take advantage of the memory limit when the execution environment of your Go program is entirely within your control, and the Go program is the only program with access to some set of resources (i.e. some kind of memory reservation, like a container memory limit).
A good example is the deployment of a web service into containers with a fixed amount of available memory.
An overview of the soft memory limit can be found in the Go 1.19 release notes. See the Guide to the Go Garbage Collector for further explanation of the new limit. The guide includes scenarios where you may want to use, or avoid using, a soft limit.
This post looks at how to apply a memory limit in Go when running in Kubernetes.
Applying a Soft Memory Limit
There are two ways you can set a soft memory limit in Go. The first is to make a call to
debug.SetMemoryLimit() (docs). The second is to use the
GOMEMLIMIT environment variable. This is likely the most common way you will specify a limit.
In Kubernetes, environment variables for a container are specified in the Pod definition. A simple example is shown below.
apiVersion: v1 kind: Pod metadata: name: go-k8s-memory spec: containers: - name: go-k8s-memory image: ko://go-k8s-memory ports: - containerPort: 8080 resources: limits: memory: 64Mi // <--- limit on container memory cpu: "1" requests: memory: 62Mi cpu: "0.2" env: - name: GOMEMLIMIT value: "67108864" // <--- limit on Go runtime
This works, but we are defining the same memory limit in two places. The link between these two values may not be obvious in future. We want to specify the value of the environment variable based on the container memory limit. One option might be to reach for a YAML templating language. In this case, Kubernetes provides an API that allows us to do this.
It is sometimes useful for a container to have information about itself, without being overly coupled to Kubernetes. The downward API allows containers to consume information about themselves or the cluster without using the Kubernetes client or API server.
A modified Pod definition, using the Downward API is shown below.
apiVersion: v1 kind: Pod metadata: name: go-k8s-memory spec: containers: - name: go-k8s-memory image: ko://go-k8s-memory ports: - containerPort: 8080 resources: limits: memory: 64Mi // <--- limit on container memory cpu: "1" requests: memory: 62Mi cpu: "0.2" env: - name: GOMEMLIMIT valueFrom: // <--- value taken from downwards API resourceFieldRef: resource: limits.memory
GOMEMLIMIT environment variable in our container runtime environment will now be set based on the container memory limit. Describing the Pod will let you confirm this.
kubectl describe pod/go-k8s-memory
// partial description for brevity Name: go-k8s-memory Namespace: default Status: Running Containers: go-k8s-memory: State: Running Ready: True Limits: cpu: 1 memory: 64Mi Requests: cpu: 200m memory: 62Mi Environment: GOMEMLIMIT: 67108864 (limits.memory) // <--- computed value
It should be noted that there is a limitation with this approach. The Go documentation includes the following note.
In this case, a good rule of thumb is to leave an additional 5-10% of headroom to account for memory sources the Go runtime is unaware of.
If you are unsure what these memory sources might be, examples include:
- OS kernel memory held on behalf of the process
- memory allocated by C code
- memory mapped by syscall.Mmap
The challenge with using the Downard API is that we are unable to apply a multiplier to the values it returns. This means we can’t apply the recommended 5-10% headroom. One workaround I’ve seen is to use the
requests.memory resource specification to set
GOMEMLIMIT. Then set the container
limits.memory value to be 5-10% above that. If you reach for this, it is time to develop a deeper understanding of your application’s memory needs.