Go 1.19 introduced the ability to tune the way the garbage collector works by setting a soft memory limit. One recommendation is to use this new memory limit when deploying in a container environment. This post looks at using the Kubernetes Downward API to set this soft limit.
Do I need a Soft Memory Limit
Do take advantage of the memory limit when the execution environment of your Go program is entirely within your control, and the Go program is the only program with access to some set of resources (i.e. some kind of memory reservation, like a container memory limit).
A good example is the deployment of a web service into containers with a fixed amount of available memory.
An overview of the soft memory limit can be found in the Go 1.19 release notes. See the Guide to the Go Garbage Collector for further explanation of the new limit. The guide includes scenarios where you may want to use, or avoid using, a soft limit.
This post looks at how to apply a memory limit in Go when running in Kubernetes.
Applying a Soft Memory Limit
There are two ways you can set a soft memory limit in Go. The first is to make a call to debug.SetMemoryLimit() (docs). The second is to use the GOMEMLIMIT environment variable. This is likely the most common way you will specify a limit.
In Kubernetes, environment variables for a container are specified in the Pod definition. A simple example is shown below.
apiVersion: v1
kind: Pod
metadata:
name: go-k8s-memory
spec:
containers:
- name: go-k8s-memory
image: ko://go-k8s-memory
ports:
- containerPort: 8080
resources:
limits:
memory: 64Mi // <--- limit on container memory
cpu: "1"
requests:
memory: 62Mi
cpu: "0.2"
env:
- name: GOMEMLIMIT
value: "67108864" // <--- limit on Go runtime
This works, but we are defining the same memory limit in two places. The link between these two values may not be obvious in future. We want to specify the value of the environment variable based on the container memory limit. One option might be to reach for a YAML templating language. In this case, Kubernetes provides an API that allows us to do this.
It is sometimes useful for a container to have information about itself, without being overly coupled to Kubernetes. The downward API allows containers to consume information about themselves or the cluster without using the Kubernetes client or API server.
A modified Pod definition, using the Downward API is shown below.
apiVersion: v1
kind: Pod
metadata:
name: go-k8s-memory
spec:
containers:
- name: go-k8s-memory
image: ko://go-k8s-memory
ports:
- containerPort: 8080
resources:
limits:
memory: 64Mi // <--- limit on container memory
cpu: "1"
requests:
memory: 62Mi
cpu: "0.2"
env:
- name: GOMEMLIMIT
valueFrom: // <--- value taken from downwards API
resourceFieldRef:
resource: limits.memory
The GOMEMLIMIT environment variable in our container runtime environment will now be set based on the container memory limit. Describing the Pod will let you confirm this.
kubectl describe pod/go-k8s-memory
// partial description for brevity
Name: go-k8s-memory
Namespace: default
Status: Running
Containers:
go-k8s-memory:
State: Running
Ready: True
Limits:
cpu: 1
memory: 64Mi
Requests:
cpu: 200m
memory: 62Mi
Environment:
GOMEMLIMIT: 67108864 (limits.memory) // <--- computed value
One Caveat
It should be noted that there is a limitation with this approach. The Go documentation includes the following note.
In this case, a good rule of thumb is to leave an additional 5-10% of headroom to account for memory sources the Go runtime is unaware of.
If you are unsure what these memory sources might be, examples include:
- OS kernel memory held on behalf of the process
- memory allocated by C code
- memory mapped by syscall.Mmap
The challenge with using the Downard API is that we are unable to apply a multiplier to the values it returns. This means we can’t apply the recommended 5-10% headroom. One workaround I’ve seen is to use the requests.memory resource specification to set GOMEMLIMIT. Then set the container limits.memory value to be 5-10% above that. If you reach for this, it is time to develop a deeper understanding of your application’s memory needs.