There are a few reasons why OOM might happen in a JVM. For some of them a JVM will crash with an option to write heap dump to a file system. None of us wants to get OOM on prod, and have to reconfigure deployments and hope for the worst to happen again, this time with some fallback plan.
In a JVM this can be configured with:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/heapDumpDirectory' but the problem is that each heap dump will be by default saved to a file with the same or similar name: /heapDumpDirectory/<process_id>.
In this post I’ll describe things you want to consider to let JVM use own ergonomic configuration, without drastically overriding them, for which you need more advanced tuning and more metrics.
Pod sizing for GC The limit numbers of processors and memory impacts how JVM will tune its own performance characteristics.
Most importantly it impacts what GC will be used and how many threads it will start to clean up memory, which impacts how frequent and long GC pauses are.
This page lists deployment strategies I use to run JVM on Kubernetes.
Below you will find 3 sections describing more common deployment practices of JVM. The practices are listed from the most to the least expensive to run, but each strategy has other drawbacks too.
The described practices are more “realistic”, as cost-optimised ways to run Kubernetes deployments. I am for a predictable utilisation over load. There are more strategies, that argue for never setting CPU limits.
apiVersion: apps/v1 kind: Deployment metadata: namespace: media name: tdarr spec: selector: matchLabels: app: tdarr revisionHistoryLimit: 10 replicas: 1 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 template: metadata: labels: app: tdarr spec: volumes: - name: tdarr-server hostPath: path: /mnt/tdarr/server type: Directory - name: tdarr-config hostPath: path: /mnt/tdarr/config type: Directory - name: nfs hostPath: path: /nfs type: Directory - name: cache hostPath: path: /mnt/tdarr/cache type: Directory containers: - name: tdarr image: ghcr.
Our test pack is configured dynamically from environment variables. Each scenario can be configured independently with different target VUs, duration or even executor.
Let’s start from a file called main.js. It imports all our scenarios, each as a default function:
export { default as cacheCreateAll } from './runners/cacheCreateAll.js'; export { default as cacheCreateUpdateRemove } from './runners/cacheCreateUpdateRemove.js'; export { default as userSearch } from './runners/userSearch.js'; The main.js file is our entry point to the application.
My team is preparing our company to acquire another customer who at initial stages will be 5x bigger than our current biggest customer. To do it, we had to rewrite our performance tests from Gatling to k6. Improve reporting, metrics and scalability of our whole infrastructure and tune set of microservices.
To test our infrastructure we had to scale up our perf test runners too and to do that we developed a set of containerised performance tests and run our performance test pack on dedicated kubernetes nodes.
Our performance tests project is complex, we have +40 .js files, csv feeder files and use custom extensions, so we need to bundle all of that in a single image.
We want things to be version controlled, deploy performance tests in a cloud-native way and the image to be compatible with official k6 image.
# Build the k6 binary with the extension FROM golang:1.20 as builder RUN go install go.k6.io/xk6/cmd/xk6@v0.8.1 # Update README.
An example of HPA that scales up and down depending on CPU and memory consumption.
apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: identity spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: deploymentname minReplicas: 2 maxReplicas: 10 behavior: scaleUp: stabilizationWindowSeconds: 300 policies: - type: Pods value: 1 periodSeconds: 300 scaleDown: stabilizationWindowSeconds: 300 policies: - type: Pods value: 1 periodSeconds: 300 selectPolicy: Min metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 - type: Resource resource: name: memory target: type: AverageValue averageValue: 1000Mi
Run Archive Warrior in your Kubernetes cluster as DaemonSet
apiVersion: apps/v1 kind: DaemonSet metadata: name: warrior namespace: archive labels: app: warrior spec: selector: matchLabels: app: warrior template: metadata: labels: app: warrior spec: nodeSelector: kubernetes.io/arch: amd64 terminationGracePeriodSeconds: 60 containers: - image: atdr.meo.ws/archiveteam/warrior-dockerfile:latest name: warrior resources: requests: cpu: "200m" memory: "128Mi" limits: cpu: "400m" memory: "256Mi" env: - name: DOWNLOADER value: your_name - name: SELECTED_PROJECT value: auto - name: CONCURRENT_ITEMS value: "4" ports: - containerPort: 8001 imagePullPolicy: Always
A less known thing about deploying a JVM in a container is what garbage collector will be set, if you do not specify one. Let’s look at the cases of JVM running in a container and see what GC will be set by default as I experiment with different Java versions and memory limits.
Java 8 - OpenJDK8-alpine With memory limit 1791Mb podman run --memory=1791m -ti openjdk:8-alpine java -XX:+PrintFlagsFinal -XX:+UseContainerSupport | grep 'Use.