Quick command to get OnCall running using ArgoCD:
argocd app create oncall \ --repo https://grafana.github.io/helm-charts \ --helm-chart oncall \ --revision 'v1.3.*' \ --dest-namespace oncall \ --dest-server https://kubernetes.default.svc \ --values-literal-file values.yaml \ --project oncall
A Java project with published container image that contains intentionally leaky native code to observe symptoms of a memory leak in Java in podman/docker or Kubernetes.
Native code intentionally “leaks” provided number of megabytes in a loop. The project runs by default with -XX:NativeMemoryTracking=summary enabled.
I wanted to observe how JVM will report native memory, crash and what pod and JVM metrics will look like.
Java doesn’t allocate many objects, almost none.
In this post I’ll describe things you want to consider to let JVM use own ergonomic configuration, without drastically overriding them, for which you need more advanced tuning and more metrics.
Pod sizing for GC The limit numbers of processors and memory impacts how JVM will tune its own performance characteristics.
Most importantly it impacts what GC will be used and how many threads it will start to clean up memory, which impacts how frequent and long GC pauses are.
This page lists deployment strategies I use to run JVM on Kubernetes.
Below you will find 3 sections describing more common deployment practices of JVM. The practices are listed from the most to the least expensive to run, but each strategy has other drawbacks too.
The described practices are more “realistic”, as cost-optimised ways to run Kubernetes deployments. I am for a predictable utilisation over load. There are more strategies, that argue for never setting CPU limits.
apiVersion: apps/v1 kind: Deployment metadata: namespace: media name: tdarr spec: selector: matchLabels: app: tdarr revisionHistoryLimit: 10 replicas: 1 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 template: metadata: labels: app: tdarr spec: volumes: - name: tdarr-server hostPath: path: /mnt/tdarr/server type: Directory - name: tdarr-config hostPath: path: /mnt/tdarr/config type: Directory - name: nfs hostPath: path: /nfs type: Directory - name: cache hostPath: path: /mnt/tdarr/cache type: Directory containers: - name: tdarr image: ghcr.
Our test pack is configured dynamically from environment variables. Each scenario can be configured independently with different target VUs, duration or even executor.
Let’s start from a file called main.js. It imports all our scenarios, each as a default function:
export { default as cacheCreateAll } from './runners/cacheCreateAll.js'; export { default as cacheCreateUpdateRemove } from './runners/cacheCreateUpdateRemove.js'; export { default as userSearch } from './runners/userSearch.js'; The main.js file is our entry point to the application.
My team is preparing our company to acquire another customer who at initial stages will be 5x bigger than our current biggest customer. To do it, we had to rewrite our performance tests from Gatling to k6. Improve reporting, metrics and scalability of our whole infrastructure and tune set of microservices.
To test our infrastructure we had to scale up our perf test runners too and to do that we developed a set of containerised performance tests and run our performance test pack on dedicated kubernetes nodes.
Our performance tests project is complex, we have +40 .js files, csv feeder files and use custom extensions, so we need to bundle all of that in a single image.
We want things to be version controlled, deploy performance tests in a cloud-native way and the image to be compatible with official k6 image.
# Build the k6 binary with the extension FROM golang:1.20 as builder RUN go install go.k6.io/xk6/cmd/xk6@v0.8.1 # Update README.
An example of HPA that scales up and down depending on CPU and memory consumption.
apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: identity spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: deploymentname minReplicas: 2 maxReplicas: 10 behavior: scaleUp: stabilizationWindowSeconds: 300 policies: - type: Pods value: 1 periodSeconds: 300 scaleDown: stabilizationWindowSeconds: 300 policies: - type: Pods value: 1 periodSeconds: 300 selectPolicy: Min metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 - type: Resource resource: name: memory target: type: AverageValue averageValue: 1000Mi
Run Archive Warrior in your Kubernetes cluster as DaemonSet
apiVersion: apps/v1 kind: DaemonSet metadata: name: warrior namespace: archive labels: app: warrior spec: selector: matchLabels: app: warrior template: metadata: labels: app: warrior spec: nodeSelector: kubernetes.io/arch: amd64 terminationGracePeriodSeconds: 60 containers: - image: atdr.meo.ws/archiveteam/warrior-dockerfile:latest name: warrior resources: requests: cpu: "200m" memory: "128Mi" limits: cpu: "400m" memory: "256Mi" env: - name: DOWNLOADER value: your_name - name: SELECTED_PROJECT value: auto - name: CONCURRENT_ITEMS value: "4" ports: - containerPort: 8001 imagePullPolicy: Always