Skip to content

Configure automatic cleanup with TTL⚓

Automatically clean up finished master and worker jobs and their pods after tests complete using Kubernetes TTL (time-to-live).

Prerequisites⚓

  • Kubernetes 1.12+ (TTL controller support)
  • Locust Kubernetes Operator installed

What gets cleaned up⚓

When TTL is configured:

  • ✓ Cleaned up: Master and worker Jobs and their Pods
  • ✗ Kept: LocustTest CR, ConfigMaps, Secrets, Services

This allows you to review test results (via kubectl get locusttest) while automatically removing resource-consuming pods.

Set TTL via Helm values⚓

Configure TTL for all tests during operator installation:

# values.yaml
locustPods:
  ttlSecondsAfterFinished: 3600  # Clean up after 1 hour

Install or upgrade the operator:

helm upgrade --install locust-operator locust-k8s-operator/locust-k8s-operator \
  --namespace locust-system \
  -f values.yaml

All LocustTest resources will inherit this TTL value.

Set TTL via CLI⚓

Override TTL at installation time:

helm install locust-operator locust-k8s-operator/locust-k8s-operator \
  --namespace locust-system \
  --set locustPods.ttlSecondsAfterFinished=7200  # 2 hours

Common TTL values⚓

ValueDurationUse case
0ImmediateUse with caution -- risk of race condition with log collection
3005 minutesQuick tests where results are exported immediately
36001 hourStandard tests with manual result review
72002 hoursLong tests with delayed result analysis
8640024 hoursTests requiring extensive post-analysis
"" (empty)NeverDevelopment or when using external cleanup

Disable TTL⚓

To disable automatic cleanup:

# values.yaml
locustPods:
  ttlSecondsAfterFinished: ""  # Empty string disables TTL

Or omit the field entirely:

# values.yaml
locustPods:
  # ttlSecondsAfterFinished not set - no TTL

Without TTL, jobs and pods persist until manually deleted.

Verify TTL configuration⚓

Check that TTL is set on created jobs:

# Run a test
kubectl apply -f locusttest.yaml

# Check master job TTL
kubectl get job my-test-master -o yaml | grep ttlSecondsAfterFinished

Expected output:

ttlSecondsAfterFinished: 3600

Watch automatic cleanup⚓

Monitor cleanup in action:

# List all jobs with timestamps
kubectl get jobs -o wide --watch

# After TTL expires, jobs transition to deleted

Verify cleanup occurred:

# Jobs should be gone after TTL
kubectl get jobs -l performance-test-name=my-test

# Pods should also be gone
kubectl get pods -l performance-test-name=my-test

# But CR still exists
kubectl get locusttest my-test

Example: CI/CD with automatic cleanup⚓

For CI/CD pipelines, use a short TTL to allow time for log collection:

# values.yaml
locustPods:
  ttlSecondsAfterFinished: 60  # Clean up 60 seconds after completion

Avoid ttlSecondsAfterFinished: 0 in CI/CD

Setting TTL to 0 creates a race condition: Kubernetes may delete the job and its pods before your pipeline can collect logs with kubectl logs. Use at least 60 seconds to give your pipeline time to retrieve results.

In your pipeline:

# Run the test
kubectl apply -f locusttest.yaml

# Wait for completion
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \
  locusttest/ci-test --timeout=10m

# Collect results within the 60-second TTL window
kubectl logs job/ci-test-master > results.log

# Jobs and pods will be cleaned up after the TTL expires
# CR persists for historical tracking

Example: Development with manual cleanup⚓

During development, disable TTL to inspect pods:

# values.yaml
locustPods:
  ttlSecondsAfterFinished: ""  # Disable automatic cleanup

Clean up manually when done:

# Delete just the test
kubectl delete locusttest my-test

# Or delete all test resources
kubectl delete locusttest --all

Backward compatibility⚓

The operator maintains backward compatibility with the old configuration path:

# Old path (still supported)
config:
  loadGenerationJobs:
    ttlSecondsAfterFinished: 3600

# New path (recommended)
locustPods:
  ttlSecondsAfterFinished: 3600

Helper functions in the Helm chart ensure both paths work. Use the new path for future configurations.

How TTL works⚓

Kubernetes TTL controller monitors finished jobs:

  1. Test completes (phase: Succeeded or Failed)
  2. Job transitions to finished state
  3. TTL countdown starts
  4. After TTL seconds, controller deletes job
  5. Cascading deletion removes dependent pods

Important: TTL countdown starts when the job finishes, not when it starts.

What's next⚓