Kubernetes tracing

Application tracing¶

[[TOC]]

Documentation for tracing, what it is, how it works, whats the benefits https://www.elastic.co/guide/en/apm/guide/current/apm-distributed-tracing.html

Since we use an Elastic APM server it is preferred to use official APM agents from Elastic to trace the applications

List of agents supported and configuration references https://www.elastic.co/guide/en/apm/agent/index.html

Once you enable the APM agent in your application, a basic configuration of the APM agent is provided in secret elastic-apm-config

How to mount secret into application¶

From version 0.24.0 of standard service chart its possible to mount multiple secrets via helm values

application:
  extraEnvFromSecretNames:
    - elastic-apm-config

Example of default APM config provided in a secret¶

apiVersion: v1
kind: Secret
data:
  ELASTIC_APM_ENVIRONMENT: dev
  ELASTIC_APM_LOG_LEVEL: INFO
  ELASTIC_APM_TRANSACTION_SAMPLE_RATE: 0.01 #trace only 1% of requests
  ELASTIC_APM_API_KEY: some_apm_api_key 
  ELASTIC_APM_SERVER_URL: https://1001b3cf87924a4684fb7f164eed167c.apm.europe-west3.gcp.cloud.es.io:443 #elastic APM endpoint to collect traces

NOTE: Do not use the same environment variable names in your app, if you want to alter any of this configuration please contact DevOps.

PHP applications¶

First, make sure your namespace contains the APM secret and that the secret is correctly mounted via .gitlab/values file. For details see How to mount secret into application above.

Next, check the latest version of APM agent and docs here: https://www.elastic.co/guide/en/apm/agent/php/current/setup.html.

If you know the latest version, install APM in your application, by adding the following lines into your Dockerfile:

# Install Elastic APM Agent
ARG AGENT="https://github.com/elastic/apm-agent-php/releases/download/v1.13.0/apm-agent-php_1.13.0_x86_64.apk"
USER root
RUN curl -fsSL $AGENT > /tmp/apm.apk && \
    apk add --allow-untrusted /tmp/apm.apk && \
    rm /tmp/apm.apk
USER 101

NOTE: In the example we use an Alpine-based image, and PHP 8.1. Consult devops for more details and help.

In case you use php_admin_value[open_basedir]="/tmp/:/some/thing" in your FPM configuration, you will need to add the /opt/elastic/apm-agent-php/ directory to it, like this:

php_admin_value[open_basedir]="/tmp/:/some/thing/:/opt/elastic/apm-agent-php/"`

Make also sure your FPM pool configuration contains these lines, which enable FPM to load APM configuration from env:

; Custom php.ini values
env["ELASTIC_APM_SERVER_URL"] = $ELASTIC_APM_SERVER_URL
env["ELASTIC_APM_API_KEY"] = $ELASTIC_APM_API_KEY
env["ELASTIC_APM_SERVICE_NAME"] = $ELASTIC_APM_SERVICE_NAME
env["ELASTIC_APM_ENVIRONMENT"] = $ELASTIC_APM_ENVIRONMENT
env["ELASTIC_APM_LOG_LEVEL"] = $ELASTIC_APM_LOG_LEVEL
env["ELASTIC_APM_TRANSACTION_SAMPLE_RATE"] = $ELASTIC_APM_TRANSACTION_SAMPLE_RATE

Before redeploying the application, you will also need to add the ELASTIC_APM_SERVICE_NAME to the values. This is the name of the service in APM dashboard in Kibana. You can either add this line into the pipeline:

    --set-string extraEnvs.ELASTIC_APM_SERVICE_NAME="my-service-name" \

Or you can do it by adding this line into your CI/CD HELM_SECRET_VALUES:

    ELASTIC_APM_SERVICE_NAME: "my-service-name"

Finally, rebuild and redeploy the application and APM should be enabled and working.

Ingresses tracing¶

Tracing ingresses enables us to see the originating address of requests, headers, etc, in the case of KONG it also allows us to view time spent on Kong plugins

KONG¶

Public kong¶

Annotate ingress with plugin name opentelemetry

ingress:
  ...
  annotations:
    konghq.com/plugins: opentelemetry # This is a list of plugins enabled for ingress separated via ,

Internal kong¶

Annotate ingress with plugin name opentelemetry-internal

ingress:
  ...
  annotations:
    konghq.com/plugins: opentelemetry-internal # This is a list of plugins enabled for ingress separated via ,

NGINX¶

Annotate the nginx ingress with the following

ingress:
  annotations:
    nginx.ingress.kubernetes.io/enable-opentelemetry: "true"

Sanitization of data¶

Elastic APM applies a built‑in set of data filters that redact sensitive values found in HTTP headers, bodies, and other contexts. The default patterns are:

password
passwd
pwd
secret
*key
*token*
*session*
*credit*
*card*
*auth*
*principal*
set-cookie

Action required: 1. Review your application for additional sensitive fields.
2. Configure extra sanitization patterns using the guidance here: https://www.elastic.co/docs/solutions/observability/apm/built-in-data-filters#apm-filters-http-header
3. Append your custom patterns; do not remove the defaults.

Important: Always include the full default list above along with any new patterns you add.

Pro tip: Format for pattern ENV override ENV_NAME="<pattern1>,<pattern2>" ,single string with no whitespaces between.

Important notes¶

DO NOT trace probes URLs (readiness, liveness), set URLS that should be ignored, for example PYTHON docs https://www.elastic.co/guide/en/apm/agent/python/6.x/configuration.html#config-transaction-ignore-urls
DO NOT trust incoming traces and spans on public endpoints, event on those with IP-RESTRICTION or Whitelist THIS IS A SECURITY ISSUE.
KONG - should be set globally on the gateway, make sure to NOT override
NGINX - should be set globally on the gateway, make sure to NOT override

How to view tracing data ?¶

Tracing data is being ingested in the new observability Elastic Link to APM services https://ftmo-observability.kb.europe-west3.gcp.cloud.es.io:9243/app/apm/services

Access to view APM data is granted by default for IT when using FTMO Google Account