[[TOC]]
Note¶
This notes were done before the official Nginx Otel module was released. It would be nice to check it and see if it can be installed in a more straightforward way: https://github.com/nginxinc/nginx-otel
Overview¶
There is no APM agent for Nginx that would be able to send telemetry data straight to Elastic. Thats why we need to use a OpenTelemetry collector, that would collect data from Nginx and send it to ELastic APM.
We can either install a new Collector, or utilize existing collectors running in Kubernetes. For this document we will use the opentelemetry-collector service running in dev-v2 cluster in the logging namespace.
The collector needs data from Nginx. For that we must use the Nginx module from OpenTelemetry: https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx.
The module is in the form of a .so file and there are two ways how to retrieve it:
* Build .so file from sources
* Download pre-compiled .so from Github.
Since the compilation process was not very straightforward and there were a lot issues while compiling the module, we will use the second option and download a pre-compiled module from Github.
Here we hit the first problem. The latest OpenTelemetry Webserver instrumentation (currently v1.0.3) supports these versions of Nginx: 1.22.0, 1.23.0, 1.23.1. However the Nginx version in Debian 11 is 1.18.0. Therefore we must download the latest version that supports Nginx 1.18.0 which is OpenTelemetry Webserver instrumentation v1.0.1. This however uses different configuration for Nginx as the latest version.
If you are using the default Debian 11 version of Nginx - v1.18.0 please proceed with Installing OpenTelemetry module v1.0.1.
If you are using Nginx packages provided by the Nginx.inc APT repository proceed with Installing latest OpenTelemetry Nginx module.
Installing OpenTelemetry Nginx module v1.0.1¶
System configuration¶
Download the release from Github: https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv1.0.1/opentelemetry-webserver-sdk-x64-linux.tgz
cd /root/src
wget https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv1.0.1/opentelemetry-webserver-sdk-x64-linux.tgz
tar -xzvf opentelemetry-webserver-sdk-x64-linux.tgz
# rename the dir so we dont mix up versions later
mv opentelemetry-webserver-sdk opentelemetry-webserver-sdk-1.0.1
Place .so module into Nginx modules directory:
cd opentelemetry-webserver-sdk-1.0.1
cp WebServerModule/Nginx/ngx_http_opentelemetry_module.so /usr/share/nginx/modules/
The module also needs some shared libraries, which need to be placed into the system:
mkdir -p /usr/local/lib/opentelemetry
cp sdk_lib/lib/* /usr/local/lib/opentelemetry/
echo "/usr/local/lib/opentelemetry" > /etc/ld.so.conf.d/opentelemetry.conf
ldconfig
The module also needs a log configuration file in `/usr/local/conf/appdynamics_sdk_log4cxx.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">
<appender name="main" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/var/log/opentelemetry_sdk.log"/>
<param name="HeaderOnlyInNewFile" value="true"/>
<param name="MaxFileSize" value="20MB"/>
<param name="MaxBackupIndex" value="5"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid}[%t] [%c{2}] %m%n" />
<param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
</layout>
</appender>
<appender name="api" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/var/log/opentelemetry_api.log"/>
<param name="HeaderOnlyInNewFile" value="true"/>
<param name="MaxFileSize" value="20MB"/>
<param name="MaxBackupIndex" value="5"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%t] [%c{2}] %m%n" />
<param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
</layout>
</appender>
<appender name="api_user" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/var/log/opentelemetry_user.log"/>
<param name="HeaderOnlyInNewFile" value="true"/>
<param name="MaxFileSize" value="20MB"/>
<param name="MaxBackupIndex" value="5"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%t] [%c{2}] %m%n" />
<param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
</layout>
</appender>
<logger name="api" additivity="false">
<level value="info"/>
<appender-ref ref="api"/>
</logger>
<logger name="api_user" additivity="false">
<level value="info"/>
<appender-ref ref="api_user"/>
</logger>
<root>
<priority value="info" />
<appender-ref ref="main"/>
</root>
</log4j:configuration>
Nginx configuration¶
Now we can install the module into Nginx. First we need to load it:
echo "load_module modules/ngx_http_opentelemetry_module.so;" > /etc/nginx/modules-enabled/80-mod-http-opentelemetry.conf
Next, we need to configure at least some values for the OpenTelemetry collection:
# edit the file /etc/nginx/nginx.conf
# place the following into the http {} context
# make sure you have the correct IP for the endpoint, and also change the ServiceName* values
##
# OpenTelemetry Settings
##
NginxModuleEnabled ON;
NginxModuleOtelSpanExporter otlp;
NginxModuleOtelSpanProcessor batch;
NginxModuleOtelSampler AlwaysOn;
NginxModuleOtelExporterEndpoint 34.159.145.93:80;
NginxModuleServiceName Dev_NBO;
NginxModuleServiceNamespace Dev_NBO_Namespace;
NginxModuleServiceInstanceId Dev_NBO_Instance;
Restart Nginx and see the error log, and also the OpenTelemetry logs in /var/log. Please note that nginx -t will discover only some configuration errors, but only a complete nginx restart with service nginx restart or systemctl restart nginx will verify that the OpenTelemetry mod has been loaded correctly.
Connecting from Nginx to dev-v2 cluster¶
First idea was to use Kong like this:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
konghq.com/plugins: request-transformer-ch1
konghq.com/protocols: http
konghq.com/strip-path: "true"
name: otel-internal-kong
spec:
ingressClassName: kong-internal
rules:
- http:
paths:
- path: /otel
pathType: ImplementationSpecific
backend:
service:
name: opentelemetry-collector
port:
number: 4318
However this was not working since the OpenTelemetry uses gRPC and Kong didnt understand that.
The solution is to either create a internal IP for opentelemetry-collector or use some of Kong's gRPC functionality. Remains to be done.
Installing latest OpenTelemetry Nginx module¶
Use this how-to in case yu are installing OpenTelemetry on a Vm where you can switch to a Nginx.inc Nginx package.
Currently the latest version of OpenTelemetry Nginx module is v1.0.3. It has support for these versions of Nginx: 1.22.0, 1.23.0, 1.23.1.
Install appropriate version of Nginx¶
Follow the instructions below, taken from here: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#installing-prebuilt-debian-packages.
Please note: Do not overwite existing configfiles and use the default option to keep the old ones.
# install nginx.inc apt repository
apt install curl gnupg2 ca-certificates lsb-release debian-archive-keyring
curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \
http://nginx.org/packages/debian `lsb_release -cs` nginx" | sudo tee /etc/apt/sources.list.d/nginx.list
echo -e "Package: *\nPin: origin nginx.org\nPin: release o=nginx\nPin-Priority: 900\n" | sudo tee /etc/apt/preferences.d/99nginx
# check for nginx versions provided
apt-get update
apt-get show nginx
# remove any former OpenTelemetry settings, and libs, etc
# choose a version of nginx that fits with one of 1.22.0, 1.23.0, 1.23.1
# these are version of nginx supported by the latest (v1.0.3) OpenTelemetry nginx module
# future versions migt vary
apt-get install nginx=1.22.0-1~bullseye
nginx -v
System configuration¶
Download the release from Github: https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv1.0.3/opentelemetry-webserver-sdk-x64-linux.tgz
cd /root/src
wget https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv1.0.3/opentelemetry-webserver-sdk-x64-linux.tgz
tar -xzvf opentelemetry-webserver-sdk-x64-linux.tgz
# rename the dir so we dont mix up versions later
mv opentelemetry-webserver-sdk opentelemetry-webserver-sdk-1.0.3
Place .so module into Nginx modules directory:
cd opentelemetry-webserver-sdk-1.0.3
# choose correct version of Nginx - here 1.22.0
cp WebServerModule/Nginx/{NGINX_VERSION}/ngx_http_opentelemetry_module.so /usr/lib/nginx/modules/
The module also needs some shared libraries, which need to be placed into the system:
mkdir -p /usr/local/lib/opentelemetry
cp sdk_lib/lib/* /usr/local/lib/opentelemetry/
echo "/usr/local/lib/opentelemetry" > /etc/ld.so.conf.d/opentelemetry.conf
ldconfig
Nginx configuration¶
Now we can install the module into Nginx. First we need to load it:
echo "load_module modules/ngx_http_opentelemetry_module.so;" > /etc/nginx/modules-enabled/80-mod-http-opentelemetry.conf
Before doing next steps try restarting Nginx to see if there are no issues loading the module:
If there are no issues, continue with OpenTelemetry module configuration:
Next, we need to configure at least some values for the OpenTelemetry collection. Please note that even the latest release of the OpenTelemetry Nginx module can not be used according to this example: https://github.com/open-telemetry/opentelemetry-cpp-contrib/blob/main/instrumentation/nginx/test/conf/nginx.conf.
This is a drawback as the new configuration options enabled fine-grained controll of when and where OpenTelemetry should be enabled. The legacy settings below enable OpenTelemetry for all endpoints:
# edit the file /etc/nginx/nginx.conf
# place the following into the http {} context
# make sure you have the correct IP for the endpoint, and also change the ServiceName* values
##
# OpenTelemetry Settings
##
NginxModuleEnabled ON;
NginxModuleOtelSpanExporter otlp;
NginxModuleOtelSpanProcessor batch;
NginxModuleOtelSampler AlwaysOn;
NginxModuleOtelExporterEndpoint 34.159.145.93:80;
NginxModuleServiceName Dev_NBO;
NginxModuleServiceNamespace Dev_NBO_Namespace;
NginxModuleServiceInstanceId Dev_NBO_Instance;
Restart Nginx and see the error log, and also the OpenTelemetry logs in /var/log. Please note that nginx -t will discover only some configuration errors, but only a complete nginx restart with service nginx restart or systemctl restart nginx will verify that the OpenTelemetry mod has been loaded correctly.
Todo¶
- Upgrade Nginx. Use latest Nginx and latest OpenTelemetry Nginx module (1.23.1). See: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#installing-prebuilt-debian-packages
- Expose OpenTelemetry correctly. Either use a internal IP, or find a way how to expose OpenTelemetry via Kong using gRPC.