OpenTelemetry Tracing¶

Warning

Tracing support is considered experimental. The output, configuration, and any other details may change at any time.

Since version 2.1.0, when dnsdist is built with ProtoBuf support, sent messages (using e.g. RemoteLogResponseAction()) can contain OpenTelemetry traces data.

To enable tracing, use setOpenTelemetryTracing(true) in your configuration, or logging.open_telemetry_tracing.enabled to true in your YAML Logging Configuration. It is also possible to call setOpenTelemetryTracing() at runtime. Once enabled, Rules can be used to turn on tracing on a per-query basis.

Per-query tracing can be enabled using the SetTraceAction(). However, dnsdist captures some data before rules processing in order to have tracing information from before the rules are evaluated. When tracing is enabled in the query, dnsdist stores start and end times of certain (but not all) functions that are called during the lifetime of the query and the response. It is recommended to send the traces out through a RemoteLogger in ResponseRules, to capture as much information as possible.

Tracing uses more memory and CPU than usual query processing and it is recommended to enable tracing only for certain queries using specific selectors.

Example configurations¶

In this configuration, the RemoteLogger is passed directly to the SetTrace action. Doing this ensures that no matter what happens with the query (timeout, self-answered, cache-hit, dropped, answered by the backend), the trace will be sent out. When sending the trace in this way, the Protobuf message is essentially empty apart from the OpenTelemetry Trace.

YAML Lua

logging:
  open_telemetry_tracing:
    enabled: true
remote_logging:
  protobuf_loggers:
    - name: pblog
      address: 127.0.0.1:5301
query_rules:
  - name: Enable tracing
    selector:
      # Just as an example, in production don't trace all the queries
      type: All
    action:
      type: SetTrace
      value: true
      remote_loggers:
        - pblog

-- newServer should go here
rl = newRemoteLogger('127.0.0.1:5301')
setOpenTelemetryTracing(true)

addAction(AllRule(), SetTraceAction(true, {remoteLoggers={rl}}), {name="Enable tracing"})

Should you only want to receive the trace after a response was received from the backend, including a fully filled Protobuf message, a RemoteLog action can be used:

YAML Lua

logging:
  open_telemetry_tracing:
    enabled: true
remote_logging:
  protobuf_loggers:
    - name: pblog
      address: 127.0.0.1:5301
query_rules:
  - name: Enable tracing
    selector:
      # Just as an example, in production don't trace all the queries
      type: All
    action:
      type: SetTrace
      value: true
 response_rules:
   - name: Send PB log
     selector:
       type: All
     action:
       type: RemoteLog
       logger_name: pblog
       # Delay ensures that the PB message is sent
       # after the response is sent to client, instead
       # of immediately. This ensures all Trace Spans
       # have proper end timestamps.
       delay: true

To receive all trace spans, set the delay option of the addResponseAction(). This will delay the sending of the ProtoBuf message to after the response has been sent to the client.

rl = newRemoteLogger('127.0.0.1:5301')
setOpenTelemetryTracing(true)

addAction(AllRule(), SetTraceAction(true), {name="Enable tracing"})
addResponseAction(AllRule(), RemoteLogResponseAction(rl, nil, false, {}, {}, true), {name="Do PB logging"})

Passing Trace ID and Span ID to downstream servers¶

When storing traces, it is beneficial to correlate traces of the same query through different applications. The PowerDNS Recursor (since 5.3.0) supports the experimental TRACEPARENT EDNS option to pass the trace identifier.

This can be easily achieved by adding the send_downstream_traceparent option with the desired EDNS OptionCode.

YAML

query_rules:
  - name: Add TraceID to EDNS for backend
    selector:
      type: All
    action:
      type: SetTrace
      value: true
      send_downstream_traceparent: true

addAction(AllRule(), SetTraceAction(true, {sendDownstreamTraceparent=true}), {name="Enable tracing"})

Accepting TRACEPARENT from upstream servers¶

dnsdist can also use a Trace ID and optional Span ID from an incoming query. It will not do this by default, but this can be configured to do so. When set, the Trace and Span IDs from the query will be used. Should there be no ID in the incoming query, a random ID will be generated.

YAML Lua

Set the use_incoming_traceid argument to true in the SetTrace action.

query_rules:
  - name: Enable tracing
    selector:
      # Just as an example, in production don't trace all the queries
      type: All
    action:
      type: SetTrace
      value: true
      use_incoming_traceparent: true

addAction(AllRule(), SetTraceAction(true, {useIncomingTraceparent=true}), {name="Enable tracing"})

As dnsdist keeps EDNS existing options in the query, the TRACEPARENT option is passed as-is to the backend, which might not be desirable. Using the strip_incoming_traceparent boolean option, the EDNS option will be removed from the query.

By default, dnsdist uses 65500 for the TRACEPARENT option code. This code can be changed using the traceparent_edns_option_code option in the YAML config and the traceparentOptionCode for Lua.

Note that this will only happen when value is set to true.

Accepting and sending TRACEPARENT¶

The following example makes dnsdist accept a TRACEPARENT, and update it with its own Span ID before sending it downstream:

YAML Lua

query_rules:
  - name: Enable tracing
    selector:
      # Just as an example, in production don't trace all the queries
      type: All
    action:
      type: SetTrace
      value: true
      send_downstream_traceparent: true
      use_incoming_traceparent: true

addAction(AllRule(), SetTraceAction(true, {useIncomingTraceparent=true, sendDownstreamTraceparent=true}), {name="Enable tracing"})

Creating Trace Spans from Lua¶

Added in version 2.2.0.

It is possible to create Spans inside LuaRules or LuaResponseRules in order to track performance of your Lua code. To do this, you can call the withTraceSpan() function. This function takes a string that is the name of the Span and the function with will be instrumented.

Trace Spans from LuaActions¶

function myLuaAction(dq)
  setSpanAttribute("attr-in-the-rule-span", "hello from Lua!")
  withTraceSpan(
    'my-trace-span',
    function ()
      setSpanAttribute("some.key", "some-value")
      -- Do some actual things with the DNSQuestion here
    end
  )
  return DNSAction.None
end

Within the function body, you can create more spans by calling withTraceSpan() again.

function myLuaAction(dq)
  withTraceSpan(
    'my-trace-span',
    function ()
      -- Some set up
      setSpanAttribute("some.key", "some-value")

      -- This will create a child span of 'my-trace-span'
      withTraceSpan(
        'inner-span',
        function ()
          -- Do some longer-running thing
        end
      )
    end
  )
  return DNSAction.None
end

Using withTraceSpan() or setSpanAttribute() when tracing is not enabled is completely safe and transparent. The Lua code will be run, but no Trace Span will be created.

Trace Spans from maintenance functions¶

It is possible to create Spans inside Maintenance or Maintenance callback functions in order to track performance of your Lua code. To do this, you can call the withTraceSpan() function inside your function. This function takes a string that is the name of the Span and the function with will be instrumented.

function maintenance()
  setSpanAttribute("attr-in-the-span", "hello from Lua!")
  withTraceSpan(
    'my-maintenance-trace-span',
    function ()
      setSpanAttribute("some.key", "some-value")
    end
  )
end

Within the function body, you can create more spans by calling withTraceSpan() again.

function maintenance()
  setSpanAttribute("attr-in-the-span", "hello from Lua!")
  withTraceSpan(
    'my-maintenance-trace-span',
    function ()
      setSpanAttribute("some.key", "some-value")

      -- This will create a child span of 'my-maintenance-trace-span'
      withTraceSpan(
        'inner-span',
        function ()
          -- Do something here
        end
      )
    end
  )
end

Using withTraceSpan() when tracing is disabled is completely safe and transparent. The Lua code will be run, but no Trace Span will be created.

Trace Spans from Lua threads¶

When running Lua code in its own thread (created with newThread()), Trace Spans can be created since version 2.2.0. However, compared to query traces and maintenance traces, the called code itself is responsible for sending the Traces. Sending traces can be done with sendOpenTelemetryTrace().

setOpenTelemetryTracing(true)
rl = newRemoteLogger("127.0.0.1:55555")

newThread(
  [==[
    -- setup, can use withTraceSpan
    withTraceSpan("setup", function()
        -- Do your setup here
    end)
    -- if you do use WithTraceSpan, send the setup trace before opening new spans
    sendOpenTelemetryTrace()

    while true do
        -- this is your main-loop
        withTraceSpan("newThreadRootSpan", function()
            setSpanAttribute("rootspan", "I am a root span")
            -- do useful things
            withTraceSpan("innerspan", function()
                setSpanAttribute("inner", "I am an inner span")
            end)
        end)
        -- Send the trace before going back into the main-loop
        sendOpenTelemetryTrace()
    end
]==],
  {
    interval = 1, -- send all traces
    remoteloggers = { rl },
  }
)

Functions¶

The following functions are always available, but only produce Trace Spans within the following contexts:

LuaAction()
LuaResponseAction()
LuaFFIAction()
LuaFFIResponseAction()
LuaFFIPerThreadAction()
LuaFFIPerThreadResponseAction()
maintenance()
Any function added with addMaintenanceCallback()
Inside code ran with newThread(), but requires using sendOpenTelemetryTrace()

withTraceSpan(name, func)¶

Added in version 2.2.0.

Open an OpenTelemetry Trace Span called name that instruments function func. This method can be called safely when Tracing is not enabled for the query or when dnsdist is built without Protobuf support.

Parameters:¶

name (string) – The name for this Span
function (func) – The function to run. This function takes no parameters

setSpanAttribute(key, value)¶

Added in version 2.2.0.

Add an OpenTelemetry Trace Span attribute to the current span. In the context of a LuaAction() or maintenance(), this sets an attribute on the function’s Span. When used inside the function passed to withTraceSpan(), it will set the Attribute on the enclosed span.

This method can be called safely when Tracing is not enabled for the query or when dnsdist is built without Protobuf support.

Parameters:¶

key (string) – The key for attribute
value (string) – The value of the attribute

sendOpenTelemetryTrace()¶

Added in version 2.2.0.

Only available in code run inside newThread(). When using newThread(), it is customary to use a main-loop inside the thread. This means that control is never returned to dnsdist. Hence, dnsdist can’t figure out when to send the trace.

Use this function to send the completed trace. To ensure that the Trace Spans are closed and complete, it is highly recommended to call sendOpenTelemetryTrace() outside of any withTraceSpan().