Event deltas with ClickStack

Event deltas pair a latency heatmap with automatic attribute analysis so you can see the shape of your trace data and find what makes the slow spans different, without writing queries. There are three ways to use it:

• Distribution mode (always on): when no heatmap selection exists, every attribute's value distribution is shown for the current span population. Useful for spotting dominant or unusually rare values (cardinality outliers).
• Comparison mode: drag a rectangle on the heatmap to compare the spans inside (Selection) against everything outside (Background). Useful for isolating deviations.
• Iterative drill-down: click any bar to filter (or exclude) on that value. The heatmap re-renders against the filtered population, so you can keep narrowing until the cause is obvious.

The screenshot above shows a single service over a twenty-four hour window with no selection drawn. The heatmap reveals that latency is not flat: the bright band drifts upward through the day, so spans that took 1 to 10 ms in the morning are taking 10 to 100 ms by evening. That progressive degradation is exactly the kind of pattern Event deltas is built to investigate.

Prerequisites

Event deltas require a Trace data source with a duration expression. Any OpenTelemetry-instrumented service producing span data works. Available in all ClickStack deployments (Managed, Open Source, ClickHouse Cloud).

Getting started

1. From the Data Source dropdown, select a source that holds traces. Source names are arbitrary; what matters is that the source is configured as a Trace type. The Event Deltas tab is only enabled for such sources.
2. In the Analysis Mode section, click the Event Deltas tab.

Event deltas is a separate analysis mode alongside Results Table and Event Patterns. Switching to it swaps the view to a heatmap and an attribute analysis grid, but your search filters and time range are preserved and you can switch back at any time.

The heatmap

The heatmap plots spans across two dimensions:

• X axis: time
• Y axis: a numeric value, defaulting to span duration in milliseconds (logarithmic scale)

Color intensity indicates event count per bucket; brighter means more spans.

You can read patterns straight off the heatmap: bimodal latency, latency spikes at specific times, a band of consistently slow spans, or a slow band that drifts upward over time (a creeping regression). To investigate a region, click and drag a rectangle on it. That becomes your Selection and switches the analysis below into comparison mode.

Distribution mode: cardinality outliers

With no selection on the heatmap, the analysis panel shows one bar chart per attribute, computed across all matching spans. The legend reads All spans (visible in the overview screenshot above).

Attributes are ranked by how concentrated their values are: those dominated by a few values appear first; uniform, high-entropy attributes are deprioritized.

Use distribution mode when you want to understand the cardinality shape of your data:

• Highs: which services, endpoints, status codes, or hosts dominate your span population? Often surfaces a single tenant, version, or route doing most of the traffic.
• Lows: values that occur but rarely. A status code that appears in just 0.5% of spans, or one host that barely shows up, can be the most interesting signal. The long tail is where regressions and bad actors hide.

Combine with the search bar to narrow the population first (e.g., only error spans, only client spans, only one endpoint), then read the distributions for that subset.

Comparison mode: deviations from normal

Click and drag a rectangle on the heatmap to enter comparison mode. The selected spans become the Selection (orange bars); everything outside becomes the Background (green bars). Each attribute chart then shows both populations side by side, sorted so the attributes with the largest divergence appear first. A value present almost exclusively in one side, or absent from one side, is the strongest candidate for what differs.

The shape of the rectangle you draw changes the question you're asking. Two common shapes are described below; a third is covered under Other selection shapes.

Use case 1: Before vs after a regression

When the heatmap shows latency drifting upward over the timeline (the slow band thickens, the bright band climbs, or a clear inflection point separates a healthy period from a degraded one), drag a tall rectangle that covers the full latency axis but only the time window after the change. The degraded period becomes the Selection; the healthy period becomes the Background.

In the example above, the Selection covers roughly the second half of the day, where the bright band has climbed from a 1 to 10 ms range into a 10 to 100 ms range. The bars below surface attributes that diverge between the degraded period and the earlier healthy one: ScopeVersion (0.57.2 strongly Selection-biased, 0.1.0 Background-biased), ScopeName (payment more frequent in the Selection), SpanAttributes.app.payment.tokenization_method (apple-pay-token Selection-biased), and SpanAttributes.app.transaction.installments (1 Selection-biased, 12 Background-biased). Read together, these are the fingerprint of a deployment: a new instrumentation library version came out alongside a code change that altered which payment paths the service is exercising.

This is the right shape when the heatmap shows a time-localized change in latency and you want to ask "what changed?". The comparison answers that question without you needing to correlate deploy logs by hand.

Use case 2: Slow versus fast

When the heatmap shows two distinct latency bands but no obvious time-localized anomaly, drag a wide rectangle that spans the entire time range but covers only the upper latency band. The slow tail becomes the Selection; the fast bulk becomes the Background.

The example above is a different service whose heatmap separates cleanly into two horizontal bands: a dense slow band at 1 to 100 ms sits above a sparser fast band at 10 µs to 1 ms. The Selection covers the full time range but only the slow band; the fast band is the Background. What's interesting here is what the comparison doesn't surface: every tracked attribute (SpanName, SpanKind, ServiceName, k8s pod metadata, container.id, host.name, OS build) shows nearly identical Selection and Background distributions. The slow and fast populations are systemically identical from the application's point of view.

When no attribute on the span explains the divergence, the cause is usually below the application: GC pauses, I/O contention, scheduler latency, cold-cache effects, noisy neighbors. The next step is to inspect host- and runtime-level metrics for the same pod over the same time window. Comparison mode reaching this verdict ("nothing on the span differs") is itself a useful result; it tells you to stop looking at span attributes and start looking at infrastructure.

This is the right shape when you want to ask "what makes the slow spans different from the fast ones?" rather than chasing a specific anomaly. A divergent attribute points at a code-path or input cause; a flat comparison points at a systemic one.

Other selection shapes

• A small focused rectangle around a localized cluster: when a tight knot of slow spans doesn't fit the rest of the heatmap (a brief burst on the right edge, a knot in the middle of an otherwise quiet band), draw a small box around just that cluster. The Selection is the cluster, the Background is everything else, and the bars surface what makes that specific cluster different. Use this when you've already spotted the anomaly visually and want to investigate its specific shape.

The choice of shape determines the question. A vertical strip asks what changed; a horizontal band asks what's slower; a small focused box asks what's special about this cluster.

Iterative drill-down

Comparison and distribution modes are most powerful when chained. Click any bar to open a popover with three actions:

• Filter: keep only spans with this value
• Exclude: remove spans with this value
• Copy: copy the value to the clipboard

After applying a filter or exclude, the heatmap selection is cleared, the heatmap re-renders against the new population, and distribution mode resumes against that filtered set. Watch how the heatmap reshapes; a successful filter visibly removes the slow band, collapses the bimodal split, or flattens the upward drift. Repeat: spot the next suspicious value, filter, look at the new heatmap, look at the new distributions. A few iterations usually narrow a regression to one or two attributes.

Note

Aggregated Other (N) buckets that collapse low-frequency values aren't clickable. To filter for a specific value within that bucket, use the search bar directly.

When the population is small enough, switch to the Results Table tab to inspect individual traces; your filters carry over.

Customize the heatmap

The gear icon in the top-right of the heatmap opens the Display Settings drawer.

Parameter	Default	Description
Scale	Log	Log handles wide latency ranges; Linear is better for narrow, uniform distributions.
Value	(Duration)/1e6	Any numeric expression: response size, error rate, a custom span attribute.
Count	count()	Aggregation for color. Switch to avg(), sum(), p95(), or expressions like countDistinct(field).

Click Apply to update the heatmap; the attribute analysis below follows.

Common scenarios where you'd change these defaults:

• Switch Scale to Linear when the latency band is narrow (for example, a service whose spans all run between 5 and 50 ms). Log scale wastes vertical range on the upper end where there is no data.
• Plot something other than duration on the Y axis. Setting Value to SpanAttributes.http.response.size lets you investigate slow large responses; an expression like if(StatusCode = 'Error', 1, 0) plots error frequency over time across services.
• Color by something other than count. Setting Count to p95(Duration) colors each bucket by tail latency rather than volume, surfacing rare-but-slow pockets that a count-based view washes out. countDistinct(TraceId) distinguishes trace volume from span volume when one trace produces many spans.

Tips for effective use

A few practices make Event deltas substantially more useful:

• Filter to a single service first. Latency varies widely across services and mixing them obscures the signal. Use the search bar to narrow to one ServiceName (or one endpoint) before you start, so the heatmap and distributions reflect a comparable population.
• Pick selections with clear visual contrast. Comparison mode works best when the Selection band is visibly distinct from the Background, for example a degraded period that begins at a recognizable moment, or a slow tail clearly separated from the bulk. Selections that overlap heavily with the rest of the data tend to surface noise rather than the actual deviation.
• Iterate filter, heatmap, filter. A single selection rarely identifies the cause. Treat the first comparison as a hypothesis, filter on the most divergent value, and re-read the new heatmap and distributions. Two or three iterations usually narrow a regression to one or two attributes.
• Use distribution mode without a selection when no contrast is yet visible (you know there is an issue but the heatmap looks uniform). Apply a hypothesis filter such as only error spans, only client spans, or only one endpoint, and let the attribute distributions point you at the highest-impact values before you draw any rectangle.

Troubleshooting

Event Deltas tab isn't visible

The Event Deltas tab under Analysis Mode only appears when a Trace source with a duration expression is selected. Verify that your data source is configured as a Trace type and has span data with duration information.

Attribute charts show few or no results

If the sample is too small (fewer than a few dozen spans), distributions may not be statistically meaningful. Widen the time range or relax your search filters.