Filtering noisy game telemetry and app events

Heartbeats, per-frame telemetry and test-account events sit in Mixpanel, Amplitude and the warehouse next to the events your team analyzes, and each one adds to what those tools count. They come from the same app SDK or game client as everything else, so the fix belongs in the pipeline.

A Jitsu function on each connection drops them before that destination sees them and keeps a stable 1-in-N sample of performance telemetry. It runs in Jitsu, so changing it needs no app release, and the racing game numbers show what removing them saves.

TL;DR

• Heartbeats, frame ticks and events from signed-in QA accounts never reach a destination whose connection runs drop-noise, and performance samples arrive from 1 in N installs.
• A volume query shows which events to remove before you remove anything.

The noise and what happens to it

The full tracking plan CSV has the properties and types for each event.

Find the noise first

Before writing any filter, count what each event contributes to a typical week's volume in ClickHouse:

-- Which track events make up the volume. Run it before deciding what to drop.
SELECT
    event,
    count() AS events,
    round(count() / sum(count()) OVER (), 3) AS share_of_track_events,
    uniqExact(anonymous_id) AS installs_sending_it
FROM events FINAL
WHERE type = 'track' AND timestamp >= today() - 7
GROUP BY event
ORDER BY events DESC
LIMIT 20;

Sort the results by installs_sending_it as well as by raw event volume: nearly every install sends a heartbeat, while a debug event comes from a handful of internal installs running a test build of the game. Run the same query on events_unplanned too if plan-guard from the tracking plan page is on, since events outside the plan, such as a debug event an old build still sends, land there. A week after the function goes live, the same query shows the removed events gone and performance_sample at about a tenth of its old count.

The function

Build drop-noise with jitsu-cli, the same way as the tracking plan functions:

import { JitsuFunction } from "@jitsu/protocols/functions";
import { AnalyticsServerEvent } from "@jitsu/protocols/analytics";

export const config = {
  slug: "drop-noise",
  name: "Drop and sample telemetry",
  description: "Drops heartbeats, frame ticks and QA accounts. Keeps performance samples from 1 in N players.",
};

// Events nobody reads one row at a time. Returning "drop" keeps them out of every destination.
const DROP_EVENTS = new Set(["heartbeat", "frame_tick"]);

// FNV-1a (32-bit). Stable across runs, so the same player always lands in the same bucket.
function bucketOf(id: string, buckets: number): number {
  let h = 0x811c9dc5;
  for (let i = 0; i < id.length; i++) {
    h ^= id.charCodeAt(i);
    h = Math.imul(h, 0x01000193);
  }
  return (h >>> 0) % buckets;
}

const dropNoise: JitsuFunction<AnalyticsServerEvent> = async event => {
  const name = event.type === "track" && typeof event.event === "string" ? event.event : "";
  if (DROP_EVENTS.has(name)) {
    return "drop";
  }

  // Comma-separated user IDs of internal test accounts. Set QA_USER_IDS on the connection.
  const qaUsers = (process.env.QA_USER_IDS ?? "").split(",").map(id => id.trim()).filter(Boolean);
  if (event.userId && qaUsers.includes(String(event.userId))) {
    return "drop";
  }

  if (name === "performance_sample") {
    // Keep 1 in PERF_SAMPLE_EVERY players (default 10). A kept player keeps every sample.
    const every = Math.max(1, Math.floor(Number(process.env.PERF_SAMPLE_EVERY) || 10));
    const player = event.anonymousId || event.userId;
    if (!player || bucketOf(String(player), every) !== 0) {
      return "drop";
    }
    return { ...event, properties: { ...(event.properties ?? {}), sample_rate: every } };
  }

  return event;
};

export default dropNoise;

Set two environment variables on the connection: QA_USER_IDS, a comma-separated list of internal account IDs, and PERF_SAMPLE_EVERY, the sampling rate, which defaults to 10. Functions read them through process.env (runtime docs), so adding a new tester means editing one variable on the connection. DROP_EVENTS is a list inside the function itself, so adding a name to it means redeploying the function. Before it goes live, send it a test heartbeat and a level_end the way the tracking plan page tests functions: the heartbeat should come back dropped and the level_end unchanged.

Sampling is per install, not per event. A kept install keeps every sample, so its frame-rate distribution stays intact. Raise PERF_SAMPLE_EVERY to sample fewer installs, or set it to 1 to keep every install while you chase a frame-rate problem on one device model.

QA_USER_IDS catches testers once they sign in with their internal accounts on a production build. Whatever a tester's build sends before sign-in carries no userId and passes through to every destination on the stream. Give internal builds their own Jitsu site and write key to keep all of their events, signed in or not, out of production destinations. Events QA accounts sent before the filter went live stay in the warehouse, where a delete by user_id clears them from past funnels.

The billing rule

Where the function runs decides what a drop saves: Jitsu bills an event once if it reaches any destination, so a heartbeat dropped on the Mixpanel connection but delivered to ClickHouse still counts. An event goes unbilled only when every connection drops it and no function called fetch before the drop, so attach drop-noise to every connection on the stream: ClickHouse or another warehouse, Mixpanel, Amplitude, PostHog, Braze, Statsig and Webhook. The cost table on the overview works three more volumes through with the same pricing.

What not to drop

• Anything you count or sum. Purchases, subscriptions, sign-ups and level_end stay unsampled.
• Events a destination triggers on. A Braze campaign that fires on an event needs that event on the Braze connection, so check campaign triggers in Braze before adding a name to DROP_EVENTS.
• First-session events. Application Installed, onboarding steps and the first Application Opened define retention cohorts.
• Events you will want later. Jitsu Cloud keeps a raw backup of every accepted event, 7 days on Free and up to 90 days on paid plans, and the Jitsu team restores from it if a filter rule removed too much (event backups).

Reading sampled frame rates

After sampling, read performance data back in ClickHouse with the rate applied, one row per device model:

-- Frame rate by device model from a 1-in-N player sample.
-- sum(sample_rate) estimates how many samples the full player base sent.
SELECT
    device_model,
    sum(sample_rate) AS estimated_samples,
    uniqExact(anonymous_id) AS sampled_installs,
    quantileExact(0.5)(fps_p10) AS median_fps_p10
FROM events FINAL
WHERE type = 'track' AND event = 'performance_sample' AND timestamp >= today() - 7
GROUP BY device_model
ORDER BY estimated_samples DESC
LIMIT 20;

Only sampled installs feed median_fps_p10, a tenth of installs at the default rate, while estimated_samples multiplies each kept row by its sample_rate.

The numbers for a racing game

A racing game with an online lobby and 100,000 monthly players sends 30M events a month: 12M lobby heartbeats, 10M performance samples and 8M race results, menu events and purchases. Its stream feeds ClickHouse, Amplitude and a Webhook to an internal anti-cheat service, with the QA team's account IDs in QA_USER_IDS.

With the filter in place, the heartbeats go entirely, and sampling 1 in 10 installs keeps 1M of the samples, so 9M active events are billed: $379 a month on Business, against $1,219 for all 30M. For players, the lobby works exactly as before, since heartbeats still do their job inside the app. Amplitude and the anti-cheat service receive every race result without the heartbeats between them, so dashboards and checks built on race results keep working, and the 1M kept samples in ClickHouse are enough to rank the game's device models by frame rate on every build.

Questions people ask

Does every destination have to use the same sampling rate? No. PERF_SAMPLE_EVERY is set on each connection, like every function environment variable, so ClickHouse can keep 1 in 10 installs while Amplitude keeps 1 in 50.

Can the filter match on a property instead of an event name? With a small change to the function. drop-noise checks event names and user IDs today, and a condition on any event property fits in the same place.

Where does a raw, unfiltered copy go? Functions attach per connection, so an Amazon S3 or Google Cloud Storage connection can run without the filter and keep one; the billing rule section above covers what that copy costs.

Get started with Jitsu

Sign up for Jitsu Cloud and try drop-noise on a test stream before attaching it to production. A demo call can size the saving against your own volume.