Functions Pipeline

Overview

Jitsu provides a powerful event processing pipeline that allows you to filter, transform, and enrich events before sending them to destinations.

Pipeline Architecture

The event processing pipeline consists of 3 main steps executed sequentially:

Step 1: Builtin Transformations

These are system-level functions like Identity Stitching that run before user functions. Currently includes:

• Identity Stitching: Recognizes and merges user identities across sessions

Step 2: User Defined Function Pipeline

Your custom JavaScript functions that can:

• Filter events (exclude unwanted data)
• Transform events (modify structure/fields)
• Enrich events (add external data)

Multiple UDFs are executed sequentially in the order defined.

Step 3: Builtin Destination Function

The final function that sends the processed event to the configured destination (e.g. Mixpanel, Amplitude, Webhook, etc.)

All data warehouse destinations uses the same bridge function that passes event payload to the bulker component.

Since they are functions as any other user defined functions, they report logs and errors in the similar way. You can check logs and errors of functions attached to certain connection in the Data - Live Events section of the Workspace.

Unrecoverable Errors

Unrecoverable Errors are function or destination errors that cannot be resolved by retrying an operation, as well as retriable errors that have exceeded the maximum number of retry attempts.

Summing up, Unrecoverable Errors include:

• Any standard error thrown in a builtin function (e.g. destination functions)
• NoRetryError thrown in a user defined function
• RetryError thrown in any function that has exhausted all retry attempts

All events that encounter Unrecoverable Errors are sent to the Unrecoverable Events storage, where they can be viewed in the Live Events section of the UI.

Error Handling in Builtin Functions

Builtin transformation and destination functions follow strict error handling:

Any Standard Error

• Error is logged to destination log
• Event is sent to Unrecoverable Events storage (viewable in UI Live Events section)

RetryError

throw new RetryError("Temporary failure");

• Error is logged to destination log
• Event is sent to retry queue. After retry, processing resumes from the failed step with payload changes retained.

Error Handling in User Defined Functions

User-defined functions have more flexible error handling to support partial enrichment and graceful degradation:

Any Standard Error

throw new Error("Enrichment API failed");

• Error is logged to destination log
• Event continues to next function in pipeline
• Processing is NOT stopped
• Use case: Allows non-enriched events to reach destination

RetryError (default)

throw new RetryError("Temporary enrichment failure");

• Error is logged to destination log
• Event goes to retry queue (will be retried later)
• Event continues to next function in pipeline
• Use case: Enrichment that can be retried later, allows partial data now, full data after retry

RetryError with drop option

throw new RetryError("Critical enrichment failed", { drop: true });

• Error is logged to destination log
• Event goes to retry queue (will be retried later)
• Pipeline STOPS immediately
• Event does NOT reach destination
• Use case: When partial/unenriched data should not be stored

NoRetryError

throw new NoRetryError("Invalid data format");

• Error is logged to destination log
• Event goes to Unrecoverable Events storage
• Pipeline STOPS immediately
• Event does NOT reach destination
• No retries
• Use case: Permanent errors that won't be fixed by retrying

Error Handling Summary Table

Retry Attempts and Delay

Default Retry Policy

When a function throws a RetryError, Jitsu implements automatic retry logic with the following defaults:

• Retry attempts: 3 attempts maximum
• Delays between attempts: 10 minutes, 100 minutes, and 1000 minutes (16.7 hours)
• Maximum delay: 1440 minutes (24 hours) - any delay exceeding this will be capped

Important: When all retry attempts are exhausted, the event is sent to Unrecoverable Events storage where it can be viewed in the Live Events section of the UI.

Custom Retry Policy Configuration

You can override the default retry policy by adding a configuration block to your function code:

export const config = {
  retryPolicy: {
    attempts: 2,        // Number of retry attempts (max 3)
    delays: [60, 1440]  // Delays in minutes before each attempt (max 1440 per delay)
  }
}
 
export default async function(event, ctx) {
  // Your function logic here
}

Configuration Constraints

• attempts: Cannot exceed 3 (system limit)
• delays:
  • Individual delays cannot exceed 1440 minutes (24 hours)
  • Array length must match the attempts count
  • Delays are specified in minutes

How Retries Work

1. First Failure: When a function throws RetryError, the event is sent to the retry queue
2. Scheduled Retry: The event is scheduled for retry after the delay specified in delays[0]
3. Subsequent Failures: If the function fails again, the event is retried after delays[1], then delays[2], etc.
4. Exhausted Attempts: After all retry attempts are exhausted, the event is moved to Unrecoverable Events storage
5. Success: If any retry succeeds, the event continues through the pipeline normally

Retry Behavior with Pipeline Execution

• Retried events skip already processed steps: When an event is retried, it resumes from the failed pipeline step, not from the beginning
• Payload changes are retained: Any modifications made in previous successful steps are preserved
• User Defined Function Pipeline considered as a single step: If a function in the UDF pipeline fails and event is retried, the entire UDF pipeline is re-executed
• Pipeline continuation:
  • With RetryError (no drop): Event continues to next function while also being queued for retry
  • With RetryError({ drop: true }): Event is queued for retry but does NOT continue to next function

Retry Mechanism Flow Diagram