# Functions Pipeline
## Overview
Jitsu provides a powerful event processing pipeline that allows you to filter, transform, and enrich events before sending them to destinations.
## Pipeline Architecture
The event processing pipeline consists of **3 main steps** executed sequentially:
### Step 1: Builtin Transformations
These are system-level functions like Identity Stitching that run before user functions. Currently includes:
- **Identity Stitching**: Recognizes and merges user identities across sessions
### Step 2: User Defined Function Pipeline
Your custom JavaScript functions that can:
- Filter events (exclude unwanted data)
- Transform events (modify structure/fields)
- Enrich events (add external data)
Multiple UDFs are executed **sequentially** in the order defined.
### Step 3: Builtin Destination Function
The final function that sends the processed event to the configured destination (e.g. Mixpanel, Amplitude, Webhook, etc.)
All data warehouse destinations uses the same bridge function that passes event payload to the bulker component.
Since they are functions as any other user defined functions, they report logs and errors in the similar way.
You can check logs and errors of functions attached to certain connection in the Data - Live Events section of the Workspace.
## Unrecoverable Errors
Unrecoverable Errors are function or destination errors that cannot be resolved by retrying an operation,
as well as retriable errors that have exceeded the maximum number of retry attempts.
Summing up, Unrecoverable Errors include:
- Any standard error thrown in a builtin function (e.g. destination functions)
- `NoRetryError` thrown in a user defined function
- `RetryError` thrown in any function that has exhausted all retry attempts
All events that encounter Unrecoverable Errors are sent to the **Unrecoverable Events storage**,
where they can be viewed in the **Live Events** section of the UI.
## Error Handling in Builtin Functions
Builtin transformation and destination functions follow strict error handling:
### Any Standard Error
- Error is **logged** to destination log
- Event is sent to **Unrecoverable Events storage** (viewable in UI Live Events section)
### RetryError
```javascript
throw new RetryError("Temporary failure");
```
- Error is **logged** to destination log
- Event is sent to **retry queue**. After retry, processing resumes from the failed step with payload changes retained.
## Error Handling in User Defined Functions
User-defined functions have **more flexible error handling** to support partial enrichment and graceful degradation:
### Any Standard Error
```javascript
throw new Error("Enrichment API failed");
```
- Error is **logged** to destination log
- Event **continues to next function** in pipeline
- Processing is **NOT stopped**
- Use case: Allows non-enriched events to reach destination
### RetryError (default)
```javascript
throw new RetryError("Temporary enrichment failure");
```
- Error is **logged** to destination log
- Event goes to **retry queue** (will be retried later)
- Event **continues to next function** in pipeline
- Use case: Enrichment that can be retried later, allows partial data now, full data after retry
### RetryError with drop option
```javascript
throw new RetryError("Critical enrichment failed", { drop: true });
```
- Error is **logged** to destination log
- Event goes to **retry queue** (will be retried later)
- Pipeline **STOPS** immediately
- Event does **NOT** reach destination
- Use case: When partial/unenriched data should not be stored
### NoRetryError
```javascript
throw new NoRetryError("Invalid data format");
```
- Error is **logged** to destination log
- Event goes to **Unrecoverable Events storage**
- Pipeline **STOPS** immediately
- Event does **NOT** reach destination
- No retries
- Use case: Permanent errors that won't be fixed by retrying
### Error Handling Summary Table
| Error Type | Location | Retry Queue | Unrecoverable Events | Continue Pipeline | Use Case |
|-------------------|------------------|-------------|-------------|-------------------|------------------------------------------|
| Standard Error | Builtin Function | ❌ | ✅ | ❌ | Permanent builtin failure |
| RetryError | Builtin Function | ✅ | ❌ | ❌ (after retry) | Temporary builtin failure |
| Standard Error | User Function | ❌ | ❌ | ✅ | Non-critical UDF failure |
| RetryError | User Function | ✅ | ❌ | ✅ | Retriable enrichment, allow partial data |
| RetryError (drop) | User Function | ✅ | ❌ | ❌ | Retriable enrichment, no partial data |
| NoRetryError | User Function | ❌ | ✅ | ❌ | Permanent data issue |
## Retry Attempts and Delay
### Default Retry Policy
When a function throws a `RetryError`, Jitsu implements automatic retry logic with the following defaults:
- **Retry attempts**: 3 attempts maximum
- **Delays between attempts**: 10 minutes, 100 minutes, and 1000 minutes (16.7 hours)
- **Maximum delay**: 1440 minutes (24 hours) - any delay exceeding this will be capped
**Important**: When all retry attempts are exhausted, the event is sent to **Unrecoverable Events storage** where it can be viewed in the Live Events section of the UI.
### Custom Retry Policy Configuration
You can override the default retry policy by adding a configuration block to your function code:
```javascript
export const config = {
retryPolicy: {
attempts: 2, // Number of retry attempts (max 3)
delays: [60, 1440] // Delays in minutes before each attempt (max 1440 per delay)
}
}
export default async function(event, ctx) {
// Your function logic here
}
```
#### Configuration Constraints
- **`attempts`**: Cannot exceed 3 (system limit)
- **`delays`**:
- Individual delays cannot exceed 1440 minutes (24 hours)
- Array length must match the `attempts` count
- Delays are specified in minutes
### How Retries Work
1. **First Failure**: When a function throws `RetryError`, the event is sent to the retry queue
2. **Scheduled Retry**: The event is scheduled for retry after the delay specified in `delays[0]`
3. **Subsequent Failures**: If the function fails again, the event is retried after `delays[1]`, then `delays[2]`, etc.
4. **Exhausted Attempts**: After all retry attempts are exhausted, the event is moved to **Unrecoverable Events storage**
5. **Success**: If any retry succeeds, the event continues through the pipeline normally
### Retry Behavior with Pipeline Execution
- **Retried events skip already processed steps**: When an event is retried, it resumes from the failed pipeline step, not from the beginning
- **Payload changes are retained**: Any modifications made in previous successful steps are preserved
- **User Defined Function Pipeline** considered as a single step: If a function in the UDF pipeline fails and event is retried, the entire UDF pipeline is re-executed
- **Pipeline continuation**:
- With `RetryError` (no drop): Event continues to next function while also being queued for retry
- With `RetryError({ drop: true })`: Event is queued for retry but does NOT continue to next function
### Retry Mechanism Flow Diagram