Like Jitsu? Give us a star on ⭐ GitHub!

πŸ“œ Configuration

Configuration UI

πŸ‘©β€πŸ”¬ Extending Jitsu

Overview
Destination Extensions
Source Extensions
API Specs

Jitsu Internals

Events Cache

Jitsu supports caching last events in storage (currently only Redis is supported). All income events will be stored in meta storage as well as processed events with DB data types and errors if occurred. Default cache size is 100 events per destination. It is configured in server.cache.events.size.

This feature requires meta.storage configuration.

server:
  cache:
    #true by default. If false Jitsu doesn't cache any events
    enabled: true
    events:
      #default value. Amount of events which are saved per token or destination.
      #If there are more events, old ones will be replaced with new ones.
      size: 100
      #by default Jitsu applies rate limiting for not saving all events into the cache.
      #Jitsu saves in cache ${size} events per ${time_window_sec}. Other events are skipped.
      time_window_sec: 60
      #Every ${trim_interval_ms} value Jitsu will delete events from cache
      #if size of the cache collection by token or destination is greater than ${size}
      trim_interval_ms: 500
      #When Jitsu receives malformed event JSON, event will be cut to ${max_malformed_event_size_bytes} bytes (10KB) and saved to cache
      max_malformed_event_size_bytes: 10000
    pool:
      #number of goroutines which are process events cache
      size: 10


destinations:
...

meta:
  storage:
    redis:
      host: redis_host
      port: 6379
      password: secret_password

Endpoint

Jitsu has an endpoint that return last events from cache. Last event structure:

FieldTypeDescription
malformedstringMalformed event as a string in case when Jitsu can't parse input JSON events because it is not valid JSON
originaljson objectRaw event.
successjson objectSuccessfully processed event (after enrichments, flattening, mappings, typecasting). Contains destination_id, table_name, and array of fields where every field contains name, type, value.
errorstringError if occurred.

GET/api/v1/events/cache

Method returns last events per destination in time interval with limit

X-Admin-Token
required
header
string
Admin token authorization (read more about admin auth)
limit
optional
Query String Parameter
integer
Limit per destination
ids
optional
Query String Parameter
string
Comma-separated api keys or destination ids array
namespace
optional
Query String Parameter
string
Namespace of the events. Can be destination or token constants. Default value is destination.
status
optional
Query String Parameter
string
Events status filter. If no status provided all events will be returned. Available value is 'error'. If error provided - only events with error status will be returned.

Response

Return last events with DB schema and errors

{
  "last_minute_limited": 0,
  "cache_capacity_per_interval": 100,
  "interval_seconds": 60,
  "total_events": 95,
  "response_events": 2,
  "events": [
    {
      "original": {
        "api_key": "api_secret",
        "event_type": "pageview",
        "eventn_ctx": {
          "click_id": {},
          "event_id": "dna3b213b",
          "ids": {},
          "local_tz_offset": "0",
          "page_title": "Jitsu: Open-source data integration and event collection",
          "referer": "https://referer-site.com/",
          "url": "https://jitsu.com/",
          "user": {
            "anonymous_id": "enoq4o12"
          },
          "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0",
          "utc_time": "2020-11-30T21:48:12.754000Z",
          "utm": {}
        },
        "eventn_data": {},
        "src": "eventn"
      },
      "error": "Unsupported column [local_tz_offset] type changing from: INT64 to: STRING"
    },
    {
      "original": {
        "api_key": "api_secret",
        "event_type": "pageview",
        "eventn_ctx": {
          "click_id": {},
          "event_id": "213ndisand",
          "ids": {},
          "local_tz_offset": 0,
          "page_title": "Jitsu: Open-source data integration and event collection",
          "referer": "https://referer-site.com/",
          "url": "https://jitsu.com/",
          "user": {
            "anonymous_id": "dano2131"
          },
          "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0",
          "utc_time": "2020-11-30T21:48:12.754000Z",
          "utm": {}
        },
        "eventn_data": {},
        "src": "eventn"
      },
      "success": {
        "destination_id": "my_destination",
        "table": "jitsu_events",
        "record": [
          {
            "field": "eventn_ctx_utc_time",
            "type": "timestamp",
            "value": "2020-11-30T21:48:12.754000Z"
          },
          {
            "field": "eventn_ctx_user_anonymous_id",
            "type": "character varying(8192)",
            "value": "dano2131"
          },
          {
            "field": "eventn_ctx_page_title",
            "type": "character varying(8192)",
            "value": "Jitsu: Open-source data integration and event collection"
          },
          {
            "field": "eventn_ctx_location_latitude",
            "type": "numeric(38,18)",
            "value": "50.8757"
          },
          {
            "field": "eventn_ctx_location_city",
            "type": "character varying(8192)",
            "value": "Verwood"
          },
          {
            "field": "_timestamp",
            "type": "timestamp",
            "value": "2020-11-30T21:48:13.609858Z"
          },
          {
            "field": "eventn_ctx_parsed_ua_ua_family",
            "type": "character varying(8192)",
            "value": "Firefox"
          },
          {
            "field": "eventn_ctx_parsed_ua_ua_version",
            "type": "character varying(8192)",
            "value": "78.0."
          },
          {
            "field": "eventn_ctx_parsed_ua_os_family",
            "type": "character varying(8192)",
            "value": "Linux"
          },
          {
            "field": "eventn_ctx_event_id",
            "type": "character varying(8192)",
            "value": "213ndisand"
          },
          {
            "field": "eventn_ctx_referer",
            "type": "character varying(8192)",
            "value": "https://referer-site.com/"
          },
          {
            "field": "eventn_ctx_location_zip",
            "type": "character varying(8192)",
            "value": "BH31"
          },
          {
            "field": "eventn_ctx_location_region",
            "type": "character varying(8192)",
            "value": "ENG"
          },
          {
            "field": "eventn_ctx_location_country",
            "type": "character varying(8192)",
            "value": "GB"
          },
          {
            "field": "src",
            "type": "character varying(8192)",
            "value": "eventn"
          },
          {
            "field": "event_type",
            "type": "character varying(8192)",
            "value": "pageview"
          },
          {
            "field": "eventn_ctx_local_tz_offset",
            "type": "bigint",
            "value": "0"
          },
          {
            "field": "eventn_ctx_user_agent",
            "type": "character varying(8192)",
            "value": "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
          },
          {
            "field": "eventn_ctx_url",
            "type": "character varying(8192)",
            "value": "https://jitsu.com/"
          },
          {
            "field": "source_ip",
            "type": "character varying(8192)",
            "value": "86.149.97.182"
          },
          {
            "field": "api_key",
            "type": "character varying(8192)",
            "value": "api_secret"
          },
          {
            "field": "eventn_ctx_location_longitude",
            "type": "numeric(38,18)",
            "value": "-1.8721"
          }
        ]
      }
    }
  ]
}