Like Jitsu? Give us a star on ⭐ GitHub!

πŸ“œ Configuration

Configuration UI

πŸ‘©β€πŸ”¬ Extending Jitsu

Overview
Destination Extensions
Source Extensions
API Specs

Jitsu Internals

Redis optimization

Most of those configuration parameters are available in beta version only (read more about release cycles).

Retroactive User Recognition is using Redis to store all anonymous incoming events. Usually, the volume of such event can be pretty huge, so Redis can take a lot of RAM (note: Redis keeps all data in RAM, disk is used only for persistence)

Approximate of event in Redis is 1KB. So if the project processes requests per second, and let's say 80% of them are anonymous, the daily consumption will be:

8 events * 86,400 (seconds per day) = ~700Mb

By default, Jitsu will mark users not active for 7 days for eviction, so total Redis consumption will be around ~5G (number of inactivity days is controlled by USER_RECOGNITION_TTL_MINUES)

Redis keys for user recognition format is anonymous_events:destination_id#${destinationID}:anonymous_id#${userAnonymousID}. Example: anonymous_events:destination_id#my_postgres:anonymous_id#bqsyuwusxh. All events generated by this user will be stored under this key is list

The most reliable way to handle the RAM problem is having a dedicated Redis instance for user recognition with memory limit and keys eviction configuration. You can configure it with the following steps:

1. Configure separate Redis in Jitsu Server configuration file (eventnative.yaml) with anonymous events TTL (default value is 7 days):

users_recognition:
  enabled: true
  identification_nodes:
    - /user/id
    - /user/email
  redis:
    host: redis_host
    port: 6379
    password: secret_password
    ttl_minutes: #Optional. Anonymous events TTL in minutes. Default value: 10080 (7 days)
      anonymous_events: 10080 #7 days

Alternatively, you can configure dedicated redis with env variable (USER_RECOGNITION_REDIS_URL=redis://:password@host:port) and TTL with USER_RECOGNITION_TTL_MINUTES

TTL (time to live) is applied to Redis keys. It means that events from anonymous users that no longer visit your web pages will be deleted from Redis at the expiration time. Under the hood Jitsu uses expire command after saving every anonymous event JSON. Since Redis anonymous events key contains user anonymous ID, Redis will automatically delete key if there was no events from a certain anonymous user 7 days (TTL value).

2. Specify Redis memory limit and eviction policy by adding the following configuration into the redis.conf file:

maxmemory 25gb maxmemory-policy allkeys-lru maxmemory-samples 5

Or use command CONFIG SET with redis-cli:

CONFIG SET maxmemory 25gb CONFIG SET maxmemory-policy allkeys-lru CONFIG SET maxmemory-samples 5

CONFIG SET command change current Redis configuration, but doesn't persist the configuration between server restarts. Make sure that you have added new values to the configuration file after using CONFIG SET command.

After this configuration, user recognition Redis will automatically rewrite older keys when RAM is almost full.

Add Redis compression configuration, so all Redis hash tables with length > 1 (anonymous with more than 1 event inside 1 key) will be converted to the Redis ziplists and compressed. Compression rate is about 8x-10x. It saves about 800-1000% of RAM.

  • Compression isn't applied to old data. Compression applies only to Redis hash tables which have been created after this configuration.
  • Compression applies only to hash tables with length > 1. It means that size of a hash table with 1 event =~ 1KB, and size with hash table with 2 events =~1.08KB, 3 events =~1.16KB.

Configuration:

list-compress-depth 1 hash-max-ziplist-value 2000

Or use command CONFIG SET with redis-cli:

CONFIG SET list-compress-depth 1 CONFIG SET hash-max-ziplist-value 2000

You can check how much memory any key consumes with redis-cli command:

DEBUG OBJECT <REDIS KEY>

If you can't change the redis configuration, you can enable GZIP compression on the Jitsu side. It will approximately reduce Redis memory consumption by 30%. Just add users_recognition.compression: 'gzip' to the Jitsu Server configuration (eventnative.yaml):

users_recognition:
  enabled: true
  compression: 'gzip'
  identification_nodes:
    - /user/id
    - /user/email
  redis:
    host: redis_host
    port: 6379
    password: secret_password
    ttl_minutes: #Optional. Anonymous events TTL in minutes. Default value: 10080 (7 days)
      anonymous_events: 10080 #7 days

We do not recommend use Redis ziplist compression and Jitsu GZIP compression at the same time