Most of those configuration parameters are available in
beta version only (read more about release cycles).
Retroactive User Recognition is using Redis to store all anonymous incoming events. Usually, the volume of such event can be pretty huge, so Redis can take a lot of RAM (note: Redis keeps all data in RAM, disk is used only for persistence)
Approximate of event in Redis is
1KB. So if the project processes
requests per second, and let's say 80% of them are anonymous, the daily consumption will be:
8 events * 86,400 (seconds per day) = ~700Mb
By default, Jitsu will mark users not active for 7 days for eviction, so total Redis consumption will be around ~5G
(number of inactivity days is controlled by
Redis keys for user recognition format is
anonymous_events:destination_id#my_postgres:anonymous_id#bqsyuwusxh. All events generated by this user
will be stored under this key is list
The most reliable way to handle the RAM problem is having a dedicated Redis instance for user recognition with memory limit and keys eviction configuration. You can configure it with the following steps:
1. Configure separate Redis in Jitsu Server configuration file (eventnative.yaml) with anonymous events TTL (default value is 7 days):#
users_recognition: enabled: true identification_nodes: - /user/id - /user/email redis: host: redis_host port: 6379 password: secret_password ttl_minutes: #Optional. Anonymous events TTL in minutes. Default value: 10080 (7 days) anonymous_events: 10080 #7 days
Alternatively, you can configure dedicated redis with env variable (
USER_RECOGNITION_REDIS_URL=redis://:password@host:port) and TTL with
TTL (time to live) is applied to Redis keys. It means that events from anonymous users that no longer visit your web pages will be deleted from Redis at the expiration time. Under the hood Jitsu uses expire command after saving every anonymous event JSON. Since Redis anonymous events key contains user anonymous ID, Redis will automatically delete key if there was no events from a certain anonymous user 7 days (TTL value).
2. Specify Redis memory limit and eviction policy by adding the following configuration into the
maxmemory 25gb maxmemory-policy allkeys-lru maxmemory-samples 5
Or use command
CONFIG SET with
CONFIG SET maxmemory 25gb CONFIG SET maxmemory-policy allkeys-lru CONFIG SET maxmemory-samples 5
CONFIG SET command change current Redis configuration, but doesn't persist the configuration between server restarts. Make sure that you have added new values to the configuration file after using
CONFIG SET command.
After this configuration, user recognition Redis will automatically rewrite older keys when RAM is almost full.
Add Redis compression configuration, so all Redis hash tables with length > 1 (anonymous with more than 1 event inside 1 key) will be converted to the Redis ziplists and compressed. Compression rate is about 8x-10x. It saves about 800-1000% of RAM.
- Compression isn't applied to old data. Compression applies only to Redis hash tables which have been created after this configuration.
- Compression applies only to hash tables with length > 1. It means that size of a hash table with 1 event =~ 1KB, and size with hash table with 2 events =~1.08KB, 3 events =~1.16KB.
list-compress-depth 1 hash-max-ziplist-value 2000
Or use command
CONFIG SET with
CONFIG SET list-compress-depth 1 CONFIG SET hash-max-ziplist-value 2000
You can check how much memory any key consumes with
DEBUG OBJECT <REDIS KEY>
If you can't change the redis configuration, you can enable GZIP compression on the Jitsu side. It will approximately reduce Redis memory consumption by 30%.
users_recognition.compression: 'gzip' to the Jitsu Server configuration (eventnative.yaml):
users_recognition: enabled: true compression: 'gzip' identification_nodes: - /user/id - /user/email redis: host: redis_host port: 6379 password: secret_password ttl_minutes: #Optional. Anonymous events TTL in minutes. Default value: 10080 (7 days) anonymous_events: 10080 #7 days
We do not recommend use Redis ziplist compression and Jitsu GZIP compression at the same time