cachegrand uses io_uring, a standard kernel component, to perform disk and network I/O operations; this component shares some memory directly with the kernel which can't get swapped and has to be locked in place once allocated.
The max_clients parameter (explained in the next section) is dependant on the amount of lockable memory, not having
enough might prevent cachegrand from starting or, as it's used by other software e.g. browsers, desktop managers, etc.,
it might cause other kind of failures
It is also important to have a value high enough for the open files as in Linux the network connections are counted
towards this limit as well, to manage 10000 connections the open files limit must be greater than 10000, it is
usually safe to set it to unlimited or to a very high value.
$ ulimit -a
...
max locked memory (kbytes, -l) 64
...
open files (-n) 1024
...In this case, both the max locked memory and the open files limits are too low and need to be increased.
These two limits can be safely set very high or to unlimited, cachegrand will try to set very high values at the start but depending on the distribution or on the security settings the operation might not be allowed.
cachegrand uses mimalloc, a novel memory allocator, capable of performing extremely fast allocations and frees, to
allocate memory, delivering better performances than the most known jemalloc and tcmalloc.
The memory allocator is capable of leveraging the 2MB hugepages if enabled, a simple way to enable them is to set
/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages to the desired number of hugepages to allocate bearing in mind
that each page is 2 MB.
Here an example on how initialize 512 hugepages (1 GB of memory).
echo 512 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepagesThe hugepages can also be configured to at the system boot via sysctl, the specifics depend on the distribution in use.
Bear in mind that the hugepages are shared between all the processes running on the system and cachegrand has no control over them.
cachegrand uses YAML for its configuration file, an example can be found in etc/cachegrand.yaml.skel.
| Parameter name | Value type | Default value | Documentation |
|---|---|---|---|
| cpus | list (numeric, numeric ranges or all keyword) | List of cpus to bind on, values can be provided as numbers, numeric ranges or the special keyword all to bind on all the available cpus |
|
| workers_per_cpus | numeric | 1 | Number of workers per cpu, suggested 1 |
| run_in_foreground | bool | false | True/false flag to run cachegrand in the foreground (currently unsupported) |
| pidfile_path | string | /var/run/cachegrand/cachegrand.pid | Path to the pid file |
| network.backend | enum (io_uring) | io_uring | Set the backend for the network, allowed values io_uring |
| network.max_clients | numeric | 250 | Max amount of clients that can connect |
| network.listen_backlog | numeric | 100 | Max listen backlog |
| module | list (modules) | ||
| module.type | enum (redis,prometheus) | Module name, allowed values redis and prometheus | |
| module.redis.max_key_length | numeric | 8192 | Maximum allowed key length, it can't be greater than 65536 bytes |
| module.redis.max_command_length | numeric | 536870912 | Maximum allowed command length |
| module.redis.max_command_arguments | numeric | 10000 | Maximum allowed command arguments |
| module.redis.strict_parsing | bool | false | If set to true, reports error when invalid combination of arguments are used (e.g., SORT with both ASC and DESC set at the same time) |
| module.redis.require_authentication | bool | false | If set to true, require the users to authenticate via the AUTH command or via the HELLO AUTH command |
| module.redis.username | string | Username to be used for the authentication, if not specified defaults to default |
|
| module.redis.password | string | Password to be used for the authentication | |
| module.redis.disabled_commands | list (string) | List of Redis commands to disable | |
| module.network.timeout.read_ms | numeric | -1 | Read timeout in milliseconds, -1 to disable it or greater than 0 to enable it |
| module.network.timeout.write_ms | numeric | 10000 | Write timeout in milliseconds, -1 to disable it or greater than 0 to enable it |
| module.network.keepalive.time | numeric | 0 | Currently unsupported |
| module.network.keepalive.interval | numeric | 0 | Currently unsupported |
| module.network.keepalive.probes | numeric | 0 | Currently unsupported |
| module.network.tls | Optional block of parameters to enable encryption for the module_id | ||
| module.network.tls.certificate_path | string | Path to the certificate in pem format | |
| module.network.tls.private_key_path | string | Path to the private key in pem format | |
| module.network.tls.ca_certificate_chain_path | string | Optional, path to the ca certificate chain in pem format | |
| module.network.tls.verify_client_certificate | bool | false | Optional, if set to true a client must send a certificate and the certificate must be signed by a CA in the ca_certificate_chain_path |
| module.network.tls.min_version | enum (any, tls1.0, tls1.1, tls1.2, tls1.3) | any | Max TLS version allowed, allowed options any:, tls1.0, tls1.1, tls1.2, tls1.3 |
| module.network.tls.max_version | enum (any, tls1.0, tls1.1, tls1.2, tls1.3) | any | Max TLS version allowed, allowed options any:, tls1.0, tls1.1, tls1.2, tls1.3 |
| module.network.tls.cipher_suites | list | Cipher suites allowed, run //path/to/cachegrand-server --list-tls-cipher-suites to get the full list | |
| module.network.bindings | list | List of bindings to listen on (host / port tuples) | |
| module.network.bindings.host | string | 0.0.0.0 | IP Address to bind on, can be IPv4 or IPv6 if enabled in the system |
| module.network.bindings.port | numeric | 6379 | Port to listen on, ports <= 1024 require root |
| module.network.bindings.tls | bool | false | Enable or disable TLS for a specific binding |
| database.limits | |||
| database.limits.hard | |||
| database.limits.hard.max_keys | numeric | 1000000 | Maximum amount of allowed Keys, currently cachegrand doesn't autoresize the hashtable so the hashtable is always initialized with the maximum amount of keys allowed |
| database.limits.soft | |||
| database.limits.soft.max_keys | numeric | (not enabled) | Soft limit for the maximum amount of allowed keys, if the limit is reached the server will start to evict keys |
| database.snapshots | |||
| database.snapshots.path | |||
| database.snapshots.interval | |||
| database.snapshots.min_keys_changed | |||
| database.snapshots.min_data_changed | |||
| database.snapshots.rotation | |||
| database.snapshots.rotation.max_files | |||
| database.backend | enum (memory, file) | memory | Set the type of backend, allowed values memory and file |
| database.memory.limits | |||
| database.memory.limits.hard | |||
| database.memory.limits.hard.max_keys | numeric, with or without size suffix, or percentage | 75% | Maximum amount of usable memory, if the threshold is hit future inserts will be blocked |
| database.memory.limits.soft | |||
| database.memory.limits.soft.max_keys | numeric, with or without size suffix, or percentage | 70% | Soft limit for the maximum amount of usable memory, if the limit is reached the server will start to evict keys |
| database.file | list | The current implementation of the file backend is a PoC and it's limited in performances and functionalities | |
| database.file.path | string | /var/lib/cachegrand | Path to a folder to be used for the shards |
| database.file.shard_size_mb | numeric | 100 | Maximum size of a shard in MB |
| database.file.max_opened_shards | numeric | 1000 | Maximum number of shards opened (unsupported) |
| database.file.limits | |||
| database.file.limits.hard | |||
| database.file.limits.hard.max_keys | numeric, with or without size suffix, or percentage | 75% | Maximum amount of usable disk space, if the threshold is hit future inserts will be blocked |
| database.file.limits.soft | |||
| database.file.limits.soft.max_keys | numeric, with or without size suffix, or percentage | 70% | Soft limit for the maximum amount of usable disk space, if the limit is reached the server will start to evict keys |
| database.keys_eviction | |||
| database.keys_eviction.policy | enum (lru, lfu, random, ttl) | Eviction policy when the soft limits are hit, when using TTL the parameter only_ttl has to be set to true |
|
| database.keys_eviction.only_ttl | bool | When evicting, consider only keys with expiration time | |
| sentry.enable | bool | false | If enabled and if the dsn is provided, in case of a crash a minidump is automatically generated and uploaded to sentry.io - data stored in cachegrand get be uploaded if part of the stacktrace! |
| sentry.dsn | string | https://05dd54814d8149cab65ba2987d560340@o590814.ingest.sentry.io/5740234 | DSN to use with the sentry.io service |
| logs | list | List of log sinks | |
| logs.type | enum (console, file, syslog) | console, file and syslog | |
| logs.level | list set (all, debug, verbose, info, warning, recoverable, error, no-debug, no-verbose, no-info, no-warning, no-recoverable, no-error) | all, no-verbose, no-debug | Log level |
| logs.file.path | string | /var/log/cachegrand/cachegrand.log | Path to the log file |