Questions tagged [rocksdb]
RocksDB is an embeddable persistent key-value store for fast storage. RocksDB can also be the foundation for a client-server database but our current focus is on embedded workloads.
rocksdb
509
questions
0
votes
0
answers
16
views
Why is RocksDB Memtable exceeding its memory limit?
We are using Kafka Streams and Kafka and we are having an Out of Memory issue with one of our applications because the non-jvm memory is exceeding its limit.
On further investigation, we can observe ...
- 1
0
votes
1
answer
58
views
+50
How to create data-frame on rocks db (SST files)
We hold our documents in rocks-db. We will be syncing these rocks-db sst files to S3. I would like to create a dataframe on the SST files and later run an sql. When googled, I was not able to find any ...
- 729
1
vote
2
answers
96
views
sigaction handler is waiting for lock indefinitely
I recently written a signal handler which uses backtrace from execinfo.h and it is working fine on MacOs but when it is used on Linux (Ubuntu Debian) it is waiting for lock indefinitely. I'm not sure ...
- 110
0
votes
0
answers
18
views
RocksDB Exception - Missing Manifest file
We are running a spring kafka stream binder based stateful application which has some stateful processing and uses internal rocksdb for local state store replication from kafka topic. The application ...
- 9
0
votes
0
answers
15
views
is it good idea to merge CDC data with MongoDB snapshot using RockDB and generate final snapshot?
Currently we have an inhouse mongo-cdc process which is generating JSON files and writes them to s3. We have an initial snapshot-parquet-dump of mongodb on s3. We have EOD spark job that merges the ...
- 729
0
votes
0
answers
49
views
RocksDB Put binary data
RocksDB claims to support "arbitrary byte array" insertion. However in C++ implementation I only see the following declaration of key Type Slice:
Slice(const char* d, size_t n) : data_(d), ...
- 13
0
votes
0
answers
27
views
Reading .sst file generated by apache flink as part of saving state to rocksdb
I am using rocksdb statebackend with incremental checkpoints as my check points size keeps growing i wanted to read and understand what is present inside my checkpoints.
In rocks db local dir, sst ...
- 161
0
votes
0
answers
27
views
How max_background_compactions in RocksDB DBench affects Compaction Stats (L0'Rnp1)?
I run rocksdb db_bench with configurations
dbench --benchmarks="fillrandom,stats" --max_background_compactions=1 --num=100000000 --key_size=24 --value_size=1000
and
dbench --benchmarks=&...
0
votes
2
answers
151
views
Comparison of .NET Key value pair Collections
I write in C# and am faced with the requirement of storing/retrieving a large number of string Key Value pairs from disk. To minimize the total amount of Ram consumed by the application, I would read ...
- 321
0
votes
0
answers
28
views
How is flink rocks DB live data size and checkpoint size is related?
I have flink pipeline with RMQ source, filters, keyby, enrichers, map, aggregator and tumbling window. Rockdb incremental checkpoints is enabled.
I have enabled rocks db metrics to monitor my growing ...
- 161
0
votes
0
answers
31
views
High Disk Usage when using RocksDBStateStoreProvider in Spark Structured Streaming
I have a stateful Spark SS (version 3.3.1) application that processes input events in pattern as the image shown.
And I use RoksDBStateStoreProvider to maintain state in memory and disk. There are ...
- 509
0
votes
0
answers
41
views
Rocksdb Compilation issue after upgrading Macos
I upgraded my macos to latest version and from then I am facing the below issue while compiling Rocksdb-5.8
error: no matching constructor for initialization of 'std::lock_guard<std::mutex>'
...
- 110
0
votes
0
answers
18
views
Understanding rocksdb statistics
I am storing key-value pairs in two ways on rocksdb using their Java API and comparing their stats to see which gives better results. These are the two approaches that I have followed-
version 1
...
- 394
0
votes
1
answer
55
views
How to test rocksDB compactions in apache flink checkpoints
I have a flink pipeline running with some filters,maps,aggregator and windows operators. RocksDB backend with incremental checkpointing is enabled.
My checkpoint data size is growing gradually. I ...
- 161
0
votes
1
answer
80
views
How to configure State TTL for aggregateFunction along with windows operator
I am having a flink streaming pipeline with Rabbitmq source,some filter, map , aggregatorFunction and windows opertors (Tumbling window with 5mins), Rabbitmq sink configured. And I'm using incremental ...
- 161