Hi @vanvan , I had a similar issue, my hint is to use rsyslog (or syslog-ng) server to take and write logs in files, then use the HF to read these files and elaborate and send them to Indexers. In this way you have two advantages: better performaces and less issues related to Splunk overload, syslogs are received also when Splunk is down. Then, if you have fast disks and you haven't a slow network, you can use parallel_pipeline to better use your CPUs. How many CPUs there are in your HFs? I had 16 CPUs and I passed to 24 to have more performant queues. then you could optimize your configuration enlarging queues that you can check on your search heads running this search: index=_internal source=*metrics.log sourcetype=splunkd group=queue
| eval name=case(name=="aggqueue","2 - Aggregation Queue",
name=="indexqueue", "4 - Indexing Queue",
name=="parsingqueue", "1 - Parsing Queue",
name=="typingqueue", "3 - Typing Queue",
name=="splunktcpin", "0 - TCP In Queue",
name=="tcpin_cooked_pqueue", "0 - TCP In Queue")
| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size)
| eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size)
| eval fill_perc=round((curr/max)*100,2)
| bin _time span=1m
| stats Median(fill_perc) AS "fill_percentage" perc90(fill_perc) AS "90_perc" max(max) AS max max(curr) AS curr by host, _time, name
| where (fill_percentage>70 AND name!="4 - Indexing Queue") OR (fill_percentage>70 AND name="4 - Indexing Queue")
| sort -_time Last hint: check the regexes on your custom add-ons to avoid unuseful overload. Ciao. Giuseppe
... View more