cloud - why storm performance are getting very slow after few minutes? -

cloud - why storm performance are getting very slow after few minutes? -

i'm running throughput topology testing performance. in first 2 minutes i'm getting performance average of 450k emitted/sec after 10 minutes goes downwards average of 100k per/sec.

what cause?

throughput test: https://github.com/stormprocessor/storm-benchmark/blob/master/src/jvm/storm/benchmark/throughputtest.java

there can several reasons possible: 1. check if heap requirement met , same provided worker jvm. via linux top command/gc logs can verify same. in case want increment utilize xmx parameters. parameters :

-xmsag -xmxag -xx:+printgcdetails -xx:-printgcapplicationstoppedtime -xx:+printgctimestamps -xloggc:/tmp/gc.log

huge major gc cycles major gc cycles take lot of time , cause application pauses big heap. read blog optimize same : https://engineering.linkedin.com/garbage-collection/garbage-collection-optimization-high-throughput-and-low-latency-java-applications

a zombie arraylist/hashmap clogging code - utilize profiler jvisualvm or yourkit point issue out if unable via code review.

check storm ui find out non-performant bolts , tweak them accordingly. also, prefer cms gc tuning parameters per requirement. g1 garbage collector give speedy collection cycles minimal application stopped time show blown process memory compared cms. prefer g1, in-case don't mind larger process size(much big compared heap size).

cloud bigdata apache-storm