Errors reading data from 1G file on localCluster mode with apache storm -
hi i'm using storm local cluster mode developing. ran simple code contains spout , 2 bolts, code illustration count words log file. code illustration url : http://kaviddiss.com/2013/05/17/how-to-get-started-with-storm-framework-in-5-minutes/
the code works little log files (7.3m), when seek run big log file (100m-1000m) i'm getting exceptions.
i set long delay till cluster going down. may miss configuration options here?
exceptions:
11326 [thread-6] info backtype.storm.daemon.supervisor - launching worker assignment #backtype.storm.daemon.supervisor.localassignment{:storm-id "hellostorm-1-1403522378", :executors ([3 3] [ 4 4] [2 2] [1 1])} supervisor 868aff95-7b63-44d1-ad55-2dd07d9c7ba2 on port 1024 id df052251-45ec-4bc3-a486-c2bf11a8a0fa 11336 [thread-6] info backtype.storm.daemon.worker - launching worker hellostorm-1-1403522378 on 868aff95-7b63-44d1-ad55-2dd07d9c7ba2:1024 id df052251-45ec-4bc3-a486-c2bf11a8a0fa , conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.ma x.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-xmx768m", "storm.zookeeper. session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-xmx128m", "java.library.path" "/usr/local/li b:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/tmp//77d5cd63-9539-44a4-892a-9e91553987df", "storm.messaging.netty.buffer_size" 5242880, "supervisor.w orker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.sha red.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_t hreads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts " nil, "drpc.queue.size" 128, "worker.childopts" "-xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor. frequency.secs" 3, "drpc.childopts" "-xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.w ait.strategy" "backtype.storm.spout.sleepspoutwaitstrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout. wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.defaulttopologyvalidator", "supervisor.slots.ports" (1024 1025 1026), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-xmx256m", "nimbus.thrift.port" 6627, "topol ogy.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.listdelegateserializer", "topology.disruptor.wait.strategy" "com.lm ax.disruptor.blockingwaitstrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.defaultkryofactory", "drpc.invoc ations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.simpletransportplugin", "topology.state.synchroniz ation.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.context", "logviewer.appender.name" "a1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-xmx1024m", "storm.cluster.mode" "local", "topolog y.optimize" true, "topology.max.task.parallelism" nil} 11337 [thread-6] info com.netflix.curator.framework.imps.curatorframeworkimpl - starting 11344 [thread-6-eventthread] info backtype.storm.zookeeper - zookeeper state update: :connected:none 11358 [thread-6] info com.netflix.curator.framework.imps.curatorframeworkimpl - starting 11611 [thread-6] info backtype.storm.daemon.executor - loading executor line-reader-spout:[2 2] 11618 [thread-6] info backtype.storm.daemon.executor - loaded executor tasks line-reader-spout:[2 2] 11632 [thread-16-line-reader-spout] info backtype.storm.daemon.executor - opening spout line-reader-spout:(2) start time: 18512885554479686 11634 [thread-16-line-reader-spout] info backtype.storm.daemon.executor - opened spout line-reader-spout:(2) 11636 [thread-16-line-reader-spout] info backtype.storm.daemon.executor - activating spout line-reader-spout:(2) 11638 [thread-6] info backtype.storm.daemon.executor - finished loading executor line-reader-spout:[2 2] 11677 [thread-6] info backtype.storm.daemon.executor - loading executor word-counter:[3 3] 11721 [thread-6] info backtype.storm.daemon.executor - loaded executor tasks word-counter:[3 3] 11725 [thread-6] info backtype.storm.daemon.executor - finished loading executor word-counter:[3 3] 11733 [thread-6] info backtype.storm.daemon.executor - loading executor word-spitter:[4 4] 11735 [thread-6] info backtype.storm.daemon.executor - loaded executor tasks word-spitter:[4 4] 11737 [thread-6] info backtype.storm.daemon.executor - finished loading executor word-spitter:[4 4] 11746 [thread-6] info backtype.storm.daemon.executor - loading executor __system:[-1 -1] 11747 [thread-6] info backtype.storm.daemon.executor - loaded executor tasks __system:[-1 -1] 11748 [thread-6] info backtype.storm.daemon.executor - finished loading executor __system:[-1 -1] 11761 [thread-6] info backtype.storm.daemon.executor - loading executor __acker:[1 1] 11765 [thread-6] info backtype.storm.daemon.executor - loaded executor tasks __acker:[1 1] 11767 [thread-6] info backtype.storm.daemon.executor - timeouts disabled executor __acker:[1 1] 11768 [thread-6] info backtype.storm.daemon.executor - finished loading executor __acker:[1 1] 11768 [thread-6] info backtype.storm.daemon.worker - launching receive-thread 868aff95-7b63-44d1-ad55-2dd07d9c7ba2:1024 11786 [thread-6] info backtype.storm.daemon.worker - worker has topology config {"storm.id" "hellostorm-1-1403522378", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.se cs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.k ryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.in terval.millis" 50, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.l ocal.dir" "/var/tmp//77d5cd63-9539-44a4-892a-9e91553987df", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "inputf ile" "test_log.log", "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "stor m.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "hellostorm", "topology.transfer.buffer.size" 1024, "topology.worker .childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "superviso r.monitor.frequency.secs" 3, "drpc.childopts" "-xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topolo gy.spout.wait.strategy" "backtype.storm.spout.sleepspoutwaitstrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" 1, "storm.zookeeper.retry.interval" 1000, "topology.slee p.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.defaulttopologyvalidator", "supervisor.slots.ports" (1024 1025 1026), "topology.debug" false, "nimbus.task.launc h.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-xmx25 6m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.listdelegateserializer", "top ology.disruptor.wait.strategy" "com.lmax.disruptor.blockingwaitstrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serializ ation.defaultkryofactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.simpletransp ortplugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.nett y.context", "logviewer.appender.name" "a1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-xmx1024m", "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil} 11786 [thread-6] info backtype.storm.daemon.worker - worker df052251-45ec-4bc3-a486-c2bf11a8a0fa storm hellostorm-1-1403522378 on 868aff95-7b63-44d1-ad55-2dd07d9c7ba2:1024 has finished loading 11801 [thread-18-word-counter] info backtype.storm.daemon.executor - preparing bolt word-counter:(3) 11821 [thread-18-word-counter] info backtype.storm.daemon.executor - prepared bolt word-counter:(3) 11823 [thread-20-word-spitter] info backtype.storm.daemon.executor - preparing bolt word-spitter:(4) 11825 [thread-20-word-spitter] info backtype.storm.daemon.executor - prepared bolt word-spitter:(4) 11838 [thread-24-__acker] info backtype.storm.daemon.executor - preparing bolt __acker:(1) 11840 [thread-22-__system] info backtype.storm.daemon.executor - preparing bolt __system:(-1) 11854 [thread-24-__acker] info backtype.storm.daemon.executor - prepared bolt __acker:(1) 12173 [thread-22-__system] info backtype.storm.daemon.executor - prepared bolt __system:(-1) 112055 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: suspended 112058 [main-eventthread] warn backtype.storm.cluster - received event :disconnected::none: disconnected zookeeper. 112058 [thread-6-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: suspended 112058 [thread-6-eventthread] warn backtype.storm.cluster - received event :disconnected::none: disconnected zookeeper. 121441 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: suspended 121442 [main-eventthread] warn backtype.storm.cluster - received event :disconnected::none: disconnected zookeeper. 121442 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: suspended 121442 [main-eventthread] warn backtype.storm.cluster - received event :disconnected::none: disconnected zookeeper. 121443 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: suspended 121443 [main-eventthread] warn backtype.storm.cluster - received event :disconnected::none: disconnected zookeeper. 121443 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 121444 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 134654 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: suspended 134655 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 134655 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 134656 [main-eventthread] warn com.netflix.curator.connectionstate - session expired event received 134656 [main-eventthread] warn backtype.storm.cluster - received event :disconnected::none: disconnected zookeeper. 134656 [main-eventthread] warn com.netflix.curator.connectionstate - session expired event received 134657 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: lost 134657 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 134657 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: lost 139931 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 149745 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 149745 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 149746 [main-eventthread] warn com.netflix.curator.connectionstate - session expired event received 149746 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: lost 149747 [main-eventthread] warn backtype.storm.cluster - received event :expired::none: disconnected zookeeper. 149747 [main-eventthread] warn com.netflix.curator.connectionstate - session expired event received 149747 [main-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: lost 149747 [main-eventthread] warn backtype.storm.cluster - received event :expired::none: disconnected zookeeper. 158929 [main-eventthread] warn backtype.storm.cluster - received event :expired::none: disconnected zookeeper. 158931 [main-eventthread] warn backtype.storm.cluster - received event :expired::none: disconnected zookeeper. 158931 [thread-6-eventthread] warn com.netflix.curator.connectionstate - session expired event received 158931 [thread-6-eventthread] info com.netflix.curator.framework.state.connectionstatemanager - state change: lost 158931 [thread-6-eventthread] warn backtype.storm.cluster - received event :expired::none: disconnected zookeeper. 158932 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 158933 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 176934 [connectionstatemanager-0] warn com.netflix.curator.framework.state.connectionstatemanager - there no connectionstatelisteners registered. 357333 [curatorframework-5] error com.netflix.curator.connectionstate - connection timed out org.apache.zookeeper.keeperexception$connectionlossexception: keepererrorcode = connectionloss @ com.netflix.curator.connectionstate.getzookeeper(connectionstate.java:72) ~[curator-client-1.0.1.jar:na] @ com.netflix.curator.curatorzookeeperclient.getzookeeper(curatorzookeeperclient.java:74) [curator-client-1.0.1.jar:na] @ com.netflix.curator.framework.imps.curatorframeworkimpl.getzookeeper(curatorframeworkimpl.java:353) [curator-framework-1.0.1.jar:na] @ com.netflix.curator.framework.imps.backgroundsyncimpl.performbackgroundoperation(backgroundsyncimpl.java:39) [curator-framework-1.0.1.jar:na] @ com.netflix.curator.framework.imps.operationanddata.callperformbackgroundoperation(operationanddata.java:40) [curator-framework-1.0.1.jar:na] @ com.netflix.curator.framework.imps.curatorframeworkimpl.backgroundoperationsloop(curatorframeworkimpl.java:547) [curator-framework-1.0.1.jar:na] @ com.netflix.curator.framework.imps.curatorframeworkimpl.access$200(curatorframeworkimpl.java:50) [curator-framework-1.0.1.jar:na] @ com.netflix.curator.framework.imps.curatorframeworkimpl$2.call(curatorframeworkimpl.java:177) [curator-framework-1.0.1.jar:na] @ java.util.concurrent.futuretask$sync.innerrun(futuretask.java:303) [na:1.6.0_65] @ java.util.concurrent.futuretask.run(futuretask.java:138) [na:1.6.0_65] @ java.util.concurrent.threadpoolexecutor$worker.runtask(threadpoolexecutor.java:895) [na:1.6.0_65] @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:918) [na:1.6.0_65] @ java.lang.thread.run(thread.java:680) [na:1.6.0_65] [update]
i got new exception running 70m file: 622366 [curatorframework-9] error com.netflix.curator.framework.imps.curatorframeworkimpl - background exception not retry-able or retry gave java.lang.outofmemoryerror: gc overhead limit exceeded
the problem seems described: you've loaded more info memory jvm can support. assume happening spout. big files you'll need break processing either splitting files in advance or streaming files in instead of trying load whole file memory.
apache-storm
No comments:
Post a Comment