Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2993

Storm HDFS bolt throws ClosedChannelException when Time rotation policy is used

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.2.2
    • Component/s: None

      Description

      Storm connector throws below error in the worker logs.

       

      2018-03-12 18:14:58.123 o.a.s.h.c.r.MoveFileAction Timer-3 [INFO] Moving file hdfs://ctr-e138-1518143905142-85179-01-000004.hwx.site:8020/tmp/foo/my-bolt-3-0-1520878438104.txt to /tmp/dest2/my-bolt-3-0-15 20878438104.txt 2018-03-12 18:14:58.123 o.a.s.h.c.r.MoveFileAction Timer-0 [INFO] Moving file hdfs://ctr-e138-1518143905142-85179-01-000004.hwx.site:8020/tmp/foo/my-bolt-6-0-1520878438104.txt to /tmp/dest2/my-bolt-6-0-15 20878438104.txt 2018-03-12 18:14:58.123 o.a.s.h.c.r.MoveFileAction Timer-1 [INFO] Moving file hdfs://ctr-e138-1518143905142-85179-01-000004.hwx.site:8020/tmp/foo/my-bolt-5-0-1520878438104.txt to /tmp/dest2/my-bolt-5-0-15 20878438104.txt 2018-03-12 18:14:58.124 o.a.s.h.c.r.MoveFileAction Timer-2 [INFO] Moving file hdfs://ctr-e138-1518143905142-85179-01-000004.hwx.site:8020/tmp/foo/my-bolt-4-0-1520878438104.txt to /tmp/dest2/my-bolt-4-0-15 20878438104.txt 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Timer-2 [INFO] File rotation took 28 ms. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Timer-0 [INFO] File rotation took 29 ms. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Timer-3 [INFO] File rotation took 28 ms. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Timer-1 [INFO] File rotation took 28 ms. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Thread-12-my-bolt-executor[6 6] [INFO] Tuple failed to write, forcing a flush of existing data. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Thread-8-my-bolt-executor[3 3] [INFO] Tuple failed to write, forcing a flush of existing data. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Thread-16-my-bolt-executor[5 5] [INFO] Tuple failed to write, forcing a flush of existing data. 2018-03-12 18:14:58.132 o.a.s.d.executor Thread-8-my-bolt-executor[3 3] [ERROR] java.nio.channels.ClosedChannelException: null at org.apache.hadoop.hdfs.ExceptionLastSeen.throwException4Close(ExceptionLastSeen.java:73) ~[stormjar.jar:?] at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:153) ~[stormjar.jar:?] at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105) ~[stormjar.jar:?] at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57) ~[stormjar.jar:?] at java.io.DataOutputStream.write(DataOutputStream.java:107) ~[?:1.8.0_161] at java.io.FilterOutputStream.write(FilterOutputStream.java:97) ~[?:1.8.0_161] at org.apache.storm.hdfs.common.HDFSWriter.doWrite(HDFSWriter.java:48) ~[stormjar.jar:?] at org.apache.storm.hdfs.common.AbstractHDFSWriter.write(AbstractHDFSWriter.java:40) ~[stormjar.jar:?] at org.apache.storm.hdfs.bolt.AbstractHdfsBolt.execute(AbstractHdfsBolt.java:158) [stormjar.jar:?] at org.apache.storm.daemon.executor$fn_10189$tuple_action_fn10191.invoke(executor.clj:745) [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at org.apache.storm.daemon.executor$mk_task_receiver$fn10108.invoke(executor.clj:473) [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at org.apache.storm.disruptor$clojure_handler$reify4115.onEvent(disruptor.clj:41) [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509) [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487) [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:74) [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at org.apache.storm.daemon.executor$fn10189$fn10202$fn10257.invoke(executor.clj:868) [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at org.apache.storm.util$async_loop$fn_1221.invoke(util.clj:484) [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] 2018-03-12 18:14:58.133 o.a.s.d.executor Thread-16-my-bolt-executor[5 5] 

       


       

      Apparently the Timed rotation policy does not synchronize properly so its possible that the HDFS bolt code can attempt to write to a closed writer.

       

        Attachments

          Activity

            People

            • Assignee:
              arunmahadevan Arun Mahadevan
              Reporter:
              arunmahadevan Arun Mahadevan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h
                1h