Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2095

Nimbus dies and never recovers due to java.nio.file.DirectoryNotEmptyException

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.0.3
    • Fix Version/s: 2.0.0, 1.1.0, 1.0.3
    • Component/s: blobstore
    • Labels:
      None

      Description

      To Recreate:
      --------------------------------------
      1) Create a blobstore key for a large file (1 or 2 GB). Size of the file does not matter if nimbus can be killed while the blob is being created.
      2) while the blob is being created, restart nimbus (this is easiest way to regenerate, there can be various reasons due to which a blob couldn't be successfully created in nimbus)
      3) When nimbus tries to start on restart, it will keep dying due to DirectoryNotEmptyException and never come up.

      Expected Behavior
      --------------------------------------
      Partial blobstore key is deleted cleanly and doesn't affect nimbus.

      The actual, incorrect behavior.
      --------------------------------------
      2016-09-14 15:07:48.518 o.a.s.zookeeper [INFO] Queued up for leader lock.
      2016-09-14 15:07:48.576 o.a.s.zookeeper [INFO] xxx gained leadership
      2016-09-14 15:07:48.581 o.a.s.d.nimbus [ERROR] Error on initialization of server service-handler
      java.lang.RuntimeException: java.nio.file.DirectoryNotEmptyException: /opt/storm/storm-local/blobs/955/some_big_file
      at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:229)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
      at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
      at org.apache.storm.daemon.nimbus$setup_blobstore.invoke(nimbus.clj:1196)
      at org.apache.storm.daemon.nimbus$fn_7064$exec_fn2461auto___7065.invoke(nimbus.clj:1416)
      at clojure.lang.AFn.applyToHelper(AFn.java:156)
      at clojure.lang.AFn.applyTo(AFn.java:144)
      at clojure.core$apply.invoke(core.clj:630)
      at org.apache.storm.daemon.nimbus$fn_7064$service_handler_7308.doInvoke(nimbus.clj:1358)
      at clojure.lang.RestFn.invoke(RestFn.java:421)
      at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2206)
      at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2239)
      at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2262)
      at clojure.lang.AFn.applyToHelper(AFn.java:152)
      at clojure.lang.AFn.applyTo(AFn.java:144)
      at org.apache.storm.daemon.nimbus.main(Unknown Source)
      Caused by: java.nio.file.DirectoryNotEmptyException: /opt/storm/storm-local/blobs/955/some_big_file
      at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
      at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
      at java.nio.file.Files.deleteIfExists(Files.java:1165)
      at org.apache.storm.blobstore.FileBlobStoreImpl.delete(FileBlobStoreImpl.java:239)
      at org.apache.storm.blobstore.FileBlobStoreImpl.deleteKey(FileBlobStoreImpl.java:178)
      at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:226)
      ... 19 more
      2016-09-14 15:07:48.588 o.a.s.util [ERROR] Halting process: ("Error on initialization")
      java.lang.RuntimeException: ("Error on initialization")
      at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
      at clojure.lang.RestFn.invoke(RestFn.java:423)
      at org.apache.storm.daemon.nimbus$fn_7064$service_handler_7308.doInvoke(nimbus.clj:1358)
      at clojure.lang.RestFn.invoke(RestFn.java:421)
      at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2206)
      at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2239)
      at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2262)
      at clojure.lang.AFn.applyToHelper(AFn.java:152)
      at clojure.lang.AFn.applyTo(AFn.java:144)
      at org.apache.storm.daemon.nimbus.main(Unknown Source)

      [root]# ls -l /opt/storm/storm-local/blobs/955/some_big_file
      total 591060
      rw-rr- 1 storm storm 605241344 Sep 14 15:07 1473865562841.tmp

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mwissman Matt Wissman
                Reporter:
                abid.m Abid Mohammed
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h