Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7253

getBlockLocationsUpdateTimes missing handle exception may cause fsLock dead lock

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 2.2.0
    • None
    • namenode
    • None

    Description

      One day my active namenode hanged and I dumped the program stacks by jstack.In the stacks file, I saw most threads were waiting FSNamesystem.fsLock, both readLock and writeLock were unacquirable, but no thread was holding writeLock.
      I tried to access the web interface of this namenode but was blocked. and I tried to failover the active node to another namenode manually (zkfs did not discover this node was hanging) but it was also failed. So I killed this namenode trying to recover the production environment, then the failover was triggered, standby nn transited to active, and then, the new active namenode hanged.
      My following steps are useless and can be ignored. At last, I thought it was caused by an incorrect lock handling in FSNamesystem.getBlockLocationsUpdateTimes, which I will describe in the first comment.

      Attachments

        1. nn1013.jstack
          260 kB
          Carrey Zhan
        2. Tester.java
          2 kB
          Carrey Zhan
        3. Trigger.tgz
          3 kB
          Carrey Zhan

        Issue Links

          Activity

            People

              Unassigned Unassigned
              CarreyZhan Carrey Zhan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: