Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12757

DeadLock Happened Between DFSOutputStream and LeaseRenewer when LeaseRenewer#renew SocketTimeException

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      Java stack is :

      Found one Java-level deadlock:
      =============================
      "Topology-2 (735/2000)":
        waiting to lock monitor 0x00007fff4523e6e8 (object 0x00000005d3521078, a org.apache.hadoop.hdfs.client.impl.LeaseRenewer),
        which is held by "LeaseRenewer:admin@na61storage"
      "LeaseRenewer:admin@na61storage":
        waiting to lock monitor 0x00007fff5d41e838 (object 0x00000005ec0dfa88, a org.apache.hadoop.hdfs.DFSOutputStream),
        which is held by "Topology-2 (735/2000)"
      
      Java stack information for the threads listed above:
      ===================================================
      "Topology-2 (735/2000)":
              at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.addClient(LeaseRenewer.java:227)
              - waiting to lock <0x00000005d3521078> (a org.apache.hadoop.hdfs.client.impl.LeaseRenewer)
              at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.getInstance(LeaseRenewer.java:86)
              at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:467)
              at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:479)
              at org.apache.hadoop.hdfs.DFSOutputStream.setClosed(DFSOutputStream.java:776)
              at org.apache.hadoop.hdfs.DFSOutputStream.closeThreads(DFSOutputStream.java:791)
              at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:848)
              - locked <0x00000005ec0dfa88> (a org.apache.hadoop.hdfs.DFSOutputStream)
              at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:805)
              - locked <0x00000005ec0dfa88> (a org.apache.hadoop.hdfs.DFSOutputStream)
              at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
              at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
      ......
      "LeaseRenewer:admin@na61storage":
              at org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:750)
              - waiting to lock <0x00000005ec0dfa88> (a org.apache.hadoop.hdfs.DFSOutputStream)
              at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:586)
              at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:453)
              - locked <0x00000005d3521078> (a org.apache.hadoop.hdfs.client.impl.LeaseRenewer)
              at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$700(LeaseRenewer.java:76)
              at org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:310)
              at java.lang.Thread.run(Thread.java:834)
      
      Found 1 deadlock.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                yangjiandan Jiandan Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: