Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-925

Make it harder to accidentally close a shared DFSClient

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 0.21.0
    • Fix Version/s: None
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      Every so often I get stack traces telling me that DFSClient is closed, usually in {{org.apache.hadoop.hdfs.DFSClient.checkOpen() }} . The root cause of this is usually that one thread has closed a shared fsclient while another thread still has a reference to it. If the other thread then asks for a new client it will get one and the cache repopulated but if has one already, then I get to see a stack trace.

      It's effectively a race condition between clients in different threads.

      1. HADOOP-5933.patch
        2 kB
        steve_l
      2. HADOOP-5933.patch
        1 kB
        steve_l
      3. HDFS-925.patch
        5 kB
        Steve Loughran
      4. HDFS-925.patch
        4 kB
        Steve Loughran
      5. HDFS-925.patch
        2 kB
        Steve Loughran
      6. HDFS-925.patch
        2 kB
        steve_l

        Issue Links

          Activity

          steve_l created issue -
          steve_l made changes -
          Field Original Value New Value
          Attachment HADOOP-5933.patch [ 12409280 ]
          steve_l made changes -
          Attachment HADOOP-5933.patch [ 12431423 ]
          steve_l made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          steve_l made changes -
          Project Hadoop Common [ 12310240 ] Hadoop HDFS [ 12310942 ]
          Key HADOOP-5933 HDFS-925
          Affects Version/s 0.21.0 [ 12314046 ]
          Affects Version/s 0.21.0 [ 12313563 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Component/s hdfs client [ 12312928 ]
          Component/s fs [ 12310689 ]
          steve_l made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          steve_l made changes -
          Attachment HDFS-925.patch [ 12435708 ]
          steve_l made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jakob Homan made changes -
          Assignee Steve Loughran [ steve_l ]
          Gavin made changes -
          Reporter Steve Loughran [ steve_l ] Steve Loughran [ stevel@apache.org ]
          Gavin made changes -
          Assignee Steve Loughran [ steve_l ] Steve Loughran [ stevel@apache.org ]
          Nigel Daley made changes -
          Fix Version/s 0.22.0 [ 12314241 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HDFS-925.patch [ 12470242 ]
          Steve Loughran made changes -
          Attachment HDFS-925.patch [ 12470263 ]
          Steve Loughran made changes -
          Attachment HDFS-925.patch [ 12470272 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Link This issue is related to MAPREDUCE-437 [ MAPREDUCE-437 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development