Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-142

In 0.20, move blocks being written into a blocksBeingWritten directory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20-append
    • Fix Version/s: 0.20-append, 0.20.205.0
    • Component/s: None
    • Labels:
      None
    • Tags:
      hbase

      Description

      Before 0.18, when Datanode restarts, it deletes files under data-dir/tmp directory since these files are not valid anymore. But in 0.18 it moves these files to normal directory incorrectly making them valid blocks. One of the following would work :

      • remove the tmp files during upgrade, or
      • if the files under /tmp are in pre-18 format (i.e. no generation), delete them.

      Currently effect of this bug is that, these files end up failing block verification and eventually get deleted. But cause incorrect over-replication at the namenode before that.

      Also it looks like our policy regd treating files under tmp needs to be defined better. Right now there are probably one or two more bugs with it. Dhruba, please file them if you rememeber.

      1. deleteTmp.patch
        6 kB
        dhruba borthakur
      2. deleteTmp2.patch
        13 kB
        dhruba borthakur
      3. deleteTmp_0.18.patch
        2 kB
        dhruba borthakur
      4. handleTmp1.patch
        9 kB
        dhruba borthakur
      5. appendQuestions.txt
        8 kB
        dhruba borthakur
      6. deleteTmp5_20.txt
        35 kB
        dhruba borthakur
      7. deleteTmp5_20.txt
        32 kB
        dhruba borthakur
      8. HDFS-142_20.patch
        54 kB
        Nicolas Spiegelberg
      9. hdfs-142-minidfs-fix-from-409.txt
        2 kB
        Todd Lipcon
      10. HDFS-142-multiple-blocks-datanode-exception.patch
        3 kB
        Karthik Ranganathan
      11. testfileappend4-deaddn.txt
        2 kB
        Todd Lipcon
      12. HDFS-142-deaddn-fix.patch
        3 kB
        Nicolas Spiegelberg
      13. HDFS-142-finalize-fix.txt
        7 kB
        sam rash
      14. hdfs-142-commitBlockSynchronization-unknown-datanode.txt
        2 kB
        Todd Lipcon
      15. hdfs-142-testcases.txt
        8 kB
        Todd Lipcon
      16. hdfs-142-recovery-reassignment-and-bbw-cleanup.txt
        19 kB
        Todd Lipcon
      17. hdfs-142-testleaserecovery-fix.txt
        5 kB
        Todd Lipcon
      18. validateBlockMetaData-synchronized.txt
        0.9 kB
        Todd Lipcon
      19. recentInvalidateSets-assertion-fix.txt
        1 kB
        Todd Lipcon
      20. appendFile-recheck-lease.txt
        6 kB
        Todd Lipcon
      21. dont-recover-rwr-when-rbw-available.txt
        24 kB
        Todd Lipcon
      22. recover-rbw-v2.txt
        27 kB
        Todd Lipcon
      23. HDFS-142_20-append2.patch
        84 kB
        Nicolas Spiegelberg
      24. HDFS-142.20-security.1.patch
        88 kB
        Jitendra Nath Pandey
      25. HDFS-142.20-security.2.patch
        89 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development