Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-142

In 0.20, move blocks being written into a blocksBeingWritten directory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20-append
    • Fix Version/s: 0.20-append, 0.20.205.0
    • Component/s: None
    • Labels:
      None
    • Tags:
      hbase

      Description

      Before 0.18, when Datanode restarts, it deletes files under data-dir/tmp directory since these files are not valid anymore. But in 0.18 it moves these files to normal directory incorrectly making them valid blocks. One of the following would work :

      • remove the tmp files during upgrade, or
      • if the files under /tmp are in pre-18 format (i.e. no generation), delete them.

      Currently effect of this bug is that, these files end up failing block verification and eventually get deleted. But cause incorrect over-replication at the namenode before that.

      Also it looks like our policy regd treating files under tmp needs to be defined better. Right now there are probably one or two more bugs with it. Dhruba, please file them if you rememeber.

      1. appendFile-recheck-lease.txt
        6 kB
        Todd Lipcon
      2. appendQuestions.txt
        8 kB
        dhruba borthakur
      3. deleteTmp_0.18.patch
        2 kB
        dhruba borthakur
      4. deleteTmp.patch
        6 kB
        dhruba borthakur
      5. deleteTmp2.patch
        13 kB
        dhruba borthakur
      6. deleteTmp5_20.txt
        32 kB
        dhruba borthakur
      7. deleteTmp5_20.txt
        35 kB
        dhruba borthakur
      8. dont-recover-rwr-when-rbw-available.txt
        24 kB
        Todd Lipcon
      9. handleTmp1.patch
        9 kB
        dhruba borthakur
      10. HDFS-142_20.patch
        54 kB
        Nicolas Spiegelberg
      11. HDFS-142_20-append2.patch
        84 kB
        Nicolas Spiegelberg
      12. HDFS-142.20-security.1.patch
        88 kB
        Jitendra Nath Pandey
      13. HDFS-142.20-security.2.patch
        89 kB
        Jitendra Nath Pandey
      14. hdfs-142-commitBlockSynchronization-unknown-datanode.txt
        2 kB
        Todd Lipcon
      15. HDFS-142-deaddn-fix.patch
        3 kB
        Nicolas Spiegelberg
      16. HDFS-142-finalize-fix.txt
        7 kB
        sam rash
      17. hdfs-142-minidfs-fix-from-409.txt
        2 kB
        Todd Lipcon
      18. HDFS-142-multiple-blocks-datanode-exception.patch
        3 kB
        Karthik Ranganathan
      19. hdfs-142-recovery-reassignment-and-bbw-cleanup.txt
        19 kB
        Todd Lipcon
      20. hdfs-142-testcases.txt
        8 kB
        Todd Lipcon
      21. hdfs-142-testleaserecovery-fix.txt
        5 kB
        Todd Lipcon
      22. recentInvalidateSets-assertion-fix.txt
        1 kB
        Todd Lipcon
      23. recover-rbw-v2.txt
        27 kB
        Todd Lipcon
      24. testfileappend4-deaddn.txt
        2 kB
        Todd Lipcon
      25. validateBlockMetaData-synchronized.txt
        0.9 kB
        Todd Lipcon

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development