Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1623 High Availability Framework for HDFS NN
  3. HDFS-1108

Log newly allocated blocks

        Details

        • Target Version/s:

          Description

          The current HDFS design says that newly allocated blocks for a file are not persisted in the NN transaction log when the block is allocated. Instead, a hflush() or a close() on the file persists the blocks into the transaction log. It would be nice if we can immediately persist newly allocated blocks (as soon as they are allocated) for specific files.

          1. Text File
            HDFS-1108.patch
            6 kB
            Dmytro Molkov
          2. Text File
            hdfs-1108.txt
            10 kB
            Todd Lipcon
          3. Text File
            hdfs-1108-habranch.txt
            11 kB
            Konstantin Shvachko
          4. Text File
            hdfs-1108-habranch.txt
            11 kB
            Aaron T. Myers
          5. Text File
            hdfs-1108-habranch.txt
            10 kB
            Todd Lipcon
          6. Text File
            hdfs-1108-habranch.txt
            10 kB
            Eli Collins
          7. Text File
            hdfs-1108-habranch.txt
            9 kB
            Todd Lipcon
          8. Text File
            hdfs-1108-hadoop-1.patch
            12 kB
            Sanjay Radia
          9. Text File
            hdfs-1108-hadoop-1-v2.patch
            16 kB
            Sanjay Radia
          10. Text File
            hdfs-1108-hadoop-1-v3.patch
            17 kB
            Sanjay Radia
          11. Text File
            hdfs-1108-hadoop-1-v4.patch
            17 kB
            Sanjay Radia
          12. Text File
            hdfs-1108-hadoop-1-v5.patch
            17 kB
            Sanjay Radia

            Issue Links

            blocks

            Sub-task - The sub-task of the issue HBASE-2108 [HA] hbase cluster should be able to ride over hdfs 'safe mode' flip and namenode restart/move

            • Major - Major loss of function.
            • Resolved

            New Feature - A new feature of the product, which has yet to be developed. HDFS-2064 Warm HA NameNode going Hot

            • Major - Major loss of function.
            • Open
            is duplicated by

            Improvement - An improvement or enhancement to an existing feature or task. HDFS-978 Record every new block allocation of a file into the transaction log.

            • Major - Major loss of function.
            • Closed
            is related to

            Sub-task - The sub-task of the issue HDFS-1975 HA: Support for sharing the namenode state from active to standby.

            • Major - Major loss of function.
            • Resolved
            is superceded by

            Sub-task - The sub-task of the issue HDFS-2602 NN should log newly-allocated blocks without losing BlockInfo

            • Critical - Crashes, loss of data, severe memory leak.
            • Resolved
            relates to

            Sub-task - The sub-task of the issue HDFS-2602 NN should log newly-allocated blocks without losing BlockInfo

            • Critical - Crashes, loss of data, severe memory leak.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HDFS-3630 Modify TestPersistBlocks to use both flush and hflush

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              dhruba borthakur created issue -
              dhruba borthakur made changes -
              Field Original Value New Value
              Link This issue blocks HBASE-2108 [ HBASE-2108 ]
              dhruba borthakur made changes -
              Assignee dhruba borthakur [ dhruba ] Dmytro Molkov [ dms ]
              Dmytro Molkov made changes -
              Attachment HDFS-1108.patch [ 12447758 ]
              Dmytro Molkov made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              dhruba borthakur made changes -
              Status Patch Available [ 10002 ] Open [ 1 ]
              Todd Lipcon made changes -
              Link This issue is related to HDFS-978 [ HDFS-978 ]
              Todd Lipcon made changes -
              Attachment hdfs-1108-habranch.txt [ 12490196 ]
              Todd Lipcon made changes -
              Parent HDFS-1623 [ 12498318 ]
              Issue Type Improvement [ 4 ] Sub-task [ 7 ]
              Todd Lipcon made changes -
              Assignee Dmytro Molkov [ dms ] Todd Lipcon [ tlipcon ]
              Eli Collins made changes -
              Summary ability to create a file whose newly allocated blocks are automatically persisted immediately Log newly allocated blocks
              Fix Version/s HA branch (HDFS-1623) [ 12317568 ]
              Todd Lipcon made changes -
              Attachment hdfs-1108.txt [ 12490750 ]
              Todd Lipcon made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Eli Collins made changes -
              Attachment hdfs-1108-habranch.txt [ 12501211 ]
              Jeff Hammerbacher made changes -
              Link This issue is related to HDFS-1975 [ HDFS-1975 ]
              Todd Lipcon made changes -
              Attachment hdfs-1108-habranch.txt [ 12505289 ]
              Todd Lipcon made changes -
              Link This issue relates to HDFS-2602 [ HDFS-2602 ]
              Aaron T. Myers made changes -
              Attachment hdfs-1108-habranch.txt [ 12505835 ]
              Konstantin Shvachko made changes -
              Attachment hdfs-1108-habranch.txt [ 12506412 ]
              Konstantin Shvachko made changes -
              Status Patch Available [ 10002 ] Open [ 1 ]
              Todd Lipcon made changes -
              Status Open [ 1 ] Resolved [ 5 ]
              Resolution Duplicate [ 3 ]
              Todd Lipcon made changes -
              Link This issue is superceded by HDFS-2602 [ HDFS-2602 ]
              Aaron T. Myers made changes -
              Component/s ha [ 12316609 ]
              Sanjay Radia made changes -
              Attachment hdfs-1108-hadoop-1.patch [ 12533928 ]
              Sanjay Radia made changes -
              Attachment hdfs-1108-hadoop-1-v2.patch [ 12534030 ]
              Sanjay Radia made changes -
              Attachment hdfs-1108-hadoop-1-v3.patch [ 12535929 ]
              Sanjay Radia made changes -
              Link This issue relates to HDFS-3630 [ HDFS-3630 ]
              Sanjay Radia made changes -
              Attachment hdfs-1108-hadoop-1-v4.patch [ 12536661 ]
              Sanjay Radia made changes -
              Attachment hdfs-1108-hadoop-1-v5.patch [ 12536726 ]
              Sanjay Radia made changes -
              Fix Version/s 1.1.0 [ 12317959 ]
              Target Version/s 1.1.0 [ 12317959 ]
              Matt Foley made changes -
              Link This issue is duplicated by HDFS-978 [ HDFS-978 ]
              Matt Foley made changes -
              Link This issue is related to HDFS-978 [ HDFS-978 ]
              Matt Foley made changes -
              Link This issue blocks HDFS-2064 [ HDFS-2064 ]
              Matt Foley made changes -
              Resolution Duplicate [ 3 ]
              Status Resolved [ 5 ] Reopened [ 4 ]
              Matt Foley made changes -
              Status Reopened [ 4 ] Resolved [ 5 ]
              Resolution Fixed [ 1 ]
              Suresh Srinivas made changes -
              Fix Version/s 1.2.0 [ 12321657 ]
              Fix Version/s 1.1.0 [ 12317959 ]
              Matt Foley made changes -
              Target Version/s 1.1.0 [ 12317959 ] 1.2.0 [ 12321657 ]
              Matt Foley made changes -
              Fix Version/s 1.1.1 [ 12321656 ]
              Fix Version/s 1.2.0 [ 12321657 ]
              Matt Foley made changes -
              Target Version/s 1.2.0 [ 12321657 ] 1.1.1 [ 12321656 ]
              Matt Foley made changes -
              Status Resolved [ 5 ] Closed [ 6 ]

                People

                • Assignee:
                  Todd Lipcon
                  Reporter:
                  dhruba borthakur
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  18 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development