Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-2437

Refactor HLog splitLog

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.0
    • Fix Version/s: 0.90.0
    • Component/s: master
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      the HLog.splitLog got really long and complex and hard to verify for correctness.
      I started to refactor it and also ported changes from hbase-2337 that deals with premature deletion of log files in case of errors. Further improvements will be possible, however the scope of this issue is to clean the code and make it behave correctly (i.e. not lose any edits)

      Added a suite of unit tests that might be ported to 0.20 as well.

      Added a setting (hbase.skip.errors - feel free to suggest a better name) that, when set to false will make the process less tolerant to failures or corrupted files: in case a log file is corrupted or an error stops the process from consistently splitting the log, will abort the entire operation to avoid losing any edits. When hbase.skip.errors is on any corrupted files will be partially parsed and then moved to the corrupted logs archive (see hbase-2337).

      Like hbase-2337 the splitLog method will first split all the logs and then proceed to archive them. If any splitted log file (oldlogfile.log) that is the result of an earlier splitLog attempt is found in the region directory, it will be deleted - this is safe since we won't move the original log files until the splitLog process completes.

        Attachments

        1. HBASE-2437-v5.patch
          65 kB
          Cosmin Lehene
        2. 2437-v4.patch
          49 kB
          Michael Stack
        3. 2437-v3.txt
          49 kB
          Michael Stack
        4. 2437-v2.txt
          61 kB
          Michael Stack
        5. 2437.txt
          49 kB
          Michael Stack
        6. HBASE-2437_for_HBase-0.21_with_unit_tests_for_HDFS-0.21.patch
          42 kB
          Cosmin Lehene

          Issue Links

            Activity

              People

              • Assignee:
                clehene Cosmin Lehene
                Reporter:
                clehene Cosmin Lehene
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 120h
                  120h
                  Remaining:
                  Remaining Estimate - 120h
                  120h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified