Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22617

Recovered WAL directories not getting cleaned up

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.3.3, 2.2.0, 1.4.8, 2.1.1, 1.4.9, 2.1.2, 1.4.10, 2.1.3, 1.3.4, 2.1.4, 2.1.5, 1.3.5
    • 3.0.0-alpha-1, 2.3.0, 2.0.6, 2.2.1, 2.1.6
    • wal
    • None
    • Reviewed
    • Hide
      In HBASE-20734 we moved the recovered.edits onto the wal file system but when constructing the directory we missed the BASE_NAMESPACE_DIR('data'). So when using the default config, you will find that there are lots of new directories at the same level with the 'data' directory.

      In this issue, we add the BASE_NAMESPACE_DIR back, and also try our best to clean up the wrong directories. But we can only clean up the region level directories, so if you want a clean fs layout on HDFS you still need to manually delete the empty directories at the same level with 'data'.

      The effect versions are 2.2.0, 2.1.[1-5], 1.4.[8-10], 1.3.[3-5].
      Show
      In HBASE-20734 we moved the recovered.edits onto the wal file system but when constructing the directory we missed the BASE_NAMESPACE_DIR('data'). So when using the default config, you will find that there are lots of new directories at the same level with the 'data' directory. In this issue, we add the BASE_NAMESPACE_DIR back, and also try our best to clean up the wrong directories. But we can only clean up the region level directories, so if you want a clean fs layout on HDFS you still need to manually delete the empty directories at the same level with 'data'. The effect versions are 2.2.0, 2.1.[1-5], 1.4.[8-10], 1.3.[3-5].

    Description

      While colocating the recovered edits directory with hbase.wal.dir, BASE_NAMESPACE_DIR got missed. This results in recovered edits being put in a separate directory rather than the default region directory even if the hbase.wal.dir is not overridden. Eg. if data is stored in /hbase/data/namespace/table1, recovered edits are put in  /hbase/namespace/table1. This also messes up the regular cleaner chores which never operate on this new directory and these directories will never be deleted, even for split parents or dropped tables. We should change the default back to have the base namespace directory in path.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HBASE-21598 HBASE_WAL_DIR if not configured, recovered.edits directory's are sidelined from the table dir path.

          • Major - Major loss of function.
          • Closed
          is broken by

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-20734 Colocate recovered edits directory with hbase.wal.dir

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #330

          Activity

            People

              zhangduo Duo Zhang
              abhishek.chouhan Abhishek Singh Chouhan
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: