Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22617

Recovered WAL directories not getting cleaned up

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.3.3, 2.2.0, 1.4.8, 2.1.1, 1.4.9, 2.1.2, 1.4.10, 2.1.3, 1.3.4, 2.1.4, 2.1.5, 1.3.5
    • Fix Version/s: 3.0.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6
    • Component/s: wal
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      In HBASE-20734 we moved the recovered.edits onto the wal file system but when constructing the directory we missed the BASE_NAMESPACE_DIR('data'). So when using the default config, you will find that there are lots of new directories at the same level with the 'data' directory.

      In this issue, we add the BASE_NAMESPACE_DIR back, and also try our best to clean up the wrong directories. But we can only clean up the region level directories, so if you want a clean fs layout on HDFS you still need to manually delete the empty directories at the same level with 'data'.

      The effect versions are 2.2.0, 2.1.[1-5], 1.4.[8-10], 1.3.[3-5].
      Show
      In HBASE-20734 we moved the recovered.edits onto the wal file system but when constructing the directory we missed the BASE_NAMESPACE_DIR('data'). So when using the default config, you will find that there are lots of new directories at the same level with the 'data' directory. In this issue, we add the BASE_NAMESPACE_DIR back, and also try our best to clean up the wrong directories. But we can only clean up the region level directories, so if you want a clean fs layout on HDFS you still need to manually delete the empty directories at the same level with 'data'. The effect versions are 2.2.0, 2.1.[1-5], 1.4.[8-10], 1.3.[3-5].

      Description

      While colocating the recovered edits directory with hbase.wal.dir, BASE_NAMESPACE_DIR got missed. This results in recovered edits being put in a separate directory rather than the default region directory even if the hbase.wal.dir is not overridden. Eg. if data is stored in /hbase/data/namespace/table1, recovered edits are put in  /hbase/namespace/table1. This also messes up the regular cleaner chores which never operate on this new directory and these directories will never be deleted, even for split parents or dropped tables. We should change the default back to have the base namespace directory in path.

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                zhangduo Duo Zhang
                Reporter:
                abhishek.chouhan Abhishek Singh Chouhan
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: