Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14247

Separate the old WALs into different regionserver directories

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha-4, 2.0.0
    • Component/s: wal
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      Add a new config hbase.separate.oldlogdir.by.regionserver. The default value is false. If this config is true, the old wal dir will be separated by regionservers. This will change the oldWALs layout. The oldWALs is used by replication. So if a cluster didn't use replication, it can be rolling upgrade (upgrade this config from false to true) directly. If a cluster use replication, the oldWALs will be not found when layout changed. So the cluster need rolling upgrade twice. Firstly, only rolling cluster to use new version code. Secondly rolling the config from false to true. Because the cluster already rolling to new version code, so it can find the oldWALs in the new dir layout.
      Show
      Add a new config hbase.separate.oldlogdir.by.regionserver. The default value is false. If this config is true, the old wal dir will be separated by regionservers. This will change the oldWALs layout. The oldWALs is used by replication. So if a cluster didn't use replication, it can be rolling upgrade (upgrade this config from false to true) directly. If a cluster use replication, the oldWALs will be not found when layout changed. So the cluster need rolling upgrade twice. Firstly, only rolling cluster to use new version code. Secondly rolling the config from false to true. Because the cluster already rolling to new version code, so it can find the oldWALs in the new dir layout.

      Description

      Currently all old WALs of regionservers are achieved into the single directory of oldWALs. In big clusters, because of long TTL of WAL or disabled replications, the number of files under oldWALs may reach the max-directory-items limit of HDFS, which will make the hbase cluster crashed.

      Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded: limit=1048576 items=1048576

      A simple solution is to separate the old WALs into different directories according to the server name of the WAL.

      Suggestions are welcomed~ Thanks

        Attachments

        1. HBASE-14247.master.005.patch
          49 kB
          Guanghao Zhang
        2. HBASE-14247.master.004.patch
          48 kB
          Guanghao Zhang
        3. HBASE-14247.master.003.patch
          47 kB
          Guanghao Zhang
        4. HBASE-14247.master.002.patch
          42 kB
          Guanghao Zhang
        5. HBASE-14247.master.001.patch
          42 kB
          Guanghao Zhang
        6. HBASE-14247-v003.diff
          38 kB
          Shaohui Liu
        7. HBASE-14247-v002.diff
          37 kB
          Shaohui Liu
        8. HBASE-14247-v001.diff
          35 kB
          Shaohui Liu

          Issue Links

            Activity

              People

              • Assignee:
                zghao Guanghao Zhang
                Reporter:
                liushaohui Shaohui Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: