Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22200

WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.2.0, 2.0.6, 2.1.5
    • Component/s: wal
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir when checking for recovered.edits files, instead of taking FS instance as additional method parameter. When specifying different file systems for wal dir and root dir, WALSplitter.hasRecoveredEdits current implementation will crash or give wrong results. As of now, it's being used indirectly by SplitTableRegionProcedure. When running tests with WAL dir on HDFS and root dir on S3, for example, noticed region split failing with below error:

      2019-04-08 13:53:58,064 ERROR org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception: pid=98, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=test-tbl, parent=4c5db01611e97e3abbe02e781e867212, daughterA=28a0a5e4ef7618899f6bd6dfb5335fe7, daughterB=05fa26feaf03ebf9e87e099cbd1eabac
      java.lang.IllegalArgumentException: Path hdfs://host-1.example.com:8020/wal_dir/default/test-tbl/4c5db01611e97e3abbe02e781e867212/recovered.edits scheme must be s3a
              at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
              at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.checkPath(DynamoDBMetadataStore.java:1127)
              at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:437)
              at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2110)
              at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
              at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
              at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
              at org.apache.hadoop.hbase.wal.WALSplitter.getSplitEditFilesSorted(WALSplitter.java:576)
              at org.apache.hadoop.hbase.wal.WALSplitter.hasRecoveredEdits(WALSplitter.java:558)
              at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.hasRecoveredEdits(SplitTableRegionProcedure.java:148)
              at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:255)
      

      Since WALSplitter.hasRecoveredEdits already resolves the proper WAL dir for the region, we can simply re-use FS instance from the path instance for the WAL dir region, when searching for recovered.edits.

        Attachments

        1. HBASE-22200-master-002.patch
          7 kB
          Wellington Chevreuil
        2. HBASE-22200-master-001.patch
          6 kB
          Wellington Chevreuil
        3. HBASE-22200-branch-2.1-003.patch
          7 kB
          Wellington Chevreuil
        4. HBASE-22200-branch-2.1-002.patch
          7 kB
          Wellington Chevreuil
        5. HBASE-22200-branch-2.1-001.patch
          6 kB
          Wellington Chevreuil

          Issue Links

            Activity

              People

              • Assignee:
                wchevreuil Wellington Chevreuil
                Reporter:
                wchevreuil Wellington Chevreuil
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: