HBase
  1. HBase
  2. HBASE-4510

Check and workaround usage of internal HDFS APIs in HBase

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.92.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

      Two solutions:

      • We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)
      • HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          I think option 2 is more viable.
          We should keep backward compatibility.

          Show
          Ted Yu added a comment - I think option 2 is more viable. We should keep backward compatibility.
          Hide
          stack added a comment -

          Ugh. Thanks Harsh for opening this issue. Would be nice if 0.92 was working w/ current state of 0.23 branch at time of release.

          Show
          stack added a comment - Ugh. Thanks Harsh for opening this issue. Would be nice if 0.92 was working w/ current state of 0.23 branch at time of release.
          Hide
          Harsh J added a comment -

          Since this may require a lot of review+discussion, I've opened https://reviews.apache.org/r/2108/

          Show
          Harsh J added a comment - Since this may require a lot of review+discussion, I've opened https://reviews.apache.org/r/2108/
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)
          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.
          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs


          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937

          Diff: https://reviews.apache.org/r/2108/diff

          Testing
          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- Review request for hbase. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          Ted Yu added a comment -

          I think the above diff is in the right direction.

          Show
          Ted Yu added a comment - I think the above diff is in the right direction.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/
          -----------------------------------------------------------

          (Updated 2011-10-18 03:42:06.955026)

          Review request for hbase.

          Changes
          -------

          Updated patch that should fix 0.23 builds.

          Summary
          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)
          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.
          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937
          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b

          Diff: https://reviews.apache.org/r/2108/diff

          Testing
          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- (Updated 2011-10-18 03:42:06.955026) Review request for hbase. Changes ------- Updated patch that should fix 0.23 builds. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs (updated) src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/#review2641
          -----------------------------------------------------------

          Ship it!

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          <https://reviews.apache.org/r/2108/#comment5946>

          Typo: retrieved

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          <https://reviews.apache.org/r/2108/#comment5947>

          This condition is covered by isInSafeMode() already.

          • Ted

          On 2011-10-18 03:42:06, Harsh J wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2108/

          -----------------------------------------------------------

          (Updated 2011-10-18 03:42:06)

          Review request for hbase.

          Summary

          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)

          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.

          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b

          Diff: https://reviews.apache.org/r/2108/diff

          Testing

          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/#review2641 ----------------------------------------------------------- Ship it! src/main/java/org/apache/hadoop/hbase/util/FSUtils.java < https://reviews.apache.org/r/2108/#comment5946 > Typo: retrieved src/main/java/org/apache/hadoop/hbase/util/FSUtils.java < https://reviews.apache.org/r/2108/#comment5947 > This condition is covered by isInSafeMode() already. Ted On 2011-10-18 03:42:06, Harsh J wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- (Updated 2011-10-18 03:42:06) Review request for hbase. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs ----- src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/
          -----------------------------------------------------------

          (Updated 2011-10-18 04:01:36.605193)

          Review request for hbase.

          Changes
          -------

          Fix Ted's comments.

          Commenting further on JIRA itself, since this isn't as urgent as before.

          Summary
          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)
          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.
          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937
          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b

          Diff: https://reviews.apache.org/r/2108/diff

          Testing
          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- (Updated 2011-10-18 04:01:36.605193) Review request for hbase. Changes ------- Fix Ted's comments. Commenting further on JIRA itself, since this isn't as urgent as before. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs (updated) src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          Harsh J added a comment -

          Unfortunately my vacation's made me go way off-sync surrounding this issue.

          Todd has fixed build for 0.23 via HDFS-2412 already, and that patch would be available for 0.23 at least (graceful deprecation being the plan).

          We're proposing adding a public 'check' of SafeMode status in HDFS upstream, via HDFS-2413, which HBASE and other projects can then rely on.

          Changes made to ticket:

          • Changed title to reflect new goal.
          • Changed the severity down from blocker as it isn't anymore, after HDFS-2412 went in.
          Show
          Harsh J added a comment - Unfortunately my vacation's made me go way off-sync surrounding this issue. Todd has fixed build for 0.23 via HDFS-2412 already, and that patch would be available for 0.23 at least (graceful deprecation being the plan). We're proposing adding a public 'check' of SafeMode status in HDFS upstream, via HDFS-2413 , which HBASE and other projects can then rely on. Changes made to ticket: Changed title to reflect new goal. Changed the severity down from blocker as it isn't anymore, after HDFS-2412 went in.
          Hide
          Ted Yu added a comment -

          +1 on latest patch.

          Show
          Ted Yu added a comment - +1 on latest patch.
          Hide
          Harsh J added a comment -

          Attaching the +1'd patch.

          Show
          Harsh J added a comment - Attaching the +1'd patch.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/#review2717
          -----------------------------------------------------------

          Ship it!

          • Michael

          On 2011-10-18 04:01:36, Harsh J wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2108/

          -----------------------------------------------------------

          (Updated 2011-10-18 04:01:36)

          Review request for hbase.

          Summary

          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)

          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.

          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b

          Diff: https://reviews.apache.org/r/2108/diff

          Testing

          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/#review2717 ----------------------------------------------------------- Ship it! Michael On 2011-10-18 04:01:36, Harsh J wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- (Updated 2011-10-18 04:01:36) Review request for hbase. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs ----- src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          stack added a comment -

          Applied to 0.92 branch and TRUNK. Thanks Harsh.

          Show
          stack added a comment - Applied to 0.92 branch and TRUNK. Thanks Harsh.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2346 (See https://builds.apache.org/job/HBase-TRUNK/2346/)
          HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase

          stack :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2346 (See https://builds.apache.org/job/HBase-TRUNK/2346/ ) HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase stack : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #75 (See https://builds.apache.org/job/HBase-0.92/75/)
          HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase

          stack :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #75 (See https://builds.apache.org/job/HBase-0.92/75/ ) HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #96 (See https://builds.apache.org/job/HBase-0.92/96/)
          HBASE-4708 Revert safemode related pieces of hbase-4510

          Show
          Hudson added a comment - Integrated in HBase-0.92 #96 (See https://builds.apache.org/job/HBase-0.92/96/ ) HBASE-4708 Revert safemode related pieces of hbase-4510
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2398 (See https://builds.apache.org/job/HBase-TRUNK/2398/)
          HBASE-4708 Revert safemode related pieces of hbase-4510

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2398 (See https://builds.apache.org/job/HBase-TRUNK/2398/ ) HBASE-4708 Revert safemode related pieces of hbase-4510

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development