HBase
  1. HBase
  2. HBASE-4510

Check and workaround usage of internal HDFS APIs in HBase

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.92.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

      Two solutions:

      • We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)
      • HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

        Issue Links

          Activity

          Harsh J created issue -
          Harsh J made changes -
          Field Original Value New Value
          Assignee Harsh J [ qwertymaniac ]
          Hide
          Ted Yu added a comment -

          I think option 2 is more viable.
          We should keep backward compatibility.

          Show
          Ted Yu added a comment - I think option 2 is more viable. We should keep backward compatibility.
          Hide
          stack added a comment -

          Ugh. Thanks Harsh for opening this issue. Would be nice if 0.92 was working w/ current state of 0.23 branch at time of release.

          Show
          stack added a comment - Ugh. Thanks Harsh for opening this issue. Would be nice if 0.92 was working w/ current state of 0.23 branch at time of release.
          Hide
          Harsh J added a comment -

          Since this may require a lot of review+discussion, I've opened https://reviews.apache.org/r/2108/

          Show
          Harsh J added a comment - Since this may require a lot of review+discussion, I've opened https://reviews.apache.org/r/2108/
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)
          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.
          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs


          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937

          Diff: https://reviews.apache.org/r/2108/diff

          Testing
          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- Review request for hbase. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          Ted Yu added a comment -

          I think the above diff is in the right direction.

          Show
          Ted Yu added a comment - I think the above diff is in the right direction.
          Ted Yu made changes -
          Link This issue blocks HBASE-4254 [ HBASE-4254 ]
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/
          -----------------------------------------------------------

          (Updated 2011-10-18 03:42:06.955026)

          Review request for hbase.

          Changes
          -------

          Updated patch that should fix 0.23 builds.

          Summary
          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)
          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.
          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937
          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b

          Diff: https://reviews.apache.org/r/2108/diff

          Testing
          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- (Updated 2011-10-18 03:42:06.955026) Review request for hbase. Changes ------- Updated patch that should fix 0.23 builds. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs (updated) src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Harsh J made changes -
          Link This issue requires HDFS-2466 [ HDFS-2466 ]
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/#review2641
          -----------------------------------------------------------

          Ship it!

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          <https://reviews.apache.org/r/2108/#comment5946>

          Typo: retrieved

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          <https://reviews.apache.org/r/2108/#comment5947>

          This condition is covered by isInSafeMode() already.

          • Ted

          On 2011-10-18 03:42:06, Harsh J wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2108/

          -----------------------------------------------------------

          (Updated 2011-10-18 03:42:06)

          Review request for hbase.

          Summary

          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)

          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.

          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b

          Diff: https://reviews.apache.org/r/2108/diff

          Testing

          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/#review2641 ----------------------------------------------------------- Ship it! src/main/java/org/apache/hadoop/hbase/util/FSUtils.java < https://reviews.apache.org/r/2108/#comment5946 > Typo: retrieved src/main/java/org/apache/hadoop/hbase/util/FSUtils.java < https://reviews.apache.org/r/2108/#comment5947 > This condition is covered by isInSafeMode() already. Ted On 2011-10-18 03:42:06, Harsh J wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- (Updated 2011-10-18 03:42:06) Review request for hbase. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs ----- src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/
          -----------------------------------------------------------

          (Updated 2011-10-18 04:01:36.605193)

          Review request for hbase.

          Changes
          -------

          Fix Ted's comments.

          Commenting further on JIRA itself, since this isn't as urgent as before.

          Summary
          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)
          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.
          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937
          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b

          Diff: https://reviews.apache.org/r/2108/diff

          Testing
          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- (Updated 2011-10-18 04:01:36.605193) Review request for hbase. Changes ------- Fix Ted's comments. Commenting further on JIRA itself, since this isn't as urgent as before. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs (updated) src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          Harsh J added a comment -

          Unfortunately my vacation's made me go way off-sync surrounding this issue.

          Todd has fixed build for 0.23 via HDFS-2412 already, and that patch would be available for 0.23 at least (graceful deprecation being the plan).

          We're proposing adding a public 'check' of SafeMode status in HDFS upstream, via HDFS-2413, which HBASE and other projects can then rely on.

          Changes made to ticket:

          • Changed title to reflect new goal.
          • Changed the severity down from blocker as it isn't anymore, after HDFS-2412 went in.
          Show
          Harsh J added a comment - Unfortunately my vacation's made me go way off-sync surrounding this issue. Todd has fixed build for 0.23 via HDFS-2412 already, and that patch would be available for 0.23 at least (graceful deprecation being the plan). We're proposing adding a public 'check' of SafeMode status in HDFS upstream, via HDFS-2413 , which HBASE and other projects can then rely on. Changes made to ticket: Changed title to reflect new goal. Changed the severity down from blocker as it isn't anymore, after HDFS-2412 went in.
          Harsh J made changes -
          Summary HDFS-1620 related changes downstream (For compiling with HDFS 0.23+) Check and workaround usage of internal HDFS APIs in HBase
          Priority Blocker [ 1 ] Major [ 3 ]
          Hide
          Ted Yu added a comment -

          +1 on latest patch.

          Show
          Ted Yu added a comment - +1 on latest patch.
          Hide
          Harsh J added a comment -

          Attaching the +1'd patch.

          Show
          Harsh J added a comment - Attaching the +1'd patch.
          Harsh J made changes -
          Attachment HBASE-4510.patch [ 12499509 ]
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2108/#review2717
          -----------------------------------------------------------

          Ship it!

          • Michael

          On 2011-10-18 04:01:36, Harsh J wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2108/

          -----------------------------------------------------------

          (Updated 2011-10-18 04:01:36)

          Review request for hbase.

          Summary

          -------

          HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out.

          Two solutions:

          1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?)

          2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side.

          Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes.

          The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts?

          This addresses bug HBASE-4510.

          https://issues.apache.org/jira/browse/HBASE-4510

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b

          Diff: https://reviews.apache.org/r/2108/diff

          Testing

          -------

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/#review2717 ----------------------------------------------------------- Ship it! Michael On 2011-10-18 04:01:36, Harsh J wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ ----------------------------------------------------------- (Updated 2011-10-18 04:01:36) Review request for hbase. Summary ------- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510 . https://issues.apache.org/jira/browse/HBASE-4510 Diffs ----- src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 789dd3b Diff: https://reviews.apache.org/r/2108/diff Testing ------- Thanks, Harsh
          Hide
          stack added a comment -

          Applied to 0.92 branch and TRUNK. Thanks Harsh.

          Show
          stack added a comment - Applied to 0.92 branch and TRUNK. Thanks Harsh.
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.92.0 [ 12314223 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2346 (See https://builds.apache.org/job/HBase-TRUNK/2346/)
          HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase

          stack :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2346 (See https://builds.apache.org/job/HBase-TRUNK/2346/ ) HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase stack : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #75 (See https://builds.apache.org/job/HBase-0.92/75/)
          HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase

          stack :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #75 (See https://builds.apache.org/job/HBase-0.92/75/ ) HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #96 (See https://builds.apache.org/job/HBase-0.92/96/)
          HBASE-4708 Revert safemode related pieces of hbase-4510

          Show
          Hudson added a comment - Integrated in HBase-0.92 #96 (See https://builds.apache.org/job/HBase-0.92/96/ ) HBASE-4708 Revert safemode related pieces of hbase-4510
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2398 (See https://builds.apache.org/job/HBase-TRUNK/2398/)
          HBASE-4708 Revert safemode related pieces of hbase-4510

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2398 (See https://builds.apache.org/job/HBase-TRUNK/2398/ ) HBASE-4708 Revert safemode related pieces of hbase-4510
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          21d 13h 43m 1 stack 20/Oct/11 20:04

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development