Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Remove the objects and code only needed for supporting migration to 0.96 from 0.94.

      1. HBASE-10240.master.001.patch
        38 kB
        stack
      2. HBASE-10240.master.001.patch
        38 kB
        Peter Somogyi

        Activity

        Hide
        lhofhansl Lars Hofhansl added a comment -

        Are we saying one can only upgrade from 0.94 to 0.96 (but not to 0.98+)?

        Show
        lhofhansl Lars Hofhansl added a comment - Are we saying one can only upgrade from 0.94 to 0.96 (but not to 0.98+)?
        Hide
        apurtell Andrew Purtell added a comment -

        Previously we have upgraded users through each major release, but yeah with 0.98 coming quickly after 0.96 I can see a direct migration from 0.94 to 0.98 being convenient. I have no strong opinion either way. How long do we keep migration support for 0.94?

        Show
        apurtell Andrew Purtell added a comment - Previously we have upgraded users through each major release, but yeah with 0.98 coming quickly after 0.96 I can see a direct migration from 0.94 to 0.98 being convenient. I have no strong opinion either way. How long do we keep migration support for 0.94?
        Hide
        lhofhansl Lars Hofhansl added a comment -

        With the Apache hat on, I think we want to foster upgrading to new version.
        With the Salesforce hat on, we plan to upgrade from 0.94 to a 1.x version in 2014.

        If the upgrade code is (1) not getting in the way or (2) hard to maintain, I'd vote to keep it in at least until one of these becomes true.

        Show
        lhofhansl Lars Hofhansl added a comment - With the Apache hat on, I think we want to foster upgrading to new version. With the Salesforce hat on, we plan to upgrade from 0.94 to a 1.x version in 2014. If the upgrade code is (1) not getting in the way or (2) hard to maintain, I'd vote to keep it in at least until one of these becomes true.
        Hide
        enis Enis Soztutar added a comment -

        I would prefer to keep the option of 0.94 -> 0.98 available, which would ease out the user's burden. 0.96 and 0.98 code bases are not that different, so keeping that in makes managing the patches easier as well I think.

        Show
        enis Enis Soztutar added a comment - I would prefer to keep the option of 0.94 -> 0.98 available, which would ease out the user's burden. 0.96 and 0.98 code bases are not that different, so keeping that in makes managing the patches easier as well I think.
        Hide
        v.himanshu Himanshu Vashishtha added a comment -

        Yeah, I agree with keeping this code in 0.98 as it is getting released so soon after 0.96.0.
        Otherwise, it will add one extra step for users who wish to upgrade from 0.94 to 0.98.

        Show
        v.himanshu Himanshu Vashishtha added a comment - Yeah, I agree with keeping this code in 0.98 as it is getting released so soon after 0.96.0. Otherwise, it will add one extra step for users who wish to upgrade from 0.94 to 0.98.
        Hide
        apurtell Andrew Purtell added a comment -

        Ok, unscheduled from 0.98

        Show
        apurtell Andrew Purtell added a comment - Ok, unscheduled from 0.98
        Hide
        jmspaggi Jean-Marc Spaggiari added a comment -

        Late, but I agree with others. Always easier to keep the upgrade options to late comer. I agree that since the releases are close to each others, some might want to skip some of them...

        Show
        jmspaggi Jean-Marc Spaggiari added a comment - Late, but I agree with others. Always easier to keep the upgrade options to late comer. I agree that since the releases are close to each others, some might want to skip some of them...
        Hide
        enis Enis Soztutar added a comment -

        See HBASE-11915. Scheduling this for 2.0.

        Show
        enis Enis Soztutar added a comment - See HBASE-11915 . Scheduling this for 2.0.
        Hide
        stack stack added a comment -

        Things to remove:

        ./hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java

        ... what else?

        Show
        stack stack added a comment - Things to remove: ./hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java ... what else?
        Hide
        psomogyi Peter Somogyi added a comment -

        I think these can be removed:
        hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java
        hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestReference.java
        from hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java the testMigration method
        from hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java META_TABLE_NAME constant

        Show
        psomogyi Peter Somogyi added a comment - I think these can be removed: hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestReference.java from hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java the testMigration method from hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java META_TABLE_NAME constant
        Hide
        psomogyi Peter Somogyi added a comment -
        Show
        psomogyi Peter Somogyi added a comment - Review board: https://reviews.apache.org/r/60685/
        Hide
        stack stack added a comment -

        +1 Lets see what hadoopqa says.

        Show
        stack stack added a comment - +1 Lets see what hadoopqa says.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        1 @author 0m 0s The patch does not contain any @author tags.
        1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        1 mvninstall 3m 28s master passed
        1 compile 0m 55s master passed
        1 checkstyle 0m 44s master passed
        1 mvneclipse 0m 26s master passed
        -1 findbugs 0m 34s hbase-common in master has 2 extant Findbugs warnings.
        -1 findbugs 2m 54s hbase-server in master has 10 extant Findbugs warnings.
        1 javadoc 0m 47s master passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        1 mvninstall 1m 2s the patch passed
        1 compile 0m 57s the patch passed
        1 javac 0m 57s the patch passed
        1 checkstyle 0m 40s the patch passed
        1 mvneclipse 0m 25s the patch passed
        1 whitespace 0m 0s The patch has no whitespace issues.
        1 hadoopcheck 36m 40s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3.
        1 findbugs 3m 39s the patch passed
        1 javadoc 0m 45s the patch passed
        1 unit 2m 13s hbase-common in the patch passed.
        -1 unit 120m 56s hbase-server in the patch failed.
        1 asflicense 0m 33s The patch does not generate ASF License warnings.
        178m 57s



        Reason Tests
        Failed junit tests hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37
        JIRA Issue HBASE-10240
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875939/HBASE-10240.master.001.patch
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux 300a76c835db 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 75d2eca
        Default Java 1.8.0_131
        findbugs v3.1.0-RC3
        findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/7541/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html
        findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/7541/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/7541/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/7541/testReport/
        modules C: hbase-common hbase-server U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/7541/console
        Powered by Apache Yetus 0.4.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. 1 hbaseanti 0m 0s Patch does not have any anti-patterns. 1 @author 0m 0s The patch does not contain any @author tags. 1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch 1 mvninstall 3m 28s master passed 1 compile 0m 55s master passed 1 checkstyle 0m 44s master passed 1 mvneclipse 0m 26s master passed -1 findbugs 0m 34s hbase-common in master has 2 extant Findbugs warnings. -1 findbugs 2m 54s hbase-server in master has 10 extant Findbugs warnings. 1 javadoc 0m 47s master passed 0 mvndep 0m 17s Maven dependency ordering for patch 1 mvninstall 1m 2s the patch passed 1 compile 0m 57s the patch passed 1 javac 0m 57s the patch passed 1 checkstyle 0m 40s the patch passed 1 mvneclipse 0m 25s the patch passed 1 whitespace 0m 0s The patch has no whitespace issues. 1 hadoopcheck 36m 40s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. 1 findbugs 3m 39s the patch passed 1 javadoc 0m 45s the patch passed 1 unit 2m 13s hbase-common in the patch passed. -1 unit 120m 56s hbase-server in the patch failed. 1 asflicense 0m 33s The patch does not generate ASF License warnings. 178m 57s Reason Tests Failed junit tests hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 JIRA Issue HBASE-10240 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875939/HBASE-10240.master.001.patch Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux 300a76c835db 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 75d2eca Default Java 1.8.0_131 findbugs v3.1.0-RC3 findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/7541/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/7541/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html unit https://builds.apache.org/job/PreCommit-HBASE-Build/7541/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/7541/testReport/ modules C: hbase-common hbase-server U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/7541/console Powered by Apache Yetus 0.4.0 http://yetus.apache.org This message was automatically generated.
        Hide
        stack stack added a comment -

        Retry

        Show
        stack stack added a comment - Retry
        Hide
        enis Enis Soztutar added a comment -

        I remember that the ACLs in the ACL table are not auto-migrated to the PB format (last time I checked was long time ago though). If that is the case, if any cluster has those left in the ACL table because the cluster is going through 0.94 -> 0.98 -> 1.x -> 2.0 code bases, than the ACLs will fail. Can we please check that.

        Show
        enis Enis Soztutar added a comment - I remember that the ACLs in the ACL table are not auto-migrated to the PB format (last time I checked was long time ago though). If that is the case, if any cluster has those left in the ACL table because the cluster is going through 0.94 -> 0.98 -> 1.x -> 2.0 code bases, than the ACLs will fail. Can we please check that.
        Hide
        stack stack added a comment -

        Thanks Enis Soztutar for the helpful input.

        Show
        stack stack added a comment - Thanks Enis Soztutar for the helpful input.
        Hide
        stack stack added a comment -

        Or, thinking on it Enis Soztutar, we should just not support going from 0.94 to 2.0. I think this purge is legit in 2.0. Let us know if you think different..... For the ACL issue, can I add to our migration guide how to address old serializations (do you know)? I'll add note in meantime....

        Show
        stack stack added a comment - Or, thinking on it Enis Soztutar , we should just not support going from 0.94 to 2.0. I think this purge is legit in 2.0. Let us know if you think different..... For the ACL issue, can I add to our migration guide how to address old serializations (do you know)? I'll add note in meantime....
        Hide
        stack stack added a comment -

        I added note over on HBASE-13631 in meantime.

        Show
        stack stack added a comment - I added note over on HBASE-13631 in meantime.
        Hide
        enis Enis Soztutar added a comment -

        Thanks Stack. I think the problem is that even if you have a 1.3 cluster, if you have upgraded from 0.94 before, you are left with Writable data since we do not auto-migrate. And there is no migration code to re-write the ACL table as of now.

        Show
        enis Enis Soztutar added a comment - Thanks Stack. I think the problem is that even if you have a 1.3 cluster, if you have upgraded from 0.94 before, you are left with Writable data since we do not auto-migrate. And there is no migration code to re-write the ACL table as of now.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        1 @author 0m 0s The patch does not contain any @author tags.
        1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        1 mvninstall 3m 20s master passed
        1 compile 0m 53s master passed
        1 checkstyle 0m 39s master passed
        1 mvneclipse 0m 25s master passed
        -1 findbugs 0m 32s hbase-common in master has 2 extant Findbugs warnings.
        -1 findbugs 2m 36s hbase-server in master has 10 extant Findbugs warnings.
        1 javadoc 0m 43s master passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        1 mvninstall 1m 1s the patch passed
        1 compile 0m 54s the patch passed
        1 javac 0m 54s the patch passed
        1 checkstyle 0m 39s the patch passed
        1 mvneclipse 0m 24s the patch passed
        1 whitespace 0m 0s The patch has no whitespace issues.
        1 hadoopcheck 29m 8s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3.
        1 findbugs 3m 45s the patch passed
        1 javadoc 0m 45s the patch passed
        1 unit 2m 11s hbase-common in the patch passed.
        1 unit 113m 15s hbase-server in the patch passed.
        1 asflicense 0m 32s The patch does not generate ASF License warnings.
        162m 56s



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37
        JIRA Issue HBASE-10240
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875961/HBASE-10240.master.001.patch
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux 64886360651a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 75d2eca
        Default Java 1.8.0_131
        findbugs v3.1.0-RC3
        findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/7546/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html
        findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/7546/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/7546/testReport/
        modules C: hbase-common hbase-server U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/7546/console
        Powered by Apache Yetus 0.4.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. 1 hbaseanti 0m 0s Patch does not have any anti-patterns. 1 @author 0m 0s The patch does not contain any @author tags. 1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch 1 mvninstall 3m 20s master passed 1 compile 0m 53s master passed 1 checkstyle 0m 39s master passed 1 mvneclipse 0m 25s master passed -1 findbugs 0m 32s hbase-common in master has 2 extant Findbugs warnings. -1 findbugs 2m 36s hbase-server in master has 10 extant Findbugs warnings. 1 javadoc 0m 43s master passed 0 mvndep 0m 15s Maven dependency ordering for patch 1 mvninstall 1m 1s the patch passed 1 compile 0m 54s the patch passed 1 javac 0m 54s the patch passed 1 checkstyle 0m 39s the patch passed 1 mvneclipse 0m 24s the patch passed 1 whitespace 0m 0s The patch has no whitespace issues. 1 hadoopcheck 29m 8s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. 1 findbugs 3m 45s the patch passed 1 javadoc 0m 45s the patch passed 1 unit 2m 11s hbase-common in the patch passed. 1 unit 113m 15s hbase-server in the patch passed. 1 asflicense 0m 32s The patch does not generate ASF License warnings. 162m 56s Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 JIRA Issue HBASE-10240 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875961/HBASE-10240.master.001.patch Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux 64886360651a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 75d2eca Default Java 1.8.0_131 findbugs v3.1.0-RC3 findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/7546/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/7546/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/7546/testReport/ modules C: hbase-common hbase-server U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/7546/console Powered by Apache Yetus 0.4.0 http://yetus.apache.org This message was automatically generated.
        Hide
        stack stack added a comment -

        Ok Enis Soztutar Then I suppose migration tool will need at least an instance of HbaseObjectWritableFor96Migration. You know if this discussed elsewhere Enis Soztutar ? (I can look myself.. but in case you remember...)

        Ok Peter Somogyi. We can't go further here till we figure out the auto-migration of ACL in branch-1. Thanks for the patch to get us this far.

        Show
        stack stack added a comment - Ok Enis Soztutar Then I suppose migration tool will need at least an instance of HbaseObjectWritableFor96Migration. You know if this discussed elsewhere Enis Soztutar ? (I can look myself.. but in case you remember...) Ok Peter Somogyi . We can't go further here till we figure out the auto-migration of ACL in branch-1. Thanks for the patch to get us this far.
        Hide
        enis Enis Soztutar added a comment -

        I was doing some digging. Findings so far:
        9e249881ef6a49879511be24e11d717738085dd5 (HBASE-7224) added this to the javadoc for HbaseObjectWritableFor96Migration

         * @deprecated This class is needed migrating TablePermissions written with
         * Writables.  It is needed to read old permissions written pre-0.96.  This
         * class is to be removed after HBase 0.96 ships since then all permissions
         * will have been migrated and written with protobufs.
        

        However, when reading the code TablePermissions, which extends Permission which is still Writable and never made PB I guess. There is also a couple of other Writables like AuthenticationKey.

        It seems (from my reading of the code) that we are using PB's in the RPC, but we are reading and writing these objects from the ACL table or ZK using Writables. However, I think neither of these depends on HbaseObjectWritableFor96Migration per se, so the patch maybe good as it is.

        Show
        enis Enis Soztutar added a comment - I was doing some digging. Findings so far: 9e249881ef6a49879511be24e11d717738085dd5 ( HBASE-7224 ) added this to the javadoc for HbaseObjectWritableFor96Migration * @deprecated This class is needed migrating TablePermissions written with * Writables. It is needed to read old permissions written pre-0.96. This * class is to be removed after HBase 0.96 ships since then all permissions * will have been migrated and written with protobufs. However, when reading the code TablePermissions , which extends Permission which is still Writable and never made PB I guess. There is also a couple of other Writables like AuthenticationKey. It seems (from my reading of the code) that we are using PB's in the RPC, but we are reading and writing these objects from the ACL table or ZK using Writables. However, I think neither of these depends on HbaseObjectWritableFor96Migration per se, so the patch maybe good as it is.
        Hide
        stack stack added a comment -

        Thanks Enis Soztutar Would be cool to purge this stuff if we can.

        Show
        stack stack added a comment - Thanks Enis Soztutar Would be cool to purge this stuff if we can.
        Hide
        psomogyi Peter Somogyi added a comment -

        Thanks Enis Soztutar and stack for the info! I understand that this is a critical area for upgrade. Let me know if we can go forward with this or some change is needed to the patch.

        Show
        psomogyi Peter Somogyi added a comment - Thanks Enis Soztutar and stack for the info! I understand that this is a critical area for upgrade. Let me know if we can go forward with this or some change is needed to the patch.
        Hide
        stack stack added a comment -

        I'd suggest then that the way forward is to look for classes that implement Writables. We'd write instances of each Writables-implementing class, then apply this patch. Can we still read from the Writables after this patch has gone in?

        Show
        stack stack added a comment - I'd suggest then that the way forward is to look for classes that implement Writables. We'd write instances of each Writables-implementing class, then apply this patch. Can we still read from the Writables after this patch has gone in?
        Hide
        stack stack added a comment -

        Can we purge any of the remaining Writables? We were supposed to Writables-free long time ago.... by 0.96 or 0.98 at the latest.

        Show
        stack stack added a comment - Can we purge any of the remaining Writables? We were supposed to Writables-free long time ago.... by 0.96 or 0.98 at the latest.

          People

          • Assignee:
            psomogyi Peter Somogyi
            Reporter:
            apurtell Andrew Purtell
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development