Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18622

Mitigate API compatibility concerns between branch-1 and branch-2

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 2.1.0, 2.0.2
    • API
    • None

    Description

      This project is to do what apurtell did in the issue "HBASE-18431 Mitigate compatibility concerns between branch-1.3 and branch-1.4" only do it between branch-1 and branch-2.

      Attachments

        1. report.1.2_2.0.html.gz
          321 kB
          Michael Stack

        Activity

          stack Michael Stack added a comment -

          Compared released 1.2.6 – a likely starting point – to branch-2 by doing below:

          $ ./dev-support/checkcompatibility.py 2f9b9e17d0522e36063bf52ecc58576243d20b3f fad968d99f4693278edcaefff92738e67ad8e8a0
          

          Report attached.

          stack Michael Stack added a comment - Compared released 1.2.6 – a likely starting point – to branch-2 by doing below: $ ./dev-support/checkcompatibility.py 2f9b9e17d0522e36063bf52ecc58576243d20b3f fad968d99f4693278edcaefff92738e67ad8e8a0 Report attached.
          stack Michael Stack added a comment -

          Sorry, report is 120MB, can't attach.

          stack Michael Stack added a comment - Sorry, report is 120MB, can't attach.
          stack Michael Stack added a comment -

          Here is summary:

          API compatibility report for the hbase library between 2f9b9e17d0522e36063bf52ecc58576243d20b3f and fad968d99f4693278edcaefff92738e67ad8e8a0 versions

          Test Info
          Library Name hbase
          Version #1 2f9b9e17d0522e36063bf52ecc58576243d20b3f
          Version #2 fad968d99f4693278edcaefff92738e67ad8e8a0
          Subject Binary Compatibility
          Test Results
          Total JARs 13
          Total Methods / Classes 142606 / 6306
          Compatibility 66.9%
          Problem Summary
          Severity Count
          Added Methods - 78007
          Removed Methods High 15834
          Problems with
          Data Types High 1131
          Medium 244
          Low 181
          Problems with
          Methods High 74
          Medium 0
          Low 79
          Other Changes
          in Data Types - 1004
          Other Changes
          in Methods - 160

          stack Michael Stack added a comment - Here is summary: API compatibility report for the hbase library between 2f9b9e17d0522e36063bf52ecc58576243d20b3f and fad968d99f4693278edcaefff92738e67ad8e8a0 versions Test Info Library Name hbase Version #1 2f9b9e17d0522e36063bf52ecc58576243d20b3f Version #2 fad968d99f4693278edcaefff92738e67ad8e8a0 Subject Binary Compatibility Test Results Total JARs 13 Total Methods / Classes 142606 / 6306 Compatibility 66.9% Problem Summary Severity Count Added Methods - 78007 Removed Methods High 15834 Problems with Data Types High 1131 Medium 244 Low 181 Problems with Methods High 74 Medium 0 Low 79 Other Changes in Data Types - 1004 Other Changes in Methods - 160
          stack Michael Stack added a comment -

          Excluding protocol, shaded, and archetype and new jars like backup and rsgroup makes the output more manageable:

          ./dev-support/checkcompatibility.py -e .*hbase-rsgroup.* -e .*hbase-backup.* -e .*hbase-protoco.* -e .*hbase-shaded.* -e .*origina-hbase.* -e .*hbase-archetypes.* -e .*hbase-examples.* 2f9b9e17d0522e36063bf52ecc58576243d20b3f fad968d99f4693278edcaefff92738e67ad8e8a0
          
          Test Info
          Library Name	hbase
          Version #1	2f9b9e17d0522e36063bf52ecc58576243d20b3f
          Version #2	fad968d99f4693278edcaefff92738e67ad8e8a0
          Subject	Binary Compatibility
          Test Results
          Total JARs	11
          Total Methods / Classes	40212 / 3303
          Compatibility	70.5%
          Problem Summary
          Severity	Count
          Added Methods	-	12218
          Removed Methods	High	4140
          Problems with
          Data Types	High	688
          Medium	243
          Low	178
          Problems with
          Methods	High	74
          Medium	0
          Low	79
          Other Changes
          in Data Types	-	878
          Other Changes
          in Methods	-	160
          Added Methods  12218 
          

          Admin branch-2 has 80 methods more than was in branch-1 Admin.

          There are too many adds and removes to review.

          Going over the methods flagged problematic, a good portion can be ignored since the class was InterfaceAudience Private in hbase-1.0; e.g. Call or ConnectionUtils.

          We have a few methods that differ in signature in Admin. If we say hbase1 can't admin an hbase2 cluster, then we can let below slide:

          hbase-client-1.2.6.jar, Admin.class
          package org.apache.hadoop.hbase.client
          [−] Admin.createTableAsync ( HTableDescriptor p1, byte[ ][ ] p2 ) [abstract]  :  void  1  
          org/apache/hadoop/hbase/client/Admin.createTableAsync:(Lorg/apache/hadoop/hbase/HTableDescriptor;[[B)V
          Change	Effect
          1	Return value type has been changed from void to java.util.concurrent.Future<java.lang.Void>.	This method has been removed because the return type is part of the method signature.
          
          [−] Admin.disableTableAsync ( TableName p1 ) [abstract]  :  void  1  
          org/apache/hadoop/hbase/client/Admin.disableTableAsync:(Lorg/apache/hadoop/hbase/TableName;)V
          Change	Effect
          1	Return value type has been changed from void to java.util.concurrent.Future<java.lang.Void>.	This method has been removed because the return type is part of the method signature.
          
          [−] Admin.enableTableAsync ( TableName p1 ) [abstract]  :  void  1  
          org/apache/hadoop/hbase/client/Admin.enableTableAsync:(Lorg/apache/hadoop/hbase/TableName;)V
          Change	Effect
          1	Return value type has been changed from void to java.util.concurrent.Future<java.lang.Void>.	This method has been removed because the return type is part of the method signature.
          
          

          Filing subtasks for violations found.

          Got as far as: hbase-common-1.2.6.jar, CellComparator.class Will be back.

          stack Michael Stack added a comment - Excluding protocol, shaded, and archetype and new jars like backup and rsgroup makes the output more manageable: ./dev-support/checkcompatibility.py -e .*hbase-rsgroup.* -e .*hbase-backup.* -e .*hbase-protoco.* -e .*hbase-shaded.* -e .*origina-hbase.* -e .*hbase-archetypes.* -e .*hbase-examples.* 2f9b9e17d0522e36063bf52ecc58576243d20b3f fad968d99f4693278edcaefff92738e67ad8e8a0 Test Info Library Name hbase Version #1 2f9b9e17d0522e36063bf52ecc58576243d20b3f Version #2 fad968d99f4693278edcaefff92738e67ad8e8a0 Subject Binary Compatibility Test Results Total JARs 11 Total Methods / Classes 40212 / 3303 Compatibility 70.5% Problem Summary Severity Count Added Methods - 12218 Removed Methods High 4140 Problems with Data Types High 688 Medium 243 Low 178 Problems with Methods High 74 Medium 0 Low 79 Other Changes in Data Types - 878 Other Changes in Methods - 160 Added Methods 12218 Admin branch-2 has 80 methods more than was in branch-1 Admin. There are too many adds and removes to review. Going over the methods flagged problematic, a good portion can be ignored since the class was InterfaceAudience Private in hbase-1.0; e.g. Call or ConnectionUtils. We have a few methods that differ in signature in Admin. If we say hbase1 can't admin an hbase2 cluster, then we can let below slide: hbase-client-1.2.6.jar, Admin. class package org.apache.hadoop.hbase.client [−] Admin.createTableAsync ( HTableDescriptor p1, byte [ ][ ] p2 ) [ abstract ] : void 1 org/apache/hadoop/hbase/client/Admin.createTableAsync:(Lorg/apache/hadoop/hbase/HTableDescriptor;[[B)V Change Effect 1 Return value type has been changed from void to java.util.concurrent.Future<java.lang. Void >. This method has been removed because the return type is part of the method signature. [−] Admin.disableTableAsync ( TableName p1 ) [ abstract ] : void 1 org/apache/hadoop/hbase/client/Admin.disableTableAsync:(Lorg/apache/hadoop/hbase/TableName;)V Change Effect 1 Return value type has been changed from void to java.util.concurrent.Future<java.lang. Void >. This method has been removed because the return type is part of the method signature. [−] Admin.enableTableAsync ( TableName p1 ) [ abstract ] : void 1 org/apache/hadoop/hbase/client/Admin.enableTableAsync:(Lorg/apache/hadoop/hbase/TableName;)V Change Effect 1 Return value type has been changed from void to java.util.concurrent.Future<java.lang. Void >. This method has been removed because the return type is part of the method signature. Filing subtasks for violations found. Got as far as: hbase-common-1.2.6.jar, CellComparator.class Will be back.
          stack Michael Stack added a comment -

          I might be going about this wrong. Rather, I should just review the differences and then write unit tests that have hbase1 client going against an hbase2 cluster exercising problematic areas to see if incompatibilities show up on other side of the wire.

          stack Michael Stack added a comment - I might be going about this wrong. Rather, I should just review the differences and then write unit tests that have hbase1 client going against an hbase2 cluster exercising problematic areas to see if incompatibilities show up on other side of the wire.
          stack Michael Stack added a comment -

          Here is how I generated the report:

          $ ./dev-support/checkcompatibility.py --annotation org.apache.hadoop.hbase.classification.InterfaceAudience.Public --annotation org.apache.hadoop.hbase.classification.InterfaceAudience.LimitedPrivate 6c98bff7b719efdb16f71606f3b7d8229445eb81 a0f980b573b5af134002f0cb9163384d694b4388
          

          I'd edited ./dev-support/checkcompatibiilty.py to use JACC 2.3 instead of 2.1 because it had this fix... https://github.com/lvc/japi-compliance-checker/issues/40 (via busbey

          Otherwise the reports are massive including privately annotated classes.

          stack Michael Stack added a comment - Here is how I generated the report: $ ./dev-support/checkcompatibility.py --annotation org.apache.hadoop.hbase.classification.InterfaceAudience.Public --annotation org.apache.hadoop.hbase.classification.InterfaceAudience.LimitedPrivate 6c98bff7b719efdb16f71606f3b7d8229445eb81 a0f980b573b5af134002f0cb9163384d694b4388 I'd edited ./dev-support/checkcompatibiilty.py to use JACC 2.3 instead of 2.1 because it had this fix... https://github.com/lvc/japi-compliance-checker/issues/40 (via busbey Otherwise the reports are massive including privately annotated classes.
          stack Michael Stack added a comment -

          Of note, to be complete, I need to not only compare 1.0.0 to branch-2 but 1.1.0 to branch-2 and so on...

          stack Michael Stack added a comment - Of note, to be complete, I need to not only compare 1.0.0 to branch-2 but 1.1.0 to branch-2 and so on...
          stack Michael Stack added a comment -

          List of incompatibilties are in section #6 of the 2.0 doc https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.723jjn18p2jr Will tidy it up soon (don't look yet!)

          stack Michael Stack added a comment - List of incompatibilties are in section #6 of the 2.0 doc https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.723jjn18p2jr Will tidy it up soon (don't look yet!)
          stack Michael Stack added a comment -

          JACC report of 1.2 to 2.0. I've gone through a few times. Have a glaze now and can't look at it anymore. I think I have most captured in writeup at end of this doc in compatibility chapter: https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.xjagihujxydb Need to do some cleanup to make it more palatable.

          stack Michael Stack added a comment - JACC report of 1.2 to 2.0. I've gone through a few times. Have a glaze now and can't look at it anymore. I think I have most captured in writeup at end of this doc in compatibility chapter: https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.xjagihujxydb Need to do some cleanup to make it more palatable.
          stack Michael Stack added a comment -

          InterfaceAudience.Private is saving our butt all over the place.

          HColumnDescriptor went from this in 1.0.0

          @InterfaceAudience.Public
          @InterfaceStability.Evolving
          public class HColumnDescriptor implements WritableComparable<HColumnDescriptor> {
          

          to this in 2.0

          @InterfaceAudience.Public
          @Deprecated // remove it in 3.0
          public class HColumnDescriptor implements ColumnFamilyDescriptor, Comparable<HColumnDescriptor> {
          

          Figuring this ok.

          Otherwise, filed subtasks here and updated the incompatibility section in the 'hbase2 book': https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.723jjn18p2jr

          stack Michael Stack added a comment - InterfaceAudience.Private is saving our butt all over the place. HColumnDescriptor went from this in 1.0.0 @InterfaceAudience.Public @InterfaceStability.Evolving public class HColumnDescriptor implements WritableComparable<HColumnDescriptor> { to this in 2.0 @InterfaceAudience.Public @Deprecated // remove it in 3.0 public class HColumnDescriptor implements ColumnFamilyDescriptor, Comparable<HColumnDescriptor> { Figuring this ok. Otherwise, filed subtasks here and updated the incompatibility section in the 'hbase2 book': https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.723jjn18p2jr
          stack Michael Stack added a comment -

          This is an on-going 2.0.0 project. Best list of incompatibles will only be known on day we make release. Meantime we will keep turning them up whether via JACC examination or via testing.

          stack Michael Stack added a comment - This is an on-going 2.0.0 project. Best list of incompatibles will only be known on day we make release. Meantime we will keep turning them up whether via JACC examination or via testing.
          stack Michael Stack added a comment -

          Reran the report

          Changed the reporting tool to use 2.4 instead of 2.1 (released in January).

          diff --git a/dev-support/checkcompatibility.py b/dev-support/checkcompatibility.py
          index ea9c229344..9f0d797ff3 100755
          — a/dev-support/checkcompatibility.py
          +++ b/dev-support/checkcompatibility.py
          @@ -156,7 +156,7 @@ def checkout_java_acc(force):

          logging.info("Downloading Java ACC...")

          • url = "https://github.com/lvc/japi-compliance-checker/archive/2.1.tar.gz"
            + url = "https://github.com/lvc/japi-compliance-checker/archive/2.4.tar.gz"
            scratch_dir = get_scratch_dir()
            path = os.path.join(scratch_dir, os.path.basename(url))
            jacc = urllib2.urlopen(url)
            @@ -166,7 +166,7 @@ def checkout_java_acc(force):
            subprocess.check_call(["tar", "xzf", path],
            cwd=scratch_dir)
          • shutil.move(os.path.join(scratch_dir, "japi-compliance-checker-2.1"),
            + shutil.move(os.path.join(scratch_dir, "japi-compliance-checker-2.4"),
            os.path.join(acc_dir))

          $ ./dev-support/checkcompatibility.py --annotation org.apache.hadoop.hbase.classification.InterfaceAudience.Public rel/1.2.6 2.0.0RC0

          stack Michael Stack added a comment - Reran the report Changed the reporting tool to use 2.4 instead of 2.1 (released in January). diff --git a/dev-support/checkcompatibility.py b/dev-support/checkcompatibility.py index ea9c229344..9f0d797ff3 100755 — a/dev-support/checkcompatibility.py +++ b/dev-support/checkcompatibility.py @@ -156,7 +156,7 @@ def checkout_java_acc(force): logging.info("Downloading Java ACC...") url = "https://github.com/lvc/japi-compliance-checker/archive/2.1.tar.gz" + url = "https://github.com/lvc/japi-compliance-checker/archive/2.4.tar.gz" scratch_dir = get_scratch_dir() path = os.path.join(scratch_dir, os.path.basename(url)) jacc = urllib2.urlopen(url) @@ -166,7 +166,7 @@ def checkout_java_acc(force): subprocess.check_call( ["tar", "xzf", path] , cwd=scratch_dir) shutil.move(os.path.join(scratch_dir, "japi-compliance-checker-2.1"), + shutil.move(os.path.join(scratch_dir, "japi-compliance-checker-2.4"), os.path.join(acc_dir)) $ ./dev-support/checkcompatibility.py --annotation org.apache.hadoop.hbase.classification.InterfaceAudience.Public rel/1.2.6 2.0.0RC0
          zhangduo Duo Zhang added a comment -

          Do we still have unfinished work for this issue? It is marked as blocker...

          zhangduo Duo Zhang added a comment - Do we still have unfinished work for this issue? It is marked as blocker...
          zhangduo Duo Zhang added a comment -

          Ping stack busbey. Thanks.

          zhangduo Duo Zhang added a comment - Ping stack busbey . Thanks.
          stack Michael Stack added a comment -

          We do. Need to write up report on what diffs are. It didn't block 2.0 so can't block 2.1.

          stack Michael Stack added a comment - We do. Need to write up report on what diffs are. It didn't block 2.0 so can't block 2.1.
          stack Michael Stack added a comment -

          Resolving. Subtasks done. We published a report when we released 2.0.0 (though its probably hard to digest – could have done better).

          stack Michael Stack added a comment - Resolving. Subtasks done. We published a report when we released 2.0.0 (though its probably hard to digest – could have done better).

          People

            stack Michael Stack
            stack Michael Stack
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: