Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14683

FileStatus.compareTo binary compatible issue

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.8.1
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: None
    • Labels:
      None

      Description

      See HIVE-17133. Looks like the signature change is causing issues; according to Jitendra Nath Pandey this is a public API.
      Is it possible to add the old overload back (keeping the new one presumably) in a point release on 2.8? That way we can avoid creating yet another shim for this in Hive.

      1. HADOOP-14683-branch-2-01.patch
        2 kB
        Akira Ajisaka
      2. HADOOP-14683-branch-2-02.patch
        2 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          ajisakaa Akira Ajisaka added a comment - - edited

          It is not possible to add old overload and keep new api because of compile error.

          [ERROR] /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java:[332,14] name clash: compareTo(org.apache.hadoop.fs.FileStatus) in org.apache.hadoop.fs.FileStatus overrides a method whose erasure is the same as another method, yet neither overrides the other
          [ERROR] first method:  compareTo(java.lang.Object) in org.apache.hadoop.fs.FileStatus
          [ERROR] second method: compareTo(T) in java.lang.Comparable
          

          This API is @Public and @Stable, so I'm thinking it's good to revert HADOOP-12209 from branch-2, branch-2.8, and branch-2.8.2.

          Show
          ajisakaa Akira Ajisaka added a comment - - edited It is not possible to add old overload and keep new api because of compile error. [ERROR] /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java:[332,14] name clash: compareTo(org.apache.hadoop.fs.FileStatus) in org.apache.hadoop.fs.FileStatus overrides a method whose erasure is the same as another method, yet neither overrides the other [ERROR] first method: compareTo(java.lang.Object) in org.apache.hadoop.fs.FileStatus [ERROR] second method: compareTo(T) in java.lang.Comparable This API is @Public and @Stable , so I'm thinking it's good to revert HADOOP-12209 from branch-2, branch-2.8, and branch-2.8.2.
          Hide
          busbey Sean Busbey added a comment -

          This should be upgraded to blocker on 2.8.2, IMHO.

          Show
          busbey Sean Busbey added a comment - This should be upgraded to blocker on 2.8.2, IMHO.
          Hide
          gopalv Gopal V added a comment -

          Akira Ajisaka: use Comparable<Object> ?

          Show
          gopalv Gopal V added a comment - Akira Ajisaka : use Comparable<Object> ?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Gopal V, it worked! Attaching a patch to add back the API.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Gopal V , it worked! Attaching a patch to add back the API.
          Hide
          daryn Daryn Sharp added a comment -

          Question, does this retain the binary search functionality?

          It would be wise to use instanceof prior to blindly casting.

          Show
          daryn Daryn Sharp added a comment - Question, does this retain the binary search functionality? It would be wise to use instanceof prior to blindly casting.
          Hide
          gopalv Gopal V added a comment - - edited

          does this retain the binary search functionality?

          Yes - that's why I suggested it.

          Also for ABI compat, it means that any existing code compiled with call-site compareTo(FileStatus) will continue to work, without breaking ABI from 2.8.0 -> 2.8.2.

          (edit: just realized for some calls, result will now be an Object, not FileStatus - needs a cast, to assign)

          As you said it used to handle non-FileStatus objects previously as not-equal.

          Show
          gopalv Gopal V added a comment - - edited does this retain the binary search functionality? Yes - that's why I suggested it. Also for ABI compat, it means that any existing code compiled with call-site compareTo(FileStatus) will continue to work, without breaking ABI from 2.8.0 -> 2.8.2. (edit: just realized for some calls, result will now be an Object, not FileStatus - needs a cast, to assign) As you said it used to handle non-FileStatus objects previously as not-equal.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 55s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                branch-2 Compile Tests
          +1 mvninstall 9m 25s branch-2 passed
          +1 compile 6m 8s branch-2 passed with JDK v1.8.0_131
          +1 compile 7m 12s branch-2 passed with JDK v1.7.0_131
          +1 checkstyle 0m 28s branch-2 passed
          +1 mvnsite 1m 1s branch-2 passed
          +1 findbugs 1m 44s branch-2 passed
          +1 javadoc 0m 43s branch-2 passed with JDK v1.8.0_131
          +1 javadoc 0m 52s branch-2 passed with JDK v1.7.0_131
                Patch Compile Tests
          +1 mvninstall 0m 42s the patch passed
          +1 compile 6m 11s the patch passed with JDK v1.8.0_131
          +1 javac 6m 11s the patch passed
          +1 compile 7m 13s the patch passed with JDK v1.7.0_131
          +1 javac 7m 13s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 1m 1s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 58s the patch passed
          +1 javadoc 0m 44s the patch passed with JDK v1.8.0_131
          +1 javadoc 0m 51s the patch passed with JDK v1.7.0_131
                Other Tests
          +1 unit 10m 4s hadoop-common in the patch passed with JDK v1.7.0_131.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          86m 32s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5e40efe
          JIRA Issue HADOOP-14683
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878774/HADOOP-14683-branch-2-01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fed69f16b197 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 3d8c7d7
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          findbugs v3.0.0
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12852/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12852/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 55s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       branch-2 Compile Tests +1 mvninstall 9m 25s branch-2 passed +1 compile 6m 8s branch-2 passed with JDK v1.8.0_131 +1 compile 7m 12s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 28s branch-2 passed +1 mvnsite 1m 1s branch-2 passed +1 findbugs 1m 44s branch-2 passed +1 javadoc 0m 43s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 52s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 42s the patch passed +1 compile 6m 11s the patch passed with JDK v1.8.0_131 +1 javac 6m 11s the patch passed +1 compile 7m 13s the patch passed with JDK v1.7.0_131 +1 javac 7m 13s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 58s the patch passed +1 javadoc 0m 44s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 51s the patch passed with JDK v1.7.0_131       Other Tests +1 unit 10m 4s hadoop-common in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 86m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HADOOP-14683 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878774/HADOOP-14683-branch-2-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fed69f16b197 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 3d8c7d7 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12852/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12852/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sershe Sergey Shelukhin added a comment -

          +1 non-binding

          Show
          sershe Sergey Shelukhin added a comment - +1 non-binding
          Hide
          ajisakaa Akira Ajisaka added a comment -

          It would be wise to use instanceof prior to blindly casting.

          Generally yes, but if we use instanceof, we need to throw some Exception explicitly, which is the same as casting without type check. I'll update javadoc to add @throws ClassCastException.

          Show
          ajisakaa Akira Ajisaka added a comment - It would be wise to use instanceof prior to blindly casting. Generally yes, but if we use instanceof, we need to throw some Exception explicitly, which is the same as casting without type check. I'll update javadoc to add @throws ClassCastException .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                branch-2 Compile Tests
          +1 mvninstall 6m 35s branch-2 passed
          +1 compile 5m 39s branch-2 passed with JDK v1.8.0_131
          +1 compile 6m 40s branch-2 passed with JDK v1.7.0_131
          +1 checkstyle 0m 25s branch-2 passed
          +1 mvnsite 0m 58s branch-2 passed
          +1 findbugs 1m 38s branch-2 passed
          +1 javadoc 0m 44s branch-2 passed with JDK v1.8.0_131
          +1 javadoc 0m 50s branch-2 passed with JDK v1.7.0_131
                Patch Compile Tests
          +1 mvninstall 0m 40s the patch passed
          +1 compile 5m 36s the patch passed with JDK v1.8.0_131
          +1 javac 5m 36s the patch passed
          +1 compile 6m 41s the patch passed with JDK v1.7.0_131
          +1 javac 6m 41s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 58s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 43s the patch passed with JDK v1.8.0_131
          +1 javadoc 0m 51s the patch passed with JDK v1.7.0_131
                Other Tests
          -1 unit 7m 55s hadoop-common in the patch failed with JDK v1.7.0_131.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          68m 51s



          Reason Tests
          JDK v1.8.0_131 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_131 Failed junit tests hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5e40efe
          JIRA Issue HADOOP-14683
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879112/HADOOP-14683-branch-2-02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 32b90de2241b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 4edb849
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12869/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_131.txt
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12869/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12869/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       branch-2 Compile Tests +1 mvninstall 6m 35s branch-2 passed +1 compile 5m 39s branch-2 passed with JDK v1.8.0_131 +1 compile 6m 40s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 25s branch-2 passed +1 mvnsite 0m 58s branch-2 passed +1 findbugs 1m 38s branch-2 passed +1 javadoc 0m 44s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 50s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 40s the patch passed +1 compile 5m 36s the patch passed with JDK v1.8.0_131 +1 javac 5m 36s the patch passed +1 compile 6m 41s the patch passed with JDK v1.7.0_131 +1 javac 6m 41s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 43s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 51s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 7m 55s hadoop-common in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 68m 51s Reason Tests JDK v1.8.0_131 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_131 Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HADOOP-14683 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879112/HADOOP-14683-branch-2-02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 32b90de2241b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 4edb849 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12869/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12869/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12869/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Generally yes, but if we use instanceof, we need to throw some Exception explicitly, which is the same as casting without type check. I'll update javadoc to add @throws ClassCastException.

          I think add javadoc should be fine given we cast blindly in other places, like: Path, SequenceFile, etc.

          02 patch LGTM. I think we should fix in trunk and 3.0-beta also. CC Andrew Wang.

          Show
          djp Junping Du added a comment - Generally yes, but if we use instanceof, we need to throw some Exception explicitly, which is the same as casting without type check. I'll update javadoc to add @throws ClassCastException. I think add javadoc should be fine given we cast blindly in other places, like: Path, SequenceFile, etc. 02 patch LGTM. I think we should fix in trunk and 3.0-beta also. CC Andrew Wang .
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the ping, looks like a good one to target for 3.0.0-beta1.

          I'll note that binary compatibility between 2.x and 3.0.0 is a non-goal, but I'm all for making it the transition as smooth as possible.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the ping, looks like a good one to target for 3.0.0-beta1. I'll note that binary compatibility between 2.x and 3.0.0 is a non-goal, but I'm all for making it the transition as smooth as possible.
          Hide
          djp Junping Du added a comment -

          Thanks for confirmation, Andrew.
          +1 on 02 patch. Will commit it after 24 hours if not further comments on latest patch.

          Show
          djp Junping Du added a comment - Thanks for confirmation, Andrew. +1 on 02 patch. Will commit it after 24 hours if not further comments on latest patch.
          Hide
          djp Junping Du added a comment -

          I have commit the patch to trunk, branch-2, branch-2.8 and branch-2.8.2. Thanks Sergey Shelukhin for reporting the issue and Akira Ajisaka for delivering a patch. Also, thanks all for review and comments.

          Show
          djp Junping Du added a comment - I have commit the patch to trunk, branch-2, branch-2.8 and branch-2.8.2. Thanks Sergey Shelukhin for reporting the issue and Akira Ajisaka for delivering a patch. Also, thanks all for review and comments.
          Hide
          sershe Sergey Shelukhin added a comment -

          Thanks!

          Show
          sershe Sergey Shelukhin added a comment - Thanks!

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              sershe Sergey Shelukhin
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development