Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14730

Support protobuf FileStatus in AdlFileSystem

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: 3.0.0-beta1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      2 Unit Test cases are failing Azure-data-lake Module , caused after HDFS-6984 commit.

      Issue seems to be hasAcl is hard coded to false.

      public FileStatus(long length, boolean isdir,
                          int block_replication,
                          long blocksize, long modification_time, long access_time,
                          FsPermission permission, String owner, String group, 
                          Path symlink,
                          Path path) {
          this(length, isdir, block_replication, blocksize, modification_time,
              access_time, permission, owner, group, symlink, path,
              false, false, false);
        }
      
      1. HADOOP-14730.001.patch
        0.8 kB
        Vishwajeet Dusane
      2. HADOOP-14730.002.patch
        8 kB
        Chris Douglas
      3. HADOOP-14730.003.patch
        5 kB
        Chris Douglas
      4. HADOOP-14730.004.patch
        8 kB
        Chris Douglas
      5. HADOOP-14730.005.patch
        10 kB
        Chris Douglas
      6. HADOOP-14730.006.patch
        10 kB
        Chris Douglas

        Issue Links

          Activity

          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Proposed change while initializing FileStatus. Set hasAcl value based on the FsPermission AclBit property if available.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Proposed change while initializing FileStatus. Set hasAcl value based on the FsPermission AclBit property if available.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          +1 mvninstall 16m 27s trunk passed
          +1 compile 16m 16s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 37s trunk passed
          +1 findbugs 1m 35s trunk passed
          +1 javadoc 0m 51s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 47s the patch passed
          +1 compile 12m 12s the patch passed
          -1 javac 12m 12s root generated 1 new + 1418 unchanged - 0 fixed = 1419 total (was 1418)
          +1 checkstyle 0m 38s the patch passed
          +1 mvnsite 1m 37s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 55s the patch passed
                Other Tests
          +1 unit 8m 53s hadoop-common in the patch passed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          67m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14730
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880204/HADOOP-14730.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 915bd78839d3 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c5d256c
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12941/artifact/patchprocess/diff-compile-javac-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12941/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12941/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 16m 27s trunk passed +1 compile 16m 16s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 37s trunk passed +1 findbugs 1m 35s trunk passed +1 javadoc 0m 51s trunk passed       Patch Compile Tests +1 mvninstall 0m 47s the patch passed +1 compile 12m 12s the patch passed -1 javac 12m 12s root generated 1 new + 1418 unchanged - 0 fixed = 1419 total (was 1418) +1 checkstyle 0m 38s the patch passed +1 mvnsite 1m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 55s the patch passed       Other Tests +1 unit 8m 53s hadoop-common in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 67m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14730 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880204/HADOOP-14730.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 915bd78839d3 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c5d256c Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12941/artifact/patchprocess/diff-compile-javac-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12941/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12941/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Alternative changing the ADL client to call the appropriate constructor, so we can remove the ADLPermission class.

          Show
          chris.douglas Chris Douglas added a comment - Alternative changing the ADL client to call the appropriate constructor, so we can remove the ADLPermission class.
          Hide
          chris.douglas Chris Douglas added a comment -

          Alternative patch, if the deprecated FsPermission behavior is still required.

          Show
          chris.douglas Chris Douglas added a comment - Alternative patch, if the deprecated FsPermission behavior is still required.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks Chris Douglas for Patch 003 and +1 on the change.

          Had a offline sync with Chris Douglas. Agree that HDFS-6984 is not backward compatible change. So as hadoop-azure-datalake.jar with Patch 003 would not link to Hadoop 2.x common jar, mainly for the HDFS-6984 introduced new FileStatus constructor. Patching Hadoop 2.X cluster with Hadoop Adl jar will not work.

          CC: Mingliang Liu and John Zhuge

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks Chris Douglas for Patch 003 and +1 on the change. Had a offline sync with Chris Douglas . Agree that HDFS-6984 is not backward compatible change. So as hadoop-azure-datalake.jar with Patch 003 would not link to Hadoop 2.x common jar, mainly for the HDFS-6984 introduced new FileStatus constructor. Patching Hadoop 2.X cluster with Hadoop Adl jar will not work. CC: Mingliang Liu and John Zhuge
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 17m 22s trunk passed
          +1 compile 0m 20s trunk passed
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 15s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 16s the patch passed
          -1 javac 0m 16s hadoop-tools_hadoop-azure-datalake generated 4 new + 5 unchanged - 6 fixed = 9 total (was 11)
          -0 checkstyle 0m 11s hadoop-tools/hadoop-azure-datalake: The patch generated 4 new + 16 unchanged - 4 fixed = 20 total (was 20)
          +1 mvnsite 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 36s the patch passed
          +1 javadoc 0m 12s the patch passed
                Other Tests
          +1 unit 3m 38s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          26m 30s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14730
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880286/HADOOP-14730.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e3ae722c8526 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f4c6b00
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12948/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12948/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12948/testReport/
          modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12948/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 17m 22s trunk passed +1 compile 0m 20s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 22s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 15s trunk passed       Patch Compile Tests +1 mvninstall 0m 18s the patch passed +1 compile 0m 16s the patch passed -1 javac 0m 16s hadoop-tools_hadoop-azure-datalake generated 4 new + 5 unchanged - 6 fixed = 9 total (was 11) -0 checkstyle 0m 11s hadoop-tools/hadoop-azure-datalake: The patch generated 4 new + 16 unchanged - 4 fixed = 20 total (was 20) +1 mvnsite 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 36s the patch passed +1 javadoc 0m 12s the patch passed       Other Tests +1 unit 3m 38s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 26m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14730 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880286/HADOOP-14730.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e3ae722c8526 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f4c6b00 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12948/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12948/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12948/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12948/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Vishwajeet Dusane and Chris Douglas for the patches.

          Any reason to prefer Patch 003 over 002? Considering FSPermission.getAclBit is deprecated.

          Show
          jzhuge John Zhuge added a comment - Thanks Vishwajeet Dusane and Chris Douglas for the patches. Any reason to prefer Patch 003 over 002? Considering FSPermission.getAclBit is deprecated.
          Hide
          chris.douglas Chris Douglas added a comment -

          hadoop-azure-datalake.jar with Patch 003 would not link to Hadoop 2.x common jar

          This is an unusual requirement. Attached v004 which should link against 2.x.

          Any reason to prefer Patch 003 over 002? Considering FSPermission.getAclBit is deprecated.

          Just in case applications rely on this field, presumably. HDFS-6984 preserved this for HDFS, but not for ADLS.

          I'm ambivalent among v002-v004.

          Show
          chris.douglas Chris Douglas added a comment - hadoop-azure-datalake.jar with Patch 003 would not link to Hadoop 2.x common jar This is an unusual requirement. Attached v004 which should link against 2.x. Any reason to prefer Patch 003 over 002? Considering FSPermission.getAclBit is deprecated. Just in case applications rely on this field, presumably. HDFS-6984 preserved this for HDFS, but not for ADLS. I'm ambivalent among v002-v004.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 16m 2s trunk passed
          +1 compile 0m 18s trunk passed
          +1 checkstyle 0m 11s trunk passed
          +1 mvnsite 0m 18s trunk passed
          +1 findbugs 0m 26s trunk passed
          +1 javadoc 0m 15s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 15s the patch passed
          -1 javac 0m 15s hadoop-tools_hadoop-azure-datalake generated 4 new + 5 unchanged - 6 fixed = 9 total (was 11)
          -0 checkstyle 0m 13s hadoop-tools/hadoop-azure-datalake: The patch generated 4 new + 16 unchanged - 4 fixed = 20 total (was 20)
          +1 mvnsite 0m 16s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 32s hadoop-tools/hadoop-azure-datalake generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 10s the patch passed
                Other Tests
          +1 unit 3m 32s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          24m 33s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-azure-datalake
            org.apache.hadoop.fs.adl.AdlFileStatus doesn't override org.apache.hadoop.fs.FileStatus.equals(Object) At AdlFileStatus.java:At AdlFileStatus.java:[line 1]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14730
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880445/HADOOP-14730.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cf3064ee18ef 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f44b349
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure-datalake.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/testReport/
          modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 2s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 11s trunk passed +1 mvnsite 0m 18s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 15s trunk passed       Patch Compile Tests +1 mvninstall 0m 17s the patch passed +1 compile 0m 15s the patch passed -1 javac 0m 15s hadoop-tools_hadoop-azure-datalake generated 4 new + 5 unchanged - 6 fixed = 9 total (was 11) -0 checkstyle 0m 13s hadoop-tools/hadoop-azure-datalake: The patch generated 4 new + 16 unchanged - 4 fixed = 20 total (was 20) +1 mvnsite 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 32s hadoop-tools/hadoop-azure-datalake generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 10s the patch passed       Other Tests +1 unit 3m 32s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 24m 33s Reason Tests FindBugs module:hadoop-tools/hadoop-azure-datalake   org.apache.hadoop.fs.adl.AdlFileStatus doesn't override org.apache.hadoop.fs.FileStatus.equals(Object) At AdlFileStatus.java:At AdlFileStatus.java: [line 1] Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14730 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880445/HADOOP-14730.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cf3064ee18ef 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f44b349 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure-datalake.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12960/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Leaning towards v002 for its simplicity.

          Show
          jzhuge John Zhuge added a comment - Leaning towards v002 for its simplicity.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          +1 for v004, mainly to not break 2.x linkage. Having a FileStatus shim layer to continue support linkage with 2.x is better approach.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - +1 for v004, mainly to not break 2.x linkage. Having a FileStatus shim layer to continue support linkage with 2.x is better approach.
          Hide
          jzhuge John Zhuge added a comment - - edited

          v004 looks good. Nit: AdlFileStatus#AdlFileStatus() is unused. The findbugs warning probably should be addressed: AdlFileStatus does not override equals.

          Show
          jzhuge John Zhuge added a comment - - edited v004 looks good. Nit: AdlFileStatus#AdlFileStatus() is unused. The findbugs warning probably should be addressed: AdlFileStatus does not override equals.
          Hide
          jzhuge John Zhuge added a comment -

          Verified v004 did work in this use case: 3.0.0b1 hadoop-azure-datalake jar dropped into a 2.9.0 cluster. However, is the use case supported? Is the maintenance cost over benefit?

          Andrew Wang Do we support this kind of in-place upgrade of a jar from 2.x to 3.x? Considering hadoop-azure-datalake is relatively independent.

          During future dev, how can we quickly tell which common API only exists in 3.x?

          How can we automatically test in-place upgrade of a jar?

          The inheritance of AdlFileStatus from FileStatus introduces tight coupling between these 2 classes, although AdlFileStatus is pretty simple at this point.

          Show
          jzhuge John Zhuge added a comment - Verified v004 did work in this use case: 3.0.0b1 hadoop-azure-datalake jar dropped into a 2.9.0 cluster. However, is the use case supported? Is the maintenance cost over benefit? Andrew Wang Do we support this kind of in-place upgrade of a jar from 2.x to 3.x? Considering hadoop-azure-datalake is relatively independent. During future dev, how can we quickly tell which common API only exists in 3.x? How can we automatically test in-place upgrade of a jar? The inheritance of AdlFileStatus from FileStatus introduces tight coupling between these 2 classes, although AdlFileStatus is pretty simple at this point.
          Hide
          andrew.wang Andrew Wang added a comment -

          We don't require ABI compatibility between Hadoop 2 and 3, and I don't think mixing versions of the Hadoop JARs on a client classpath is supported either.

          However, if it's easy to support and useful, we might as well.

          Show
          andrew.wang Andrew Wang added a comment - We don't require ABI compatibility between Hadoop 2 and 3, and I don't think mixing versions of the Hadoop JARs on a client classpath is supported either. However, if it's easy to support and useful, we might as well.
          Hide
          chris.douglas Chris Douglas added a comment -

          Verified v004 did work in this use case: 3.0.0b1 hadoop-azure-datalake jar dropped into a 2.9.0 cluster. However, is the use case supported? Is the maintenance cost over benefit?

          Thanks for testing this, John. I'll fix the other issues and upload a new patch.

          The inheritance of AdlFileStatus from FileStatus introduces tight coupling between these 2 classes, although AdlFileStatus is pretty simple at this point.

          It's hard to do better, unfortunately. Particularly since we deprecated, but still support the Writable behavior in 3.x.

          However, if it's easy to support and useful, we might as well.

          If it's linked against a 2.x client, this will drop the hasAcl field using Writable serialization. That... may be fixable, but FsPermissionExtension looks like it discards it, anyway. I'd ignore this case.

          Maintaining this kind of backward-compatibility is ultimately not viable, but we can sustain the complexity added by v004. I'll fix the issues John Zhuge pointed out and we can wrap this up.

          Show
          chris.douglas Chris Douglas added a comment - Verified v004 did work in this use case: 3.0.0b1 hadoop-azure-datalake jar dropped into a 2.9.0 cluster. However, is the use case supported? Is the maintenance cost over benefit? Thanks for testing this, John. I'll fix the other issues and upload a new patch. The inheritance of AdlFileStatus from FileStatus introduces tight coupling between these 2 classes, although AdlFileStatus is pretty simple at this point. It's hard to do better, unfortunately. Particularly since we deprecated, but still support the Writable behavior in 3.x. However, if it's easy to support and useful, we might as well. If it's linked against a 2.x client, this will drop the hasAcl field using Writable serialization. That... may be fixable, but FsPermissionExtension looks like it discards it, anyway. I'd ignore this case. Maintaining this kind of backward-compatibility is ultimately not viable, but we can sustain the complexity added by v004. I'll fix the issues John Zhuge pointed out and we can wrap this up.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 14m 38s trunk passed
          +1 compile 0m 16s trunk passed
          +1 checkstyle 0m 12s trunk passed
          +1 mvnsite 0m 19s trunk passed
          +1 findbugs 0m 25s trunk passed
          +1 javadoc 0m 13s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 14s the patch passed
          -1 javac 0m 14s hadoop-tools_hadoop-azure-datalake generated 4 new + 5 unchanged - 6 fixed = 9 total (was 11)
          +1 checkstyle 0m 9s hadoop-tools/hadoop-azure-datalake: The patch generated 0 new + 0 unchanged - 20 fixed = 0 total (was 20)
          +1 mvnsite 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 30s the patch passed
          +1 javadoc 0m 10s the patch passed
                Other Tests
          +1 unit 3m 32s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          22m 54s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14730
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880712/HADOOP-14730.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 57ecbf3ac5ae 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / adb84f3
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12969/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12969/testReport/
          modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12969/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 38s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 19s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 13s trunk passed       Patch Compile Tests +1 mvninstall 0m 14s the patch passed +1 compile 0m 14s the patch passed -1 javac 0m 14s hadoop-tools_hadoop-azure-datalake generated 4 new + 5 unchanged - 6 fixed = 9 total (was 11) +1 checkstyle 0m 9s hadoop-tools/hadoop-azure-datalake: The patch generated 0 new + 0 unchanged - 20 fixed = 0 total (was 20) +1 mvnsite 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 10s the patch passed       Other Tests +1 unit 3m 32s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 22m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14730 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880712/HADOOP-14730.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 57ecbf3ac5ae 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / adb84f3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12969/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12969/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12969/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment - - edited

          Thanks Chris Douglas. I am ok if both Vishwajeet Dusane and you found 2.x linkage useful.

          AdlFileStatus#AdlFileStatus() is not used.
          AdlFileStatus#hashCode is probably ok not to involve hasAcl, don't you think equals should return false when hasAcl are different in 2 objects?

          A typical implementation of these 2 methods may look like this:

            @Override
            public boolean equals(Object o) {
              if (this == o) return true;
              if (o == null || getClass() != o.getClass()) return false;
              if (!super.equals(o)) return false;
              AdlFileStatus that = (AdlFileStatus) o;
              return hasAcl == that.hasAcl;
            }
          
            @Override
            public int hashCode() {
              return Objects.hash(super.hashCode(), hasAcl);
            }
          
          Show
          jzhuge John Zhuge added a comment - - edited Thanks Chris Douglas . I am ok if both Vishwajeet Dusane and you found 2.x linkage useful. AdlFileStatus#AdlFileStatus() is not used. AdlFileStatus#hashCode is probably ok not to involve hasAcl , don't you think equals should return false when hasAcl are different in 2 objects? A typical implementation of these 2 methods may look like this: @Override public boolean equals(Object o) { if (this == o) return true; if (o == null || getClass() != o.getClass()) return false; if (!super.equals(o)) return false; AdlFileStatus that = (AdlFileStatus) o; return hasAcl == that.hasAcl; } @Override public int hashCode() { return Objects.hash(super.hashCode(), hasAcl); }
          Hide
          chris.douglas Chris Douglas added a comment -

          AdlFileStatus#AdlFileStatus() is not used.

          Updated patch.

          AdlFileStatus#hashCode is probably ok not to involve hasAcl, don't you think equals should return false when hasAcl are different in 2 objects?

          FileStatus::equals only checks that the Path matches, so ignoring the hasAcl field is consistent.

          Show
          chris.douglas Chris Douglas added a comment - AdlFileStatus#AdlFileStatus() is not used. Updated patch. AdlFileStatus#hashCode is probably ok not to involve hasAcl, don't you think equals should return false when hasAcl are different in 2 objects? FileStatus::equals only checks that the Path matches, so ignoring the hasAcl field is consistent.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 16m 6s trunk passed
          +1 compile 0m 17s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 18s trunk passed
          +1 findbugs 0m 24s trunk passed
          +1 javadoc 0m 13s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 13s the patch passed
          -1 javac 0m 13s hadoop-tools_hadoop-azure-datalake generated 4 new + 5 unchanged - 6 fixed = 9 total (was 11)
          +1 checkstyle 0m 9s hadoop-tools/hadoop-azure-datalake: The patch generated 0 new + 0 unchanged - 20 fixed = 0 total (was 20)
          +1 mvnsite 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 30s the patch passed
          +1 javadoc 0m 10s the patch passed
                Other Tests
          +1 unit 3m 29s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 13s The patch does not generate ASF License warnings.
          24m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14730
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880731/HADOOP-14730.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2dc8cc7f48d8 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bc20680
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12971/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12971/testReport/
          modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12971/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 6s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 18s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 13s trunk passed       Patch Compile Tests +1 mvninstall 0m 14s the patch passed +1 compile 0m 13s the patch passed -1 javac 0m 13s hadoop-tools_hadoop-azure-datalake generated 4 new + 5 unchanged - 6 fixed = 9 total (was 11) +1 checkstyle 0m 9s hadoop-tools/hadoop-azure-datalake: The patch generated 0 new + 0 unchanged - 20 fixed = 0 total (was 20) +1 mvnsite 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 10s the patch passed       Other Tests +1 unit 3m 29s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 13s The patch does not generate ASF License warnings. 24m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14730 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880731/HADOOP-14730.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2dc8cc7f48d8 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bc20680 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12971/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12971/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12971/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          +1 for 006. Thanks for the patience and the great work.

          javac warnings are expected.

          Show
          jzhuge John Zhuge added a comment - +1 for 006. Thanks for the patience and the great work. javac warnings are expected.
          Hide
          jzhuge John Zhuge added a comment -

          Took a crack at the jira summary.

          Show
          jzhuge John Zhuge added a comment - Took a crack at the jira summary.
          Hide
          chris.douglas Chris Douglas added a comment -

          Thanks for the review, John Zhuge. I committed this.

          Show
          chris.douglas Chris Douglas added a comment - Thanks for the review, John Zhuge . I committed this.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks a lot Chris Douglas and John Zhuge.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks a lot Chris Douglas and John Zhuge .

            People

            • Assignee:
              chris.douglas Chris Douglas
              Reporter:
              vishwajeet.dusane Vishwajeet Dusane
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development