Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.6.0
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: fs/swift
    • Labels:
      None

      Description

      Even we can use "dfs -put" or "dfs -cp" to move data between swift and secured HDFS, it will be impractical for moving huge amount of data like 10TB or larger.

      Current Hadoop code will result in :"java.lang.IllegalArgumentException: java.net.UnknownHostException: container.swiftdomain"

      Since it does not support token feature in SwiftNativeFileSystem right now, it will be reasonable that we override the "getCanonicalServiceName" method like other filesystem extensions (S3FileSystem, S3AFileSystem)

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707672/HADOOP-11762.000.patch
          against trunk revision 47782cb.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-openstack.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6009//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6009//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707672/HADOOP-11762.000.patch against trunk revision 47782cb. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-openstack. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6009//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6009//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          +1 committed to trunk.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - +1 committed to trunk. Thanks!
          Hide
          airbots Chen He added a comment -
          Show
          airbots Chen He added a comment - Thanks, Allen Wittenauer
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8195 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8195/)
          HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

          • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8195 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8195/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #994 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/994/)
          HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

          • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #994 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/994/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #264 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/264/)
          HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
          • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #264 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/264/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2191 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2191/)
          HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

          • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2191 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2191/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/253/)
          HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

          • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
          • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/253/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #261 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/261/)
          HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #261 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/261/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2210 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2210/)
          HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
          • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2210 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2210/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java

            People

            • Assignee:
              airbots Chen He
              Reporter:
              airbots Chen He
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development