Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11762

Enable swift distcp to secure HDFS

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 2.3.0, 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.6.0
        • Fix Version/s: 3.0.0-alpha1
        • Component/s: fs/swift
        • Labels:
          None

          Description

          Even we can use "dfs -put" or "dfs -cp" to move data between swift and secured HDFS, it will be impractical for moving huge amount of data like 10TB or larger.

          Current Hadoop code will result in :"java.lang.IllegalArgumentException: java.net.UnknownHostException: container.swiftdomain"

          Since it does not support token feature in SwiftNativeFileSystem right now, it will be reasonable that we override the "getCanonicalServiceName" method like other filesystem extensions (S3FileSystem, S3AFileSystem)

            Issue Links

            relates to

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-11750 distcp fails if we copy data from swift to secure HDFS

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              +1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12707672/HADOOP-11762.000.patch
              against trunk revision 47782cb.

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 1 new or modified test files.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. There were no new javadoc warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-openstack.

              Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6009//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6009//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707672/HADOOP-11762.000.patch against trunk revision 47782cb. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-openstack. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6009//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6009//console This message is automatically generated.
              Hide
              Permalink
              aw Allen Wittenauer added a comment -

              +1 committed to trunk.

              Thanks!

              Show
              aw Allen Wittenauer added a comment - +1 committed to trunk. Thanks!
              Hide
              Permalink
              airbots Chen He added a comment -

              Thanks, Allen Wittenauer

              Show
              airbots Chen He added a comment - Thanks, Allen Wittenauer
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-trunk-Commit #8195 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8195/)
              HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

              • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
              • hadoop-common-project/hadoop-common/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8195 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8195/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk #994 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/994/)
              HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

              • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
              • hadoop-common-project/hadoop-common/CHANGES.txt
              • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #994 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/994/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #264 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/264/)
              HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

              • hadoop-common-project/hadoop-common/CHANGES.txt
              • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
              • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #264 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/264/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk #2191 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2191/)
              HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

              • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
              • hadoop-common-project/hadoop-common/CHANGES.txt
              • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2191 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2191/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/253/)
              HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

              • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
              • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              • hadoop-common-project/hadoop-common/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/253/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java hadoop-common-project/hadoop-common/CHANGES.txt
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #261 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/261/)
              HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

              • hadoop-common-project/hadoop-common/CHANGES.txt
              • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #261 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/261/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2210 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2210/)
              HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172)

              • hadoop-common-project/hadoop-common/CHANGES.txt
              • hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
              • hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2210 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2210/ ) HADOOP-11762 . Enable swift distcp to secure HDFS (Chen He via aw) (aw: rev 68d1f4bfe863761d3c83b79b17ec1194edca3172) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java

                People

                • Assignee:
                  airbots Chen He
                  Reporter:
                  airbots Chen He
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development