Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3609

libhdfs: don't force the URI to look like hdfs://hostname:port

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: libhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, libhdfs forces the URI to look like hdfs://<hostname>:port.

      For configurations like HA or federation this is not ideal.

      1. HDFS-3609.001.patch
        4 kB
        Colin Patrick McCabe
      2. HDFS-3609.002.patch
        4 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536119/HDFS-3609.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2796//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2796//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536119/HDFS-3609.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2796//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2796//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Patch looks good

          Nit: what you're calling "URI prefix"/"location"/"protocol type" in the comments is called the "scheme" in URI lingo.

          Testing? Eg confirmed you can run libhdfs against an HA config (ie one w/o a port) now?

          The test failures here are obviously unrelated.

          Show
          Eli Collins added a comment - Patch looks good Nit: what you're calling "URI prefix"/"location"/"protocol type" in the comments is called the "scheme" in URI lingo. Testing? Eg confirmed you can run libhdfs against an HA config (ie one w/o a port) now? The test failures here are obviously unrelated.
          Hide
          Colin Patrick McCabe added a comment -

          I've verified I can connect without an explicit port, using hdfs_test. There will be an opportunity to add more unit tests once HDFS-3606 is in.

          Show
          Colin Patrick McCabe added a comment - I've verified I can connect without an explicit port, using hdfs_test. There will be an opportunity to add more unit tests once HDFS-3606 is in.
          Hide
          Colin Patrick McCabe added a comment -
          • refer to 'hdfs://' as a 'scheme' rather than a 'prefix'
          Show
          Colin Patrick McCabe added a comment - refer to 'hdfs://' as a 'scheme' rather than a 'prefix'
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536315/HDFS-3609.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2810//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2810//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536315/HDFS-3609.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2810//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2810//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          +1 lgtm

          Show
          Eli Collins added a comment - +1 lgtm
          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2. Thanks Colin.

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2488 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2488/)
          HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2488 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2488/ ) HDFS-3609 . libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2469 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2469/)
          HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2469 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2469/ ) HDFS-3609 . libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2534 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2534/)
          HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2534 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2534/ ) HDFS-3609 . libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1103 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1103/)
          HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1103 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1103/ ) HDFS-3609 . libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1136/)
          HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1136/ ) HDFS-3609 . libhdfs: don't force the URI to look like hdfs://hostname:port. Contributed by Colin Patrick McCabe (Revision 1361454) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361454 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development