Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs
    • Labels:
      None
    • Environment:

      Any environment

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added SFTP filesystem by using the JSch library.
    • Tags:
      sftp filesystem

      Description

      I have implemented a FileSystem that supports SFTP. It uses JSch (http://www.jcraft.com/jsch/) in order to manage SFTP.

      1. HADOOP-5732.008.patch
        44 kB
        ramtin
      2. HADOOP-5732.009.patch
        46 kB
        ramtin
      3. HADOOP-5732.010.patch
        48 kB
        Chris Douglas
      4. HADOOP-5732.patch
        29 kB
        Tom White
      5. HADOOP-5732.patch
        28 kB
        Íñigo Goiri
      6. HADOOP-5732.patch
        28 kB
        Íñigo Goiri
      7. HADOOP-5732.patch
        28 kB
        Íñigo Goiri
      8. HADOOP-5732.patch
        25 kB
        Íñigo Goiri
      9. ivy-for-hadoop-7532.patch
        2 kB
        Steve Loughran
      10. ivy-for-hadoop-7532.patch
        1 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          chris.douglas Chris Douglas added a comment -

          Would you mind posting your changes as a patch, per http://wiki.apache.org/hadoop/HowToContribute ?

          Show
          chris.douglas Chris Douglas added a comment - Would you mind posting your changes as a patch, per http://wiki.apache.org/hadoop/HowToContribute ?
          Hide
          goirix Íñigo Goiri added a comment -

          Here is the patch.

          Show
          goirix Íñigo Goiri added a comment - Here is the patch.
          Hide
          chris.douglas Chris Douglas added a comment -

          Thanks for creating the patch. This looks good; a few comments:

          • Throwing a RuntimeException (SFTPException) from FileSystem methods seems excessive; IOException should suffice.
          • SFTPInputStream could equivalently and more succinctly extend FSInputStream. It can probably be a package-private class.
          • The patch should update the ivy configs if it adds a jsch dependency. I'm sorry, but I have no good references on how to do this.
          • Defaulting "recursive" to true in delete might be too aggressive.
          • Note that delete(Path f) has been deprecated and removed in trunk, so at least two of the delete overrides can be removed
          • Re-thrown exceptions should retain the original cause, not just the message (e.g. rename)
          • Any particular reason for not supporting setting the working directory?
          Show
          chris.douglas Chris Douglas added a comment - Thanks for creating the patch. This looks good; a few comments: Throwing a RuntimeException (SFTPException) from FileSystem methods seems excessive; IOException should suffice. SFTPInputStream could equivalently and more succinctly extend FSInputStream. It can probably be a package-private class. The patch should update the ivy configs if it adds a jsch dependency. I'm sorry, but I have no good references on how to do this. Defaulting "recursive" to true in delete might be too aggressive. Note that delete(Path f) has been deprecated and removed in trunk, so at least two of the delete overrides can be removed Re-thrown exceptions should retain the original cause, not just the message (e.g. rename ) Any particular reason for not supporting setting the working directory?
          Hide
          goirix Íñigo Goiri added a comment -

          Thanks for your comments, I've solved some of the the issues and I would want to comment some of them.

          • Throwing a RuntimeException (SFTPException) from FileSystem methods seems excessive; IOException should suffice.
            It was based on the method used in the FTP FileSystem which used a runtime exception. Should I use IOException then?
            For example if it throws an IOException at "exists" method, the API should be changed, I think that's why they used FTPException. In this case, should I just solve the exception locally? or throwing the SFTPException?
          • SFTPInputStream could equivalently and more succinctly extend FSInputStream. It can probably be a package-private class.
            Done, it extends FSInputStream. Should I modify something in order to do it package-private?
          • The patch should update the ivy configs if it adds a jsch dependency. I'm sorry, but I have no good references on how to do this.
            FTPFileSystem used commons-net, which uses JSch, in theory, the dependency should be automatically resolved. Anyone knows if it is OK or should I add something?
          • Defaulting "recursive" to true in delete might be too aggressive.
            Done.
          • Note that delete(Path f) has been deprecated and removed in trunk, so at least two of the delete overrides can be removed
            I work with hadoop 0.19, which defines this methods, I prefer specifying it as deprecated.
          • Re-thrown exceptions should retain the original cause, not just the message (e.g. rename)
            Done.
          • Any particular reason for not supporting setting the working directory?
            It is the same problem with FTPFileSystem:
            "Directory on the server is changed to the parent directory of the file. The FTP client connection is closed when close() is called on the FSDataInputStream."
          Show
          goirix Íñigo Goiri added a comment - Thanks for your comments, I've solved some of the the issues and I would want to comment some of them. Throwing a RuntimeException (SFTPException) from FileSystem methods seems excessive; IOException should suffice. It was based on the method used in the FTP FileSystem which used a runtime exception. Should I use IOException then? For example if it throws an IOException at "exists" method, the API should be changed, I think that's why they used FTPException. In this case, should I just solve the exception locally? or throwing the SFTPException? SFTPInputStream could equivalently and more succinctly extend FSInputStream. It can probably be a package-private class. Done, it extends FSInputStream. Should I modify something in order to do it package-private? The patch should update the ivy configs if it adds a jsch dependency. I'm sorry, but I have no good references on how to do this. FTPFileSystem used commons-net, which uses JSch, in theory, the dependency should be automatically resolved. Anyone knows if it is OK or should I add something? Defaulting "recursive" to true in delete might be too aggressive. Done. Note that delete(Path f) has been deprecated and removed in trunk, so at least two of the delete overrides can be removed I work with hadoop 0.19, which defines this methods, I prefer specifying it as deprecated. Re-thrown exceptions should retain the original cause, not just the message (e.g. rename) Done. Any particular reason for not supporting setting the working directory? It is the same problem with FTPFileSystem: "Directory on the server is changed to the parent directory of the file. The FTP client connection is closed when close() is called on the FSDataInputStream."
          Hide
          goirix Íñigo Goiri added a comment -

          Some fixing and change in ivy.xml (not sure it is OK, anyone who knows how should ivy be modified?)

          Show
          goirix Íñigo Goiri added a comment - Some fixing and change in ivy.xml (not sure it is OK, anyone who knows how should ivy be modified?)
          Hide
          steve_l Steve Loughran added a comment -

          Patch for ivy that adds a new sftp configuration. There is no jsch dependency here -is it needed?

          Show
          steve_l Steve Loughran added a comment - Patch for ivy that adds a new sftp configuration. There is no jsch dependency here -is it needed?
          Hide
          steve_l Steve Loughran added a comment -

          patch with jsch added in

          Show
          steve_l Steve Loughran added a comment - patch with jsch added in
          Hide
          steve_l Steve Loughran added a comment -

          I've just stuck an ivy.xml that has an explicit sftp configuration, that extends the ftp one and is pulled in for testing. I've added a dependency for jsch 0.1.38. Test for this working: run ant ivy-retrieve and look in build/ivy/lib/Hadoop for the new configuration and the JARs you want

          How do you plan to test this, incidentally? I'd recommend having some test.* properties that, if set, would let you sftp back in to localhost or another named host.

          Show
          steve_l Steve Loughran added a comment - I've just stuck an ivy.xml that has an explicit sftp configuration, that extends the ftp one and is pulled in for testing. I've added a dependency for jsch 0.1.38. Test for this working: run ant ivy-retrieve and look in build/ivy/lib/Hadoop for the new configuration and the JARs you want How do you plan to test this, incidentally? I'd recommend having some test.* properties that, if set, would let you sftp back in to localhost or another named host.
          Hide
          goirix Íñigo Goiri added a comment -

          Thank you for the ivy patch!

          About the tests: I'm new with junit and all this stuff; can you show me the path of a similar test in order to create a new one for the SFTP FS?

          Show
          goirix Íñigo Goiri added a comment - Thank you for the ivy patch! About the tests: I'm new with junit and all this stuff; can you show me the path of a similar test in order to create a new one for the SFTP FS?
          Hide
          steve_l Steve Loughran added a comment -

          Basic Junit is fairly simple: extend TestCase and write a testXXX method for every test you want.

          Functional testing of sftp will be tricker as it should only work if the test runner has been told to do so,

          Have a look at org.apache.hadoop.fs.TestLocalFileSystem to see a filesystem test, and
          org.apache.hadoop.fs.TestLocalFileSystemPermission to see one that uses System.getProperty() to look for specific properties beginning with test. to control its operation -and which skips any attempts to work on windows

          for SFTP, a good test would be to take a filesystem URL as a system property (e.g. test.sftp.url) and then have a series of tests that test out the various filesystem methods, starting with the basic ones (Write that test first, get it working before doing the others)

          testConnect: try to connect to that URL, close it
          testCreateFile: connect, create a file, delete it ,close the filesystem
          testFileExists: as above, check the file exists
          testReadFile: connect, create a file, write to it, close the file, read it in, checksum the data, delete it and close the FS
          testRenameFile: create, rename a file
          testStatFile: test getFileStatus

          Although you can do all of this in one sequence, breaking it up is better for debugging problems -when you see an error report, you can often guess what the problem is

          And some trick ones
          testDeleteNonexistentFile -delete a file that isn't there
          testRenameNonexistentFile
          testRenameOntoExistingFile
          testConcurrentAccess: try to open the same file for writing twice, see what happens
          ...etc.

          Thinking up tests that break the system are the most fun, and the way to reliable code.

          Show
          steve_l Steve Loughran added a comment - Basic Junit is fairly simple: extend TestCase and write a testXXX method for every test you want. Functional testing of sftp will be tricker as it should only work if the test runner has been told to do so, Have a look at org.apache.hadoop.fs.TestLocalFileSystem to see a filesystem test, and org.apache.hadoop.fs.TestLocalFileSystemPermission to see one that uses System.getProperty() to look for specific properties beginning with test. to control its operation -and which skips any attempts to work on windows for SFTP, a good test would be to take a filesystem URL as a system property (e.g. test.sftp.url) and then have a series of tests that test out the various filesystem methods, starting with the basic ones (Write that test first, get it working before doing the others) testConnect: try to connect to that URL, close it testCreateFile: connect, create a file, delete it ,close the filesystem testFileExists: as above, check the file exists testReadFile: connect, create a file, write to it, close the file, read it in, checksum the data, delete it and close the FS testRenameFile: create, rename a file testStatFile: test getFileStatus Although you can do all of this in one sequence, breaking it up is better for debugging problems -when you see an error report, you can often guess what the problem is And some trick ones testDeleteNonexistentFile -delete a file that isn't there testRenameNonexistentFile testRenameOntoExistingFile testConcurrentAccess: try to open the same file for writing twice, see what happens ...etc. Thinking up tests that break the system are the most fun, and the way to reliable code.
          Hide
          chris.douglas Chris Douglas added a comment -

          It was based on the method used in the FTP FileSystem which used a runtime exception. Should I use IOException then? For example if it throws an IOException at "exists" method, the API should be changed, I think that's why they used FTPException. In this case, should I just solve the exception locally? or throwing the SFTPException?

          Yikes; I didn't know it did that. Yes, please throw IOException and not RuntimeException. Applications won't expect the latter.

          Done, it extends FSInputStream. Should I modify something in order to do it package-private?

          Top-level classes without a visibility modifier are package-private, i.e. are not visible to classes outside the package. No other classes should be creating SFTPInputStream instances, right?

          I work with hadoop 0.19, which defines this methods, I prefer specifying it as deprecated.

          I see. As a new feature, the earliest branch to which it should be committed is 0.21, so the @Override annotations should fail. If you'd like to attach patches for 0.19 and/or 0.20 on this issue for others to use in their own deployments, that's great, but the version committed to the mainline should not include these as they should have no callers.

          > Any particular reason for not supporting setting the working directory?

          It is the same problem with FTPFileSystem: "Directory on the server is changed to the parent directory of the file. The FTP client connection is closed when close() is called on the FSDataInputStream."

          That makes sense. Since the working directory can be a Path member in the FileSystem handle, you could make all paths relative to that path instead of the default directory, right? Applications referencing generic filesystems might use the working directory abstraction, which would have unexpected results with this FileSystem. If FTPFileSystem has the same quirk and you've documented it, it's OK if you want to leave it that way, but I'd lean towards supporting it. Applications that deal with generic FileSystems- like DistCp- usually end up creating absolute paths to avoid edge cases like this one, but it would be nice if that were unnecessary. It's up to you.

          How do you plan to test this, incidentally? I'd recommend having some test.* properties that, if set, would let you sftp back in to localhost or another named host.

          The unit tests should not require the user running the tests to have an ssh login on the host machine, neither should it be necessary to configure such a user. If jsch has a clever way to run its unit tests without actually performing these checks- perhaps in its development branch- that would be hugely, hugely preferred. If that's not possible... then like KFS, there's not much to be done to prevent regressions.

          Please create a unified patch with the ivy changes that applies to 0.21. Once that's done, you can submit the patch to automated testing using the "Submit Patch" link to the left.

          Show
          chris.douglas Chris Douglas added a comment - It was based on the method used in the FTP FileSystem which used a runtime exception. Should I use IOException then? For example if it throws an IOException at "exists" method, the API should be changed, I think that's why they used FTPException. In this case, should I just solve the exception locally? or throwing the SFTPException? Yikes; I didn't know it did that. Yes, please throw IOException and not RuntimeException. Applications won't expect the latter. Done, it extends FSInputStream. Should I modify something in order to do it package-private? Top-level classes without a visibility modifier are package-private, i.e. are not visible to classes outside the package. No other classes should be creating SFTPInputStream instances, right? I work with hadoop 0.19, which defines this methods, I prefer specifying it as deprecated. I see. As a new feature, the earliest branch to which it should be committed is 0.21, so the @Override annotations should fail. If you'd like to attach patches for 0.19 and/or 0.20 on this issue for others to use in their own deployments, that's great, but the version committed to the mainline should not include these as they should have no callers. > Any particular reason for not supporting setting the working directory? It is the same problem with FTPFileSystem: "Directory on the server is changed to the parent directory of the file. The FTP client connection is closed when close() is called on the FSDataInputStream." That makes sense. Since the working directory can be a Path member in the FileSystem handle, you could make all paths relative to that path instead of the default directory, right? Applications referencing generic filesystems might use the working directory abstraction, which would have unexpected results with this FileSystem. If FTPFileSystem has the same quirk and you've documented it, it's OK if you want to leave it that way, but I'd lean towards supporting it. Applications that deal with generic FileSystems- like DistCp- usually end up creating absolute paths to avoid edge cases like this one, but it would be nice if that were unnecessary. It's up to you. How do you plan to test this, incidentally? I'd recommend having some test.* properties that, if set, would let you sftp back in to localhost or another named host. The unit tests should not require the user running the tests to have an ssh login on the host machine, neither should it be necessary to configure such a user. If jsch has a clever way to run its unit tests without actually performing these checks- perhaps in its development branch- that would be hugely, hugely preferred. If that's not possible... then like KFS, there's not much to be done to prevent regressions. Please create a unified patch with the ivy changes that applies to 0.21. Once that's done, you can submit the patch to automated testing using the "Submit Patch" link to the left.
          Hide
          goirix Íñigo Goiri added a comment -

          Added ivy files.
          Support for public and private key.
          Bug fixing.

          Show
          goirix Íñigo Goiri added a comment - Added ivy files. Support for public and private key. Bug fixing.
          Hide
          goirix Íñigo Goiri added a comment -

          OK, I've added a patch that already supports ivy and fixes:
          *SFTPException removed
          *SFTPInputStream done private
          *Removed @override from delete; now it is compatible with 0.20

          Open issues:
          *Working directory: would be great to add it.
          *Test: I'm working in the junit test, but it is difficult to deal with authentication with remote sites... anyway, I have added support for public key authentication.

          Show
          goirix Íñigo Goiri added a comment - OK, I've added a patch that already supports ivy and fixes: *SFTPException removed *SFTPInputStream done private *Removed @override from delete; now it is compatible with 0.20 Open issues: *Working directory: would be great to add it. *Test: I'm working in the junit test, but it is difficult to deal with authentication with remote sites... anyway, I have added support for public key authentication.
          Hide
          steve_l Steve Loughran added a comment -

          Chris said
          >The unit tests should not require the user running the tests to have an ssh login on the host machine, neither should it be necessary to >configure such a user. If jsch has a clever way to run its unit tests without actually performing these checks- perhaps in its development >branch- that would be hugely, hugely preferred. If that's not possible... then like KFS, there's not much to be done to prevent regressions.

          We could talk to the jsch author for his input, but I think it's all done with functional tests -that's how ant test <ssh>. If you make it a switch which can be turned on, and you assume that the local user has the ability to log in to localhost with their own username (relying on SSH keys to be set up so that you don't need a password in that situation, localhost is trusted, then functional testing is fairly straightforward. Everyone with a copy of linux should have SSH installed, and it is there for the mac too.

          Show
          steve_l Steve Loughran added a comment - Chris said >The unit tests should not require the user running the tests to have an ssh login on the host machine, neither should it be necessary to >configure such a user. If jsch has a clever way to run its unit tests without actually performing these checks- perhaps in its development >branch- that would be hugely, hugely preferred. If that's not possible... then like KFS, there's not much to be done to prevent regressions. We could talk to the jsch author for his input, but I think it's all done with functional tests -that's how ant test <ssh>. If you make it a switch which can be turned on, and you assume that the local user has the ability to log in to localhost with their own username (relying on SSH keys to be set up so that you don't need a password in that situation, localhost is trusted, then functional testing is fairly straightforward. Everyone with a copy of linux should have SSH installed, and it is there for the mac too.
          Hide
          goirix Íñigo Goiri added a comment -

          One question: should I add a modification in hadoop-default.xml in order to add fs.impl.sftp value?

          Show
          goirix Íñigo Goiri added a comment - One question: should I add a modification in hadoop-default.xml in order to add fs.impl.sftp value?
          Hide
          goirix Íñigo Goiri added a comment -

          It does not support working directory maintainment.

          Show
          goirix Íñigo Goiri added a comment - It does not support working directory maintainment.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12407590/HADOOP-FS-SFTP.patch
          against trunk revision 772960.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 42 javac compiler warnings (more than the trunk's current 2753 warnings).

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407590/HADOOP-FS-SFTP.patch against trunk revision 772960. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 42 javac compiler warnings (more than the trunk's current 2753 warnings). -1 findbugs. The patch appears to cause Findbugs to fail. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/console This message is automatically generated.
          Hide
          goirix Íñigo Goiri added a comment -

          OK, the main parts of the fails are due to indentation; already solved configuring the Eclipse editor.

          The other main problem is the JSch library, it seems it does not find the files. It is strange since it works in my computer. Any idea why the testing is not finding the libraries?

          I submit the patch again with spaces.

          Show
          goirix Íñigo Goiri added a comment - OK, the main parts of the fails are due to indentation; already solved configuring the Eclipse editor. The other main problem is the JSch library, it seems it does not find the files. It is strange since it works in my computer. Any idea why the testing is not finding the libraries? I submit the patch again with spaces.
          Hide
          goirix Íñigo Goiri added a comment -

          Fixed style issues.

          Show
          goirix Íñigo Goiri added a comment - Fixed style issues.
          Hide
          goirix Íñigo Goiri added a comment -

          Fixed style problemes

          Show
          goirix Íñigo Goiri added a comment - Fixed style problemes
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12407698/HADOOP-FS-SFTP.patch
          against trunk revision 772960.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 42 javac compiler warnings (more than the trunk's current 2753 warnings).

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407698/HADOOP-FS-SFTP.patch against trunk revision 772960. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 42 javac compiler warnings (more than the trunk's current 2753 warnings). -1 findbugs. The patch appears to cause Findbugs to fail. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/console This message is automatically generated.
          Hide
          goirix Íñigo Goiri added a comment -

          Adds support for "create" method of the updated FileSystem

          Show
          goirix Íñigo Goiri added a comment - Adds support for "create" method of the updated FileSystem
          Hide
          goirix Íñigo Goiri added a comment -

          I have added support for new FileSystem. What is missing to be included?

          Show
          goirix Íñigo Goiri added a comment - I have added support for new FileSystem. What is missing to be included?
          Hide
          chris.douglas Chris Douglas added a comment -

          This issue resolved as "fixed," but was not committed; if the patch is ready for review, please reopen and mark PA

          Show
          chris.douglas Chris Douglas added a comment - This issue resolved as "fixed," but was not committed; if the patch is ready for review, please reopen and mark PA
          Hide
          goirix Íñigo Goiri added a comment -

          Patch ready for review.

          Show
          goirix Íñigo Goiri added a comment - Patch ready for review.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12409073/HADOOP-5732.patch
          against trunk revision 780777.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 40 javac compiler warnings (more than the trunk's current 2460 warnings).

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12409073/HADOOP-5732.patch against trunk revision 780777. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 40 javac compiler warnings (more than the trunk's current 2460 warnings). -1 findbugs. The patch appears to cause Findbugs to fail. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/console This message is automatically generated.
          Hide
          goirix Íñigo Goiri added a comment -

          Corrected margi n problems but compilation failures due to Jsch still there

          Show
          goirix Íñigo Goiri added a comment - Corrected margi n problems but compilation failures due to Jsch still there
          Hide
          goirix Íñigo Goiri added a comment -

          I don't know why but the patch fails with Jsch dependency... Anybody knows why? It runs for me on my computer; it downloads the dependency thanks to the modified ivy.xml but the automatic testing results... fail.

          Show
          goirix Íñigo Goiri added a comment - I don't know why but the patch fails with Jsch dependency... Anybody knows why? It runs for me on my computer; it downloads the dependency thanks to the modified ivy.xml but the automatic testing results... fail.
          Hide
          tomwhite Tom White added a comment -

          Unfortunately the patch no longer applies. Would you be able to regenerate it please?

          Show
          tomwhite Tom White added a comment - Unfortunately the patch no longer applies. Would you be able to regenerate it please?
          Hide
          goirix Íñigo Goiri added a comment -

          Adapted to new SVN structure.

          Show
          goirix Íñigo Goiri added a comment - Adapted to new SVN structure.
          Hide
          goirix Íñigo Goiri added a comment -

          Updated to the new SVN structure.

          Show
          goirix Íñigo Goiri added a comment - Updated to the new SVN structure.
          Hide
          chris.douglas Chris Douglas added a comment -

          Running patch through hudson

          Show
          chris.douglas Chris Douglas added a comment - Running patch through hudson
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12411419/HADOOP-5732.patch
          against trunk revision 795172.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12411419/HADOOP-5732.patch against trunk revision 795172. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/console This message is automatically generated.
          Hide
          tomwhite Tom White added a comment -

          I've updated the patch so that applies to trunk. As it stands the unit test isn't self-contained. Can this be done by having an embedded SFTP server like TestFTPFileSystem does? (Although see https://issues.apache.org/jira/browse/HDFS-441.)

          If this is not possible then the test should be renamed SFTPFileSystemTest so it isn't run as a part of the automated test suite.

          Ideally, the FileSystem interface contract should also be tested by subclassing FileSystemContractBaseTest and overriding the setUp() method to set the filesystem under test to be SFTPFileSystem. See TestHDFSFileSystemContract in the HDFS project for an example of how to do this. This test checks lots of edge cases and will help flush out common issues with writing FileSystem implementations.

          > should I add a modification in hadoop-default.xml in order to add fs.impl.sftp value?

          Yes please.

          Show
          tomwhite Tom White added a comment - I've updated the patch so that applies to trunk. As it stands the unit test isn't self-contained. Can this be done by having an embedded SFTP server like TestFTPFileSystem does? (Although see https://issues.apache.org/jira/browse/HDFS-441 .) If this is not possible then the test should be renamed SFTPFileSystemTest so it isn't run as a part of the automated test suite. Ideally, the FileSystem interface contract should also be tested by subclassing FileSystemContractBaseTest and overriding the setUp() method to set the filesystem under test to be SFTPFileSystem. See TestHDFSFileSystemContract in the HDFS project for an example of how to do this. This test checks lots of edge cases and will help flush out common issues with writing FileSystem implementations. > should I add a modification in hadoop-default.xml in order to add fs.impl.sftp value? Yes please.
          Hide
          bshi Bo Shi added a comment -

          Íñigo, thanks for the patch, it's saved us a ton of time

          I had to make some adjustments the initialize() to get some things working. In particular, in the last if/else block of initialize(), you have

          conf.set("fs.sftp.password." + host, null);

          which, (in my 0.20.1 backport at least) will throw a null pointer exception since configuration is backed by HashTable which doesn't allow null keys or values in set().

          We use public key authentication (not user and passwords) to connect, so it might be nice to have the username default to $

          {user.name}

          if none is set instead of throwing an IOException.

          With respect to Tom's comments about testing...

          > Can this be done by having an embedded SFTP server

          The MINA folks claim it takes five minutes: http://mina.apache.org/sshd/embedding-sshd-in-5-minutes.html

          I probably won't have much time in the next week or two to work on this but I'm happy to work on it after I get some breathing room.

          Show
          bshi Bo Shi added a comment - Íñigo, thanks for the patch, it's saved us a ton of time I had to make some adjustments the initialize() to get some things working. In particular, in the last if/else block of initialize(), you have conf.set("fs.sftp.password." + host, null); which, (in my 0.20.1 backport at least) will throw a null pointer exception since configuration is backed by HashTable which doesn't allow null keys or values in set(). We use public key authentication (not user and passwords) to connect, so it might be nice to have the username default to $ {user.name} if none is set instead of throwing an IOException. With respect to Tom's comments about testing... > Can this be done by having an embedded SFTP server The MINA folks claim it takes five minutes: http://mina.apache.org/sshd/embedding-sshd-in-5-minutes.html I probably won't have much time in the next week or two to work on this but I'm happy to work on it after I get some breathing room.
          Hide
          bshi Bo Shi added a comment -

          listStatus() appears to cause some trouble:

          Bar.java
           
            public static void main(String [] args) throws Exception { 
              Configuration conf = new Configuration(); 
              conf.addResource(new Path("file:///home/bshi/test.xml")); 
              FileSystem fs = FileSystem.get(new URI("sftp://user:pass@somehost"), conf); 
          
              String rpath = "/home/vmc/archive/new/NEW-20100505.dat"; 
               
              System.out.println(rpath + " exists? " + fs.exists(new Path(rpath))); 
          
              /* Enabling this will cause the program to hang. 
              for(FileStatus s: fs.listStatus(new Path(rpath))) { 
                System.out.println(s); 
              } 
              */ 
              fs.close(); 
            } 
          

          Enabling the program to hang due to the following thread no reclaiming. Quick inspection reveals no obvious places where a connection isn't being closed.

           
          "Connect thread SOMEREMOTEHOST session" prio=10 tid=0x00002aaaf8370800 nid=0x5191 runnable [0x000000004046e000..0x000000004046eb90] 
             java.lang.Thread.State: RUNNABLE 
          at java.net.SocketInputStream.socketRead0(Native Method) 
          at java.net.SocketInputStream.read(SocketInputStream.java:129) 
          at com.jcraft.jsch.IO.getByte(IO.java:82) 
          at com.jcraft.jsch.Session.read(Session.java:795) 
          at com.jcraft.jsch.Session.run(Session.java:1160) 
          at java.lang.Thread.run(Thread.java:619) 
          
             Locked ownable synchronizers: 
          - None 
          

          This behavior is produced against a 0.20 backport, I'll try it with the official patch and against trunk tomorrow.

          Show
          bshi Bo Shi added a comment - listStatus() appears to cause some trouble: Bar.java public static void main( String [] args) throws Exception { Configuration conf = new Configuration(); conf.addResource( new Path( "file: ///home/bshi/test.xml" )); FileSystem fs = FileSystem.get( new URI( "sftp: //user:pass@somehost" ), conf); String rpath = "/home/vmc/archive/ new /NEW-20100505.dat" ; System .out.println(rpath + " exists? " + fs.exists( new Path(rpath))); /* Enabling this will cause the program to hang. for (FileStatus s: fs.listStatus( new Path(rpath))) { System .out.println(s); } */ fs.close(); } Enabling the program to hang due to the following thread no reclaiming. Quick inspection reveals no obvious places where a connection isn't being closed. "Connect thread SOMEREMOTEHOST session" prio=10 tid=0x00002aaaf8370800 nid=0x5191 runnable [0x000000004046e000..0x000000004046eb90] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.jcraft.jsch.IO.getByte(IO.java:82) at com.jcraft.jsch.Session.read(Session.java:795) at com.jcraft.jsch.Session.run(Session.java:1160) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None This behavior is produced against a 0.20 backport, I'll try it with the official patch and against trunk tomorrow.
          Hide
          wnagele Wolfgang Nagele added a comment -

          I have put together an implementation that is using Ganymed SSH as library. The main advantage is that this library has a way better performance in SFTP transfers because it runs multiple data fetches in parallel (specifically noticeable on high latency connections). It's on Github: https://github.com/wnagele/hadoop-filesystem-sftp

          Show
          wnagele Wolfgang Nagele added a comment - I have put together an implementation that is using Ganymed SSH as library. The main advantage is that this library has a way better performance in SFTP transfers because it runs multiple data fetches in parallel (specifically noticeable on high latency connections). It's on Github: https://github.com/wnagele/hadoop-filesystem-sftp
          Hide
          ktham Kevin Tham added a comment -

          I know this issue dates back a few year, but wondering, are there still plans to implement SFTP on top of HDFS? It would be desirable for those who was thinking of using HDFS to replace current SFTP file server.

          Show
          ktham Kevin Tham added a comment - I know this issue dates back a few year, but wondering, are there still plans to implement SFTP on top of HDFS? It would be desirable for those who was thinking of using HDFS to replace current SFTP file server.
          Hide
          ramtinb ramtin added a comment -

          A new patch for SFTP FileSystem based on JSch library.
          Similar to FTP FileSystem and supporting multiple concurrent connections.

          Show
          ramtinb ramtin added a comment - A new patch for SFTP FileSystem based on JSch library. Similar to FTP FileSystem and supporting multiple concurrent connections.
          Hide
          chris.douglas Chris Douglas added a comment -

          Thanks for updating the patch, Ram.

          From the discussion a few years back, we may want to just punt on the unit test. Since it's ignored (and a "correct" one would be difficult to write), there's not a lot of value in committing it.

          Feedback on the patch (not necessarily blocking):

          • The "soft" limit this imposes on the connection pool should be documented
          • SFTPConnectionPool should be package-private
          • Throwing a RuntimeException instead of IOException is excessive, as in SFTPInputStream (this particular case is not correct; close() should be idempotent)
          Show
          chris.douglas Chris Douglas added a comment - Thanks for updating the patch, Ram. From the discussion a few years back, we may want to just punt on the unit test. Since it's ignored (and a "correct" one would be difficult to write), there's not a lot of value in committing it. Feedback on the patch (not necessarily blocking): The "soft" limit this imposes on the connection pool should be documented SFTPConnectionPool should be package-private Throwing a RuntimeException instead of IOException is excessive, as in SFTPInputStream (this particular case is not correct; close() should be idempotent)
          Hide
          ramtinb ramtin added a comment -

          Thanks Chris Douglas for reviewing the patch.
          In the new patch following items are fixed:

          • Using MINA SSHD and implementing the suggested unit tests.
          • Changing SFTPConnectionPool and SFTPInputStream access modifiers.
          • Throwing IOException rather than a RuntimeException(SFTPException).
          Show
          ramtinb ramtin added a comment - Thanks Chris Douglas for reviewing the patch. In the new patch following items are fixed: Using MINA SSHD and implementing the suggested unit tests. Changing SFTPConnectionPool and SFTPInputStream access modifiers. Throwing IOException rather than a RuntimeException(SFTPException) .
          Hide
          elgoiri Inigo Goiri added a comment -

          If I'm not wrong, I understand that you are asking to access HDFS with the SFTP protocol.
          If so, this patch goes the other way around: access SFTP as it was an "HDFS" filesystem.

          Show
          elgoiri Inigo Goiri added a comment - If I'm not wrong, I understand that you are asking to access HDFS with the SFTP protocol. If so, this patch goes the other way around: access SFTP as it was an "HDFS" filesystem.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 27s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 39s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 39s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 5s Tests passed in hadoop-common.
              60m 54s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732924/HADOOP-5732.009.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a815cc1
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7034/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7034/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7034/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 27s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 39s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 39s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 5s Tests passed in hadoop-common.     60m 54s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732924/HADOOP-5732.009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a815cc1 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7034/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7034/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7034/console This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -
          • Fixed SFTPInputStream::close() to be idempotent as required
          • Set exception as cause, rather than StringUtils::stringifyException as message
          • Changed test to use JUnit4 annotations
          • Changed test to use test dir instead of /tmp
          • Added an assumption that skips running the tests on Windows (haven't tested)
          • Added checks to some tests to check the result of SFTP on the local fs

          ramtin, could you look over the patch? While there are other parts of the impl that could be fussed over, if this has been working in production environments then we can leave it alone.

          Show
          chris.douglas Chris Douglas added a comment - Fixed SFTPInputStream::close() to be idempotent as required Set exception as cause, rather than StringUtils::stringifyException as message Changed test to use JUnit4 annotations Changed test to use test dir instead of /tmp Added an assumption that skips running the tests on Windows (haven't tested) Added checks to some tests to check the result of SFTP on the local fs ramtin , could you look over the patch? While there are other parts of the impl that could be fussed over, if this has been working in production environments then we can leave it alone.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 4s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 1m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 15s Tests passed in hadoop-common.
              61m 15s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742285/HADOOP-5732.010.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fe6c1bd
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7068/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7068/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7068/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 4s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 1m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 15s Tests passed in hadoop-common.     61m 15s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742285/HADOOP-5732.010.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fe6c1bd hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7068/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7068/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7068/console This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Inigo, ramtin, do you have time confirm the changes to your patch?

          Show
          chris.douglas Chris Douglas added a comment - Inigo , ramtin , do you have time confirm the changes to your patch?
          Hide
          elgoiri Inigo Goiri added a comment -

          +1 Looks good.

          Show
          elgoiri Inigo Goiri added a comment - +1 Looks good.
          Hide
          chris.douglas Chris Douglas added a comment -

          +1 I committed this. Thanks Ram and Inigo

          Show
          chris.douglas Chris Douglas added a comment - +1 I committed this. Thanks Ram and Inigo
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8125 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8125/)
          HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8125 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8125/ ) HADOOP-5732 . Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #250 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/250/)
          HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #250 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/250/ ) HADOOP-5732 . Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #980 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/980/)
          HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #980 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/980/ ) HADOOP-5732 . Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092) hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #248 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/248/)
          HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #248 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/248/ ) HADOOP-5732 . Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2196 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2196/)
          HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2196 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2196/ ) HADOOP-5732 . Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092) hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2177/)
          HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2177/ ) HADOOP-5732 . Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #238 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/238/)
          HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #238 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/238/ ) HADOOP-5732 . Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java

            People

            • Assignee:
              ramtinb ramtin
              Reporter:
              goirix Íñigo Goiri
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development