Would you mind posting your changes as a patch, per http://wiki.apache.org/hadoop/HowToContribute ?

Here is the patch.

Thanks for creating the patch. This looks good; a few comments:

• Throwing a RuntimeException (SFTPException) from FileSystem methods seems excessive; IOException should suffice.
• SFTPInputStream could equivalently and more succinctly extend FSInputStream. It can probably be a package-private class.
• The patch should update the ivy configs if it adds a jsch dependency. I'm sorry, but I have no good references on how to do this.
• Defaulting "recursive" to true in delete might be too aggressive.
• Note that delete(Path f) has been deprecated and removed in trunk, so at least two of the delete overrides can be removed
• Re-thrown exceptions should retain the original cause, not just the message (e.g. rename)
• Any particular reason for not supporting setting the working directory?

Thanks for your comments, I've solved some of the the issues and I would want to comment some of them.

• Throwing a RuntimeException (SFTPException) from FileSystem methods seems excessive; IOException should suffice.
  It was based on the method used in the FTP FileSystem which used a runtime exception. Should I use IOException then?
  For example if it throws an IOException at "exists" method, the API should be changed, I think that's why they used FTPException. In this case, should I just solve the exception locally? or throwing the SFTPException?

• SFTPInputStream could equivalently and more succinctly extend FSInputStream. It can probably be a package-private class.
  Done, it extends FSInputStream. Should I modify something in order to do it package-private?

• The patch should update the ivy configs if it adds a jsch dependency. I'm sorry, but I have no good references on how to do this.
  FTPFileSystem used commons-net, which uses JSch, in theory, the dependency should be automatically resolved. Anyone knows if it is OK or should I add something?

• Defaulting "recursive" to true in delete might be too aggressive.
  Done.

• Note that delete(Path f) has been deprecated and removed in trunk, so at least two of the delete overrides can be removed
  I work with hadoop 0.19, which defines this methods, I prefer specifying it as deprecated.

• Re-thrown exceptions should retain the original cause, not just the message (e.g. rename)
  Done.

• Any particular reason for not supporting setting the working directory?
  It is the same problem with FTPFileSystem:
  "Directory on the server is changed to the parent directory of the file. The FTP client connection is closed when close() is called on the FSDataInputStream."

Some fixing and change in ivy.xml (not sure it is OK, anyone who knows how should ivy be modified?)

Patch for ivy that adds a new sftp configuration. There is no jsch dependency here -is it needed?

patch with jsch added in

I've just stuck an ivy.xml that has an explicit sftp configuration, that extends the ftp one and is pulled in for testing. I've added a dependency for jsch 0.1.38. Test for this working: run ant ivy-retrieve and look in build/ivy/lib/Hadoop for the new configuration and the JARs you want

How do you plan to test this, incidentally? I'd recommend having some test.* properties that, if set, would let you sftp back in to localhost or another named host.

Thank you for the ivy patch!

About the tests: I'm new with junit and all this stuff; can you show me the path of a similar test in order to create a new one for the SFTP FS?

Basic Junit is fairly simple: extend TestCase and write a testXXX method for every test you want.

Functional testing of sftp will be tricker as it should only work if the test runner has been told to do so,

Have a look at org.apache.hadoop.fs.TestLocalFileSystem to see a filesystem test, and
org.apache.hadoop.fs.TestLocalFileSystemPermission to see one that uses System.getProperty() to look for specific properties beginning with test. to control its operation -and which skips any attempts to work on windows

for SFTP, a good test would be to take a filesystem URL as a system property (e.g. test.sftp.url) and then have a series of tests that test out the various filesystem methods, starting with the basic ones (Write that test first, get it working before doing the others)

testConnect: try to connect to that URL, close it
testCreateFile: connect, create a file, delete it ,close the filesystem
testFileExists: as above, check the file exists
testReadFile: connect, create a file, write to it, close the file, read it in, checksum the data, delete it and close the FS
testRenameFile: create, rename a file
testStatFile: test getFileStatus

Although you can do all of this in one sequence, breaking it up is better for debugging problems -when you see an error report, you can often guess what the problem is

And some trick ones
testDeleteNonexistentFile -delete a file that isn't there
testRenameNonexistentFile
testRenameOntoExistingFile
testConcurrentAccess: try to open the same file for writing twice, see what happens
...etc.

Thinking up tests that break the system are the most fun, and the way to reliable code.

It was based on the method used in the FTP FileSystem which used a runtime exception. Should I use IOException then? For example if it throws an IOException at "exists" method, the API should be changed, I think that's why they used FTPException. In this case, should I just solve the exception locally? or throwing the SFTPException?

Yikes; I didn't know it did that. Yes, please throw IOException and not RuntimeException. Applications won't expect the latter.

Done, it extends FSInputStream. Should I modify something in order to do it package-private?

Top-level classes without a visibility modifier are package-private, i.e. are not visible to classes outside the package. No other classes should be creating SFTPInputStream instances, right?

I work with hadoop 0.19, which defines this methods, I prefer specifying it as deprecated.

I see. As a new feature, the earliest branch to which it should be committed is 0.21, so the @Override annotations should fail. If you'd like to attach patches for 0.19 and/or 0.20 on this issue for others to use in their own deployments, that's great, but the version committed to the mainline should not include these as they should have no callers.

> Any particular reason for not supporting setting the working directory?

It is the same problem with FTPFileSystem: "Directory on the server is changed to the parent directory of the file. The FTP client connection is closed when close() is called on the FSDataInputStream."

That makes sense. Since the working directory can be a Path member in the FileSystem handle, you could make all paths relative to that path instead of the default directory, right? Applications referencing generic filesystems might use the working directory abstraction, which would have unexpected results with this FileSystem. If FTPFileSystem has the same quirk and you've documented it, it's OK if you want to leave it that way, but I'd lean towards supporting it. Applications that deal with generic FileSystems- like DistCp- usually end up creating absolute paths to avoid edge cases like this one, but it would be nice if that were unnecessary. It's up to you.

How do you plan to test this, incidentally? I'd recommend having some test.* properties that, if set, would let you sftp back in to localhost or another named host.

The unit tests should not require the user running the tests to have an ssh login on the host machine, neither should it be necessary to configure such a user. If jsch has a clever way to run its unit tests without actually performing these checks- perhaps in its development branch- that would be hugely, hugely preferred. If that's not possible... then like KFS, there's not much to be done to prevent regressions.

Please create a unified patch with the ivy changes that applies to 0.21. Once that's done, you can submit the patch to automated testing using the "Submit Patch" link to the left.

Added ivy files.
Support for public and private key.
Bug fixing.

OK, I've added a patch that already supports ivy and fixes:
*SFTPException removed
*SFTPInputStream done private
*Removed @override from delete; now it is compatible with 0.20

Open issues:
*Working directory: would be great to add it.
*Test: I'm working in the junit test, but it is difficult to deal with authentication with remote sites... anyway, I have added support for public key authentication.

Chris said
>The unit tests should not require the user running the tests to have an ssh login on the host machine, neither should it be necessary to >configure such a user. If jsch has a clever way to run its unit tests without actually performing these checks- perhaps in its development >branch- that would be hugely, hugely preferred. If that's not possible... then like KFS, there's not much to be done to prevent regressions.

We could talk to the jsch author for his input, but I think it's all done with functional tests -that's how ant test <ssh>. If you make it a switch which can be turned on, and you assume that the local user has the ability to log in to localhost with their own username (relying on SSH keys to be set up so that you don't need a password in that situation, localhost is trusted, then functional testing is fairly straightforward. Everyone with a copy of linux should have SSH installed, and it is there for the mac too.

One question: should I add a modification in hadoop-default.xml in order to add fs.impl.sftp value?

It does not support working directory maintainment.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407590/HADOOP-FS-SFTP.patch
against trunk revision 772960.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 42 javac compiler warnings (more than the trunk's current 2753 warnings).

-1 findbugs. The patch appears to cause Findbugs to fail.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/315/console

This message is automatically generated.

OK, the main parts of the fails are due to indentation; already solved configuring the Eclipse editor.

The other main problem is the JSch library, it seems it does not find the files. It is strange since it works in my computer. Any idea why the testing is not finding the libraries?

I submit the patch again with spaces.

Fixed style issues.

Fixed style problemes

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407698/HADOOP-FS-SFTP.patch
against trunk revision 772960.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 42 javac compiler warnings (more than the trunk's current 2753 warnings).

-1 findbugs. The patch appears to cause Findbugs to fail.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/317/console

This message is automatically generated.

Adds support for "create" method of the updated FileSystem

I have added support for new FileSystem. What is missing to be included?

This issue resolved as "fixed," but was not committed; if the patch is ready for review, please reopen and mark PA

Patch ready for review.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409073/HADOOP-5732.patch
against trunk revision 780777.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 40 javac compiler warnings (more than the trunk's current 2460 warnings).

-1 findbugs. The patch appears to cause Findbugs to fail.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/446/console

This message is automatically generated.

Corrected margi n problems but compilation failures due to Jsch still there

I don't know why but the patch fails with Jsch dependency... Anybody knows why? It runs for me on my computer; it downloads the dependency thanks to the modified ivy.xml but the automatic testing results... fail.

Unfortunately the patch no longer applies. Would you be able to regenerate it please?

Adapted to new SVN structure.

Updated to the new SVN structure.

Running patch through hudson

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12411419/HADOOP-5732.patch
against trunk revision 795172.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 findbugs. The patch appears to cause Findbugs to fail.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/581/console

This message is automatically generated.

I've updated the patch so that applies to trunk. As it stands the unit test isn't self-contained. Can this be done by having an embedded SFTP server like TestFTPFileSystem does? (Although see https://issues.apache.org/jira/browse/HDFS-441.)

If this is not possible then the test should be renamed SFTPFileSystemTest so it isn't run as a part of the automated test suite.

Ideally, the FileSystem interface contract should also be tested by subclassing FileSystemContractBaseTest and overriding the setUp() method to set the filesystem under test to be SFTPFileSystem. See TestHDFSFileSystemContract in the HDFS project for an example of how to do this. This test checks lots of edge cases and will help flush out common issues with writing FileSystem implementations.

> should I add a modification in hadoop-default.xml in order to add fs.impl.sftp value?

Yes please.

Íñigo, thanks for the patch, it's saved us a ton of time

I had to make some adjustments the initialize() to get some things working. In particular, in the last if/else block of initialize(), you have

conf.set("fs.sftp.password." + host, null);

which, (in my 0.20.1 backport at least) will throw a null pointer exception since configuration is backed by HashTable which doesn't allow null keys or values in set().

We use public key authentication (not user and passwords) to connect, so it might be nice to have the username default to $

if none is set instead of throwing an IOException.

With respect to Tom's comments about testing...

> Can this be done by having an embedded SFTP server

The MINA folks claim it takes five minutes: http://mina.apache.org/sshd/embedding-sshd-in-5-minutes.html

I probably won't have much time in the next week or two to work on this but I'm happy to work on it after I get some breathing room.

listStatus() appears to cause some trouble:

 
  public static void main(String [] args) throws Exception { 
    Configuration conf = new Configuration(); 
    conf.addResource(new Path("file:///home/bshi/test.xml")); 
    FileSystem fs = FileSystem.get(new URI("sftp://user:pass@somehost"), conf); 

    String rpath = "/home/vmc/archive/new/NEW-20100505.dat"; 
     
    System.out.println(rpath + " exists? " + fs.exists(new Path(rpath))); 

    /* Enabling this will cause the program to hang. 
    for(FileStatus s: fs.listStatus(new Path(rpath))) { 
      System.out.println(s); 
    } 
    */ 
    fs.close(); 
  } 

Enabling the program to hang due to the following thread no reclaiming. Quick inspection reveals no obvious places where a connection isn't being closed.

 
"Connect thread SOMEREMOTEHOST session" prio=10 tid=0x00002aaaf8370800 nid=0x5191 runnable [0x000000004046e000..0x000000004046eb90] 
   java.lang.Thread.State: RUNNABLE 
at java.net.SocketInputStream.socketRead0(Native Method) 
at java.net.SocketInputStream.read(SocketInputStream.java:129) 
at com.jcraft.jsch.IO.getByte(IO.java:82) 
at com.jcraft.jsch.Session.read(Session.java:795) 
at com.jcraft.jsch.Session.run(Session.java:1160) 
at java.lang.Thread.run(Thread.java:619) 

   Locked ownable synchronizers: 
- None 

This behavior is produced against a 0.20 backport, I'll try it with the official patch and against trunk tomorrow.

I have put together an implementation that is using Ganymed SSH as library. The main advantage is that this library has a way better performance in SFTP transfers because it runs multiple data fetches in parallel (specifically noticeable on high latency connections). It's on Github: https://github.com/wnagele/hadoop-filesystem-sftp

I know this issue dates back a few year, but wondering, are there still plans to implement SFTP on top of HDFS? It would be desirable for those who was thinking of using HDFS to replace current SFTP file server.

A new patch for SFTP FileSystem based on JSch library.
Similar to FTP FileSystem and supporting multiple concurrent connections.

Thanks for updating the patch, Ram.

From the discussion a few years back, we may want to just punt on the unit test. Since it's ignored (and a "correct" one would be difficult to write), there's not a lot of value in committing it.

Feedback on the patch (not necessarily blocking):

• The "soft" limit this imposes on the connection pool should be documented
• SFTPConnectionPool should be package-private
• Throwing a RuntimeException instead of IOException is excessive, as in SFTPInputStream (this particular case is not correct; close() should be idempotent)

Thanks Chris Douglas for reviewing the patch.
In the new patch following items are fixed:

• Using MINA SSHD and implementing the suggested unit tests.
• Changing SFTPConnectionPool and SFTPInputStream access modifiers.
• Throwing IOException rather than a RuntimeException(SFTPException).

If I'm not wrong, I understand that you are asking to access HDFS with the SFTP protocol.
If so, this patch goes the other way around: access SFTP as it was an "HDFS" filesystem.

-1 overall

This message was automatically generated.

• Fixed SFTPInputStream::close() to be idempotent as required
• Set exception as cause, rather than StringUtils::stringifyException as message
• Changed test to use JUnit4 annotations
• Changed test to use test dir instead of /tmp
• Added an assumption that skips running the tests on Windows (haven't tested)
• Added checks to some tests to check the result of SFTP on the local fs

ramtin, could you look over the patch? While there are other parts of the impl that could be fussed over, if this has been working in production environments then we can leave it alone.

+1 overall

This message was automatically generated.

Inigo, ramtin, do you have time confirm the changes to your patch?

+1 Looks good.

+1 I committed this. Thanks Ram and Inigo

FAILURE: Integrated in Hadoop-trunk-Commit #8125 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8125/)
HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
• hadoop-project/pom.xml
• hadoop-common-project/hadoop-common/pom.xml
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #250 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/250/)
HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
• hadoop-project/pom.xml
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
• hadoop-common-project/hadoop-common/pom.xml
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java

FAILURE: Integrated in Hadoop-Yarn-trunk #980 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/980/)
HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

• hadoop-common-project/hadoop-common/pom.xml
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
• hadoop-project/pom.xml
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
• hadoop-common-project/hadoop-common/CHANGES.txt

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #248 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/248/)
HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
• hadoop-project/pom.xml
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-common-project/hadoop-common/pom.xml
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2196 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2196/)
HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

• hadoop-common-project/hadoop-common/pom.xml
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
• hadoop-project/pom.xml
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
• hadoop-common-project/hadoop-common/CHANGES.txt

FAILURE: Integrated in Hadoop-Hdfs-trunk #2177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2177/)
HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
• hadoop-common-project/hadoop-common/pom.xml
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
• hadoop-project/pom.xml
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #238 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/238/)
HADOOP-5732. Add SFTPFileSystem. Contributed by Ramtin Boustani and Inigo Goiri (cdouglas: rev 559425dcb9302861c3c28b759492a68f8d597092)

• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPConnectionPool.java
• hadoop-project/pom.xml
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java
• hadoop-common-project/hadoop-common/pom.xml
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/package-info.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java

I am using hadoop 2.8 and seeing below exception in my mapreduce job. What would be the reason for it? Is there any work around?

Failed to initialize batch source 'FTP' with the error: No FileSystem for scheme: sftp. Please review your pipeline configuration and check the system logs for more details.
java.io.IOException: No FileSystem for scheme: sftp
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2798) ~[org.apache.hadoop.hadoop-common-2.8.0.jar:na]
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2809) ~[org.apache.hadoop.hadoop-common-2.8.0.jar:na]
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[org.apache.hadoop.hadoop-common-2.8.0.jar:na]
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) ~[org.apache.hadoop.hadoop-common-2.8.0.jar:na]
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) ~[org.apache.hadoop.hadoop-common-2.8.0.jar:na]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) ~[org.apache.hadoop.hadoop-common-2.8.0.jar:na]

vinisha, I'd say you need to set fs.AbstractFileSystem.sftp.impl or fs.sftp.impl (not sure which one) to org.apache.hadoop.fs.sftp.SFTPFileSystem in core-site.xml.

I have both configured on Hadoop 2.8.0:

fs.AbstractFileSystem.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem
fs.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem

I get the following error:

org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Seek not supported
	at org.apache.hadoop.fs.sftp.SFTPInputStream.seek(SFTPInputStream.java:66)
	at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
	at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:114)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:181)
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)

By looking at the code, it looks like the seek method is unsupported. Not sure what I am missing:
https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java#L66

Sergey Bahchissaraitsev, this is not implemented but I found somebody that had implemented a workaround:
https://github.com/ind9/hadoop-fs-sftp/commit/9c5805d45d8bf2eb7823db3680195972239ef39d
Even with unit test:
https://github.com/ind9/hadoop-fs-sftp/commit/7315fda25db81c2be2a7dfb5ffa1d30d22760bec
Actually the author of those have a couple fixes that we should add here.
It would be great if he could create a JIRA with those improvements but not sure he is around here.
I'll try to contact him.