Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      HFTP can now serve a specific byte range from a file

      Description

      Support should be similar to http byte-serving.

      1. hdfs-235-3.patch
        27 kB
        Bill Zeller
      2. hdfs-235-2.patch
        27 kB
        Bill Zeller
      3. hdfs-235-1.patch
        26 kB
        Bill Zeller

        Issue Links

          Activity

          Venkatesh Seetharam created issue -
          Owen O'Malley made changes -
          Field Original Value New Value
          Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
          Key HADOOP-5998 HDFS-235
          Component/s dfs [ 12310710 ]
          Fix Version/s 0.20.1 [ 12313866 ]
          Bill Zeller made changes -
          Assignee Bill Zeller [ zeller ]
          Hide
          Bill Zeller added a comment -

          Overview

          The code currently works as follows.

          1. HftpFileSystem::open(path, bufferSize) issues a GET request to, e.g., http://namenode/data/path
          2. On the namenode, /data/path is handled by FileDataServlet. FileDataServlet chooses a datanode (using JspHelper.bestNode) and issues an http redirect response to the datanode (e.g., http://datanode/streamFile?filename=path&... )
          3. /streamFile?filename=path is called on the data node, which is handled by org.apache.hadoop.hdfs.server.namenode.StreamFile. StreamFIle creates a DFSClient and serves the appropriate file.

          To handle range requests, the following can be done:

          • Modify /streamFile to handle range requests
          • Modify the way FileDataServlet chooses a datanode (it should use the block locations in the byte-range being requested, not the block locations for the entire file)
          • Add a method to HftpFileSystem that takes one or more byte range arguments (depending on the answer to the question below)
          • Confirm that when HttpURLConnection follows redirects, it maintains headers. Specifically, the Range header will need to be sent to the datanode after the redirect response comes back from the namenode.

          Question:

          The HTTP spec supports multiple byte-ranges. This returns a multi-part (mime) request (see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.2 )
          This is different from a request that contains a single byte-range, which returns data in the standard format, but with an additional Content-Range header (see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.16 )

          There are three options that I see:
          a) Support only a single byte-range. This makes more sense to me from an API point of view, since we can amend the following:
          HftpFileSystem::open(Path f, int buffersize)

          with

          HftpFileSystem::open(Path f, int buffersize, long begin, long end)

          ...which would read the file f from [begin,end].

          b) Support multiple byte-ranges. This would require ensuring that HttpURLConnection supports mime responses (I don't know if it does). Supporting this would also lead to a more complicated API (something like: )

          HftpFileSystem::open(Path f, int buffersize, List<ByteRange> ranges)

          Also, because open() returns an FSDataInputStream, supporting multiple byte-ranges would either require that reading from the FSDataInputStream would result in reading bytes from different ranges sequentially (requiring the client to figure out where bytes in the input stream begin and end) or changing open() to return a list of input streams corresponding to each byte-range.

          c) We could support multiple byte-ranges in StreamFile, but only support a single byte-range in HftpFileSystem.

          Implementation issues:

          To parse the Range requests, I plan to use a few utility classes included in jetty. Specifically, org.mortbay.jetty.InclusiveByteRange and org.mortbay.util.MultiPartOutputStream (but the latter only if we decide to support multiple byte-ranges). Additionally, the logic used to handle the byte-ranges will be heavily inspired by org.mortbay.jetty.servlet.DefaultServlet::sendData, which is also licensed under Apache 2.

          Show
          Bill Zeller added a comment - Overview The code currently works as follows. HftpFileSystem::open(path, bufferSize) issues a GET request to, e.g., http://namenode/data/path On the namenode, /data/path is handled by FileDataServlet. FileDataServlet chooses a datanode (using JspHelper.bestNode) and issues an http redirect response to the datanode (e.g., http://datanode/streamFile?filename=path& ... ) /streamFile?filename=path is called on the data node, which is handled by org.apache.hadoop.hdfs.server.namenode.StreamFile. StreamFIle creates a DFSClient and serves the appropriate file. To handle range requests, the following can be done: Modify /streamFile to handle range requests Modify the way FileDataServlet chooses a datanode (it should use the block locations in the byte-range being requested, not the block locations for the entire file) Add a method to HftpFileSystem that takes one or more byte range arguments (depending on the answer to the question below) Confirm that when HttpURLConnection follows redirects, it maintains headers. Specifically, the Range header will need to be sent to the datanode after the redirect response comes back from the namenode. Question: The HTTP spec supports multiple byte-ranges. This returns a multi-part (mime) request (see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.2 ) This is different from a request that contains a single byte-range, which returns data in the standard format, but with an additional Content-Range header (see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.16 ) There are three options that I see: a) Support only a single byte-range. This makes more sense to me from an API point of view, since we can amend the following: HftpFileSystem::open(Path f, int buffersize) with HftpFileSystem::open(Path f, int buffersize, long begin, long end) ...which would read the file f from [begin,end] . b) Support multiple byte-ranges. This would require ensuring that HttpURLConnection supports mime responses (I don't know if it does). Supporting this would also lead to a more complicated API (something like: ) HftpFileSystem::open(Path f, int buffersize, List<ByteRange> ranges) Also, because open() returns an FSDataInputStream, supporting multiple byte-ranges would either require that reading from the FSDataInputStream would result in reading bytes from different ranges sequentially (requiring the client to figure out where bytes in the input stream begin and end) or changing open() to return a list of input streams corresponding to each byte-range. c) We could support multiple byte-ranges in StreamFile, but only support a single byte-range in HftpFileSystem. Implementation issues: To parse the Range requests, I plan to use a few utility classes included in jetty. Specifically, org.mortbay.jetty.InclusiveByteRange and org.mortbay.util.MultiPartOutputStream (but the latter only if we decide to support multiple byte-ranges). Additionally, the logic used to handle the byte-ranges will be heavily inspired by org.mortbay.jetty.servlet.DefaultServlet::sendData, which is also licensed under Apache 2.
          Hide
          Tom White added a comment -

          Does this subsume HDFS-269? I think it could if the range end was optional, as it is in HTTP (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35).

          Show
          Tom White added a comment - Does this subsume HDFS-269 ? I think it could if the range end was optional, as it is in HTTP ( http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 ).
          Hide
          Bill Zeller added a comment -

          I believe you're asking if seek() could be called on the FSDataInputStream object returned by HftpFileSystem::open if byte-ranges are implemented. Seek could be called, but it would only allow seeking between the byte-range initially specified when making the open() call. I don't believe byte-ranges could be used to optimize seek, because the seek is happening after the HTTP response returns. This forces seek() to work within the confines of the bytes already requested.

          Show
          Bill Zeller added a comment - I believe you're asking if seek() could be called on the FSDataInputStream object returned by HftpFileSystem::open if byte-ranges are implemented. Seek could be called, but it would only allow seeking between the byte-range initially specified when making the open() call. I don't believe byte-ranges could be used to optimize seek, because the seek is happening after the HTTP response returns. This forces seek() to work within the confines of the bytes already requested.
          Hide
          Doug Cutting added a comment -

          > HftpFileSystem::open(Path f, int buffersize, long begin, long end)

          So this would be an Hftp-specific method. How would, e.g., a mapreduce program that uses the FileSystem API take advantage of this?

          Supporting seek() would be more generic. Perhaps the initial open() call should do nothing, with no HTTP requests made until one attempts to read a file. Then the servlet could accept a start position, but not an end position. Seeking would close any open HTTP connection and open a new one.

          Or perhaps you have a different application need that does not require generic FileSystem access? Can you describe the use case more?

          Show
          Doug Cutting added a comment - > HftpFileSystem::open(Path f, int buffersize, long begin, long end) So this would be an Hftp-specific method. How would, e.g., a mapreduce program that uses the FileSystem API take advantage of this? Supporting seek() would be more generic. Perhaps the initial open() call should do nothing, with no HTTP requests made until one attempts to read a file. Then the servlet could accept a start position, but not an end position. Seeking would close any open HTTP connection and open a new one. Or perhaps you have a different application need that does not require generic FileSystem access? Can you describe the use case more?
          Hide
          Bill Zeller added a comment -

          > Supporting seek() would be more generic. Perhaps the initial open() call should do nothing,
          > with no HTTP requests made until one attempts to read a file. Then the servlet could accept
          > a start position, but not an end position. Seeking would close any open HTTP connection and open a new one.

          Yes, the end position is not required. I believe HftpFileSystem could be easily modified to support the above semantics.

          Show
          Bill Zeller added a comment - > Supporting seek() would be more generic. Perhaps the initial open() call should do nothing, > with no HTTP requests made until one attempts to read a file. Then the servlet could accept > a start position, but not an end position. Seeking would close any open HTTP connection and open a new one. Yes, the end position is not required. I believe HftpFileSystem could be easily modified to support the above semantics.
          Hide
          Bill Zeller added a comment -

          Also, I believe it's unnecessary to support multiple byte-range requests. For example, it should be sufficient to allow a client to ask for bytes 50-100, but not 50-100,60-200,1000-2000 in the same request (the client can always issue additional requests for the other byte ranges).

          Show
          Bill Zeller added a comment - Also, I believe it's unnecessary to support multiple byte-range requests. For example, it should be sufficient to allow a client to ask for bytes 50-100, but not 50-100,60-200,1000-2000 in the same request (the client can always issue additional requests for the other byte ranges).
          Hide
          Robert Chansler added a comment -

          Venkatesh may have an additional perspective, but the principal use case is for data movement across grids with different versions. The objective is to move a subset of file so large that naively copying the whole file is a poor idea. To that end, multiple ranges or extra complexity to allow further repositioning seems unnecessary, and a distraction.

          Show
          Robert Chansler added a comment - Venkatesh may have an additional perspective, but the principal use case is for data movement across grids with different versions. The objective is to move a subset of file so large that naively copying the whole file is a poor idea. To that end, multiple ranges or extra complexity to allow further repositioning seems unnecessary, and a distraction.
          Hide
          Venkatesh Seetharam added a comment -

          Yes, as Rob C says, we want this feature to implement chunk-oriented processing of very large files that result in long tails.

          Show
          Venkatesh Seetharam added a comment - Yes, as Rob C says, we want this feature to implement chunk-oriented processing of very large files that result in long tails.
          Hide
          Bill Zeller added a comment -

          There are three additional javac warnings that are the result of creating a mock class (for testing) that implements an interface (HttpServletResponse) that has three deprecated methods.

          Show
          Bill Zeller added a comment - There are three additional javac warnings that are the result of creating a mock class (for testing) that implements an interface (HttpServletResponse) that has three deprecated methods.
          Bill Zeller made changes -
          Attachment hdfs-235-1.patch [ 12418461 ]
          Hide
          Bill Zeller added a comment -

          HsftpFileSystem has not been modified.

          Show
          Bill Zeller added a comment - HsftpFileSystem has not been modified.
          Bill Zeller made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Added support for byte ranges in HftpFileSystem and support for serving byte ranges of files in StreamFile.
          Affects Version/s 0.21.0 [ 12314046 ]
          Fix Version/s 0.21.0 [ 12314046 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418461/hdfs-235-1.patch
          against trunk revision 810631.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418461/hdfs-235-1.patch against trunk revision 810631. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/console This message is automatically generated.
          Bill Zeller made changes -
          Link This issue is related to HDFS-594 [ HDFS-594 ]
          Hide
          Bill Zeller added a comment -

          Since this patch does not modifed HsftpFileSystem, I've filed HDFS-594. I will not have time to implement this because my internship ends tomorrow.

          Show
          Bill Zeller added a comment - Since this patch does not modifed HsftpFileSystem, I've filed HDFS-594 . I will not have time to implement this because my internship ends tomorrow.
          Bill Zeller made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Bill Zeller made changes -
          Attachment hdfs-235-2.patch [ 12418569 ]
          Bill Zeller made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Bill Zeller added a comment -

          Improved comments a bit.

          Show
          Bill Zeller added a comment - Improved comments a bit.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418569/hdfs-235-2.patch
          against trunk revision 811185.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418569/hdfs-235-2.patch against trunk revision 811185. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          Review patch:

          • Class URLOpener may be better as a nested class within ByteRangeInputStream and needs JavaDoc
          • ByteRangeInputStream::seekToNewSource still has an unresolved question as to return value. I would recommend throwing NotSupportedException since the behavior is non-deterministic and unreliable.
          • Does HftpFileSystem::getNameNode(File)URL need to be public? It's better to make them package private until we have a need to support them as part of the API.
          • Rather than casting the URISyntaxException in getNameNodeURL, you can wrap it an IOException
          • There is quite a bit of commented out code in open. This needs to be removed.
          • TestStreamFile::StrToRanges should start with a lower case s
          Show
          Jakob Homan added a comment - Review patch: Class URLOpener may be better as a nested class within ByteRangeInputStream and needs JavaDoc ByteRangeInputStream::seekToNewSource still has an unresolved question as to return value. I would recommend throwing NotSupportedException since the behavior is non-deterministic and unreliable. Does HftpFileSystem::getNameNode(File)URL need to be public? It's better to make them package private until we have a need to support them as part of the API. Rather than casting the URISyntaxException in getNameNodeURL, you can wrap it an IOException There is quite a bit of commented out code in open. This needs to be removed. TestStreamFile::StrToRanges should start with a lower case s
          Bill Zeller made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Bill Zeller made changes -
          Attachment hdfs-235-3.patch [ 12418671 ]
          Bill Zeller made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Bill Zeller added a comment -

          Addressed Jakob's six issues.

          Show
          Bill Zeller added a comment - Addressed Jakob's six issues.
          Hide
          Jakob Homan added a comment -

          Thanks for the changes. Looks good. +1.

          Show
          Jakob Homan added a comment - Thanks for the changes. Looks good. +1.
          Jakob Homan made changes -
          Hadoop Flags [Reviewed]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418671/hdfs-235-3.patch
          against trunk revision 811493.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418671/hdfs-235-3.patch against trunk revision 811493. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/console This message is automatically generated.
          Suresh Srinivas made changes -
          Release Note Added support for byte ranges in HftpFileSystem and support for serving byte ranges of files in StreamFile. Add support for byte ranges in HftpFileSystem to serve range of bytes from a file.
          Hide
          Suresh Srinivas added a comment -

          I committed this. Thank you Bill.

          Show
          Suresh Srinivas added a comment - I committed this. Thank you Bill.
          Suresh Srinivas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #24 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/24/)
          . Add support for byte ranges in HftpFileSystem to serve range of bytes from a file. Contributed by Bill Zeller.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #24 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/24/ ) . Add support for byte ranges in HftpFileSystem to serve range of bytes from a file. Contributed by Bill Zeller.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/)
          . Add support for byte ranges in HftpFileSystem to serve range of bytes from a file. Contributed by Bill Zeller.

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/ ) . Add support for byte ranges in HftpFileSystem to serve range of bytes from a file. Contributed by Bill Zeller.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The committed patch has quite a few warnings:

          • Enumeration is a raw type. References to generic type Enumeration<E> should be parameterized StreamFile.java src/java/org/apache/hadoop/hdfs/server/namenode line 70 Java Problem
          • The field TestByteRangeInputStream.LOG is never read locally TestByteRangeInputStream.java src/test/hdfs/org/apache/hadoop/hdfs line 108 Java Problem
          • Enumeration is a raw type. References to generic type Enumeration<E> should be parameterized TestStreamFile.java src/test/hdfs/org/apache/hadoop/hdfs/server/namenode line 236 Java Problem
            (there are 10 warnings in TestStreamFile.java)
          Show
          Tsz Wo Nicholas Sze added a comment - The committed patch has quite a few warnings: Enumeration is a raw type. References to generic type Enumeration<E> should be parameterized StreamFile.java src/java/org/apache/hadoop/hdfs/server/namenode line 70 Java Problem The field TestByteRangeInputStream.LOG is never read locally TestByteRangeInputStream.java src/test/hdfs/org/apache/hadoop/hdfs line 108 Java Problem Enumeration is a raw type. References to generic type Enumeration<E> should be parameterized TestStreamFile.java src/test/hdfs/org/apache/hadoop/hdfs/server/namenode line 236 Java Problem (there are 10 warnings in TestStreamFile.java)
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/ )
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
          Robert Chansler made changes -
          Release Note Add support for byte ranges in HftpFileSystem to serve range of bytes from a file. HFTP can now serve a specific byte range from a file
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Bill Zeller
              Reporter:
              Venkatesh Seetharam
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development