attaching current patch.

Marking "Patch Available"

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3988/
-----------------------------------------------------------

Review request for Flume.

Summary
-------

1) All HDFS actions are now done in async mode
2) If an HDFS append timesout, the file is closed and reopened.
3) Batching is now handled by BucketWriter which was always aware of the batch size.

This addresses bug FLUME-985.
https://issues.apache.org/jira/browse/FLUME-985

Diffs

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 19b2559
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 8a6740f
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-hdfs-sink/pom.xml f27851e
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 45769f6
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5

Diff: https://reviews.apache.org/r/3988/diff

Testing
-------

1) Unit tests were added for close/reopen scenario.
2) All unit tests pass
3) I manually verified this patch improved FlumeNG's behavior when the datanode it's writing to is restarted. In the past FlumeNG had to be restarted, now Flume moves on and starts writing to a new file.

Thanks,

Brock

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3988/#review6220
-----------------------------------------------------------

Ship it!

sorry I didn't look at this earlier.
Looks fine to me. Please see if the code needs to be rebased.

• Prasad

On 2012-02-21 21:51:32, Brock Noland wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/3988/

-----------------------------------------------------------

(Updated 2012-02-21 21:51:32)

Review request for Flume.

Summary

-------

1) All HDFS actions are now done in async mode

2) If an HDFS append timesout, the file is closed and reopened.

3) Batching is now handled by BucketWriter which was always aware of the batch size.

This addresses bug FLUME-985.

https://issues.apache.org/jira/browse/FLUME-985

Diffs

-----

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 19b2559

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 8a6740f

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a

flume-ng-sinks/flume-hdfs-sink/pom.xml f27851e

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 45769f6

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5

Diff: https://reviews.apache.org/r/3988/diff

Testing

-------

1) Unit tests were added for close/reopen scenario.

2) All unit tests pass

3) I manually verified this patch improved FlumeNG's behavior when the datanode it's writing to is restarted. In the past FlumeNG had to be restarted, now Flume moves on and starts writing to a new file.

Thanks,

Brock

Rebased patch is attached.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3988/
-----------------------------------------------------------

(Updated 2012-03-23 20:55:21.762184)

Review request for Flume.

Changes
-------

Rebased patch attached. Attaching to JIRA for commit.

Summary
-------

1) All HDFS actions are now done in async mode
2) If an HDFS append timesout, the file is closed and reopened.
3) Batching is now handled by BucketWriter which was always aware of the batch size.

This addresses bug FLUME-985.
https://issues.apache.org/jira/browse/FLUME-985

Diffs (updated)

flume-ng-sinks/flume-hdfs-sink/pom.xml bef2ca7
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 45769f6
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1fdaddd
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 19b2559
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 8a6740f
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java b067c00
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1

Diff: https://reviews.apache.org/r/3988/diff

Testing
-------

1) Unit tests were added for close/reopen scenario.
2) All unit tests pass
3) I manually verified this patch improved FlumeNG's behavior when the datanode it's writing to is restarted. In the past FlumeNG had to be restarted, now Flume moves on and starts writing to a new file.

Thanks,

Brock

Patch committed. Thanks Brock!

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3988/#review6311
-----------------------------------------------------------

Ship it!

+1

• Arvind

On 2012-03-23 20:55:21, Brock Noland wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/3988/

-----------------------------------------------------------

(Updated 2012-03-23 20:55:21)

Review request for Flume.

Summary

-------

1) All HDFS actions are now done in async mode

2) If an HDFS append timesout, the file is closed and reopened.

3) Batching is now handled by BucketWriter which was always aware of the batch size.

This addresses bug FLUME-985.

https://issues.apache.org/jira/browse/FLUME-985

Diffs

-----

flume-ng-sinks/flume-hdfs-sink/pom.xml bef2ca7

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 45769f6

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1fdaddd

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 19b2559

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 8a6740f

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java b067c00

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1

Diff: https://reviews.apache.org/r/3988/diff

Testing

-------

1) Unit tests were added for close/reopen scenario.

2) All unit tests pass

3) I manually verified this patch improved FlumeNG's behavior when the datanode it's writing to is restarted. In the past FlumeNG had to be restarted, now Flume moves on and starts writing to a new file.

Thanks,

Brock

Thanks!

Integrated in flume-trunk #143 (See https://builds.apache.org/job/flume-trunk/143/)
FLUME-985. All HDFS Operations should have a timeout.

(Brock Noland via Arvind Prabhakar) (Revision 1304600)

Result = SUCCESS
arvind : http://svn.apache.org/viewvc/?view=rev&rev=1304600
Files :

• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/pom.xml
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java