Issue Details (XML | Word | Printable)

Key: HADOOP-3164
Type: Improvement Improvement
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Raghu Angadi
Reporter: Raghu Angadi
Votes: 0
Watchers: 4
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

Use FileChannel.transferTo() when data is read from DataNode.

Created: 03/Apr/08 07:22 PM   Updated: 08/Jul/09 04:43 PM
Return to search
Component/s: None
Affects Version/s: None
Fix Version/s: 0.18.0

Time Tracking:
Not Specified

File Attachments:
  File Name Date Attached Attached By Size
Text File Licensed for inclusion in ASF works HADOOP-3164.patch 2008-04-23 10:32 PM Raghu Angadi 20 kB
Text File Licensed for inclusion in ASF works HADOOP-3164.patch 2008-04-21 07:45 PM Raghu Angadi 17 kB
Text File Licensed for inclusion in ASF works HADOOP-3164.patch 2008-04-18 09:53 PM Raghu Angadi 16 kB
Text File Licensed for inclusion in ASF works HADOOP-3164.patch 2008-04-08 03:44 AM Raghu Angadi 15 kB
Text File Licensed for inclusion in ASF works HADOOP-3164.patch 2008-04-08 03:29 AM Raghu Angadi 15 kB
Text File Licensed for inclusion in ASF works HADOOP-3164.patch 2008-04-05 12:30 AM Raghu Angadi 14 kB
Issue Links:
Reference
 

Hadoop Flags: Reviewed
Release Note:
Changed data node to use FileChannel.tranferTo() to transfer block data.
Resolution Date: 25/Apr/08 12:10 AM


 Description  « Hide
HADOOP-2312 talks about using FileChannel's transferTo() and transferFrom() in DataNode.

At the time DataNode neither used NIO sockets nor wrote large chunks of contiguous block data to socket. Hadoop 0.17 does both when data is seved to clients (and other datanodes). I am planning to try using transferTo() in the trunk. This might reduce DataNode's cpu by another 50% or more.

Once HADOOP-1702 is committed, we can look into using transferFrom().

 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Raghu Angadi made changes - 03/Apr/08 07:23 PM
Field Original Value New Value
Link This issue relates to HADOOP-2312 [ HADOOP-2312 ]
Raghu Angadi made changes - 04/Apr/08 04:25 AM
Attachment HADOOP-3614.patch [ 12379348 ]
Raghu Angadi made changes - 05/Apr/08 12:30 AM
Attachment HADOOP-3164.patch [ 12379454 ]
Raghu Angadi made changes - 08/Apr/08 03:28 AM
Attachment HADOOP-3614.patch [ 12379348 ]
Raghu Angadi made changes - 08/Apr/08 03:29 AM
Attachment HADOOP-3164.patch [ 12379623 ]
Raghu Angadi made changes - 08/Apr/08 03:44 AM
Attachment HADOOP-3164.patch [ 12379624 ]
Raghu Angadi made changes - 10/Apr/08 11:15 PM
Description
HADOOP-2312 talks about using FileChannel's [{{transferTo()}}|http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html#transferTo(long,%20long,%20java.nio.channels.WritableByteChannel)] and [{{transferFrom()}}|http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html#transferFrom(java.nio.channels.ReadableByteChannel,%20long,%20long)] in DataNode.

At the time DataNode neither used NIO sockets nor wrote large chunks of contiguous block data to socket. Hadoop 0.17 does both when data is seved to clients (and other datanodes). I am planning to try using transferTo() in the trunk. This might reduce DataNode's cpu by another 50% or more.

Once HADOOP-1702 is committed, we can look into using transferFrom().
 HADOOP-2312 talks about using FileChannel's [{{transferTo()}}|http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html#transferTo(long,%20long,%20java.nio.channels.WritableByteChannel)] and [{{transferFrom()}}|http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html#transferFrom(java.nio.channels.ReadableByteChannel,%20long,%20long)] in DataNode.

At the time DataNode neither used NIO sockets nor wrote large chunks of contiguous block data to socket. Hadoop 0.17 does both when data is seved to clients (and other datanodes). I am planning to try using transferTo() in the trunk. This might reduce DataNode's cpu by another 50% or more.

Once HADOOP-1702 is committed, we can look into using transferFrom().
Fix Version/s 0.18.0 [ 12312972 ]
Raghu Angadi made changes - 18/Apr/08 09:53 PM
Attachment HADOOP-3164.patch [ 12380543 ]
Raghu Angadi made changes - 21/Apr/08 05:44 PM
Comment [ > Looks like this is fixed in Java SE 7 (Dolphin)
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6427312.

Correction: the right bug is http://bugs.sun.com/view_bug.do?bug_id=5103988 . ]
Raghu Angadi made changes - 21/Apr/08 07:45 PM
Attachment HADOOP-3164.patch [ 12380635 ]
Raghu Angadi made changes - 23/Apr/08 10:32 PM
Attachment HADOOP-3164.patch [ 12380800 ]
Raghu Angadi made changes - 23/Apr/08 11:45 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Raghu Angadi made changes - 25/Apr/08 12:10 AM
Resolution Fixed [ 1 ]
Hadoop Flags [Reviewed]
Release Note Reduce DataNode CPU usage by using FileChannel.tranferTo(). DataNode takes 5 times less CPU on Linux while serving data. Results may vary on other platforms.
Status Patch Available [ 10002 ] Resolved [ 5 ]
Robert Chansler made changes - 27/Jun/08 09:15 PM
Release Note Reduce DataNode CPU usage by using FileChannel.tranferTo(). DataNode takes 5 times less CPU on Linux while serving data. Results may vary on other platforms.
 Changed data node to use FileChannel.tranferTo() to transfer block data.
Nigel Daley made changes - 22/Aug/08 07:50 PM
Status Resolved [ 5 ] Closed [ 6 ]
Owen O'Malley made changes - 08/Jul/09 04:43 PM
Component/s dfs [ 12310710 ]

Powered by a free Atlassian JIRA open source license for Apache Software Foundation. Try JIRA - bug tracking software for your team.
Atlassian JIRA the Professional Issue Tracker. (Enterprise Edition, Version: 3.13.2-#335) - Bug/feature request - Atlassian news - Contact Administrators