Hadoop Common
  1. Hadoop Common
  2. HADOOP-3429

Increase the buffersize for the streaming parent java process's streams

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Increased the size of the buffer used in the communication between the Java task and the Streaming process to 128KB.

      Description

      We saw improved performance when we increased the buffersize for Pipes (HADOOP-1788). In the streaming case, the buffersize is 8K (default for BufferedOutputStream). We should set that to 128k.

      1. patch-3429.txt
        3 kB
        Amareshwari Sriramadasu

        Activity

        Devaraj Das created issue -
        Amareshwari Sriramadasu made changes -
        Field Original Value New Value
        Attachment patch-3429.txt [ 12382627 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Component/s contrib/streaming [ 12310972 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Devaraj Das made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Release Note Increases the size of the buffersize used in the communication between the Java task and the Streaming process to 128K. Gives performance improvements.
        Robert Chansler made changes -
        Release Note Increases the size of the buffersize used in the communication between the Java task and the Streaming process to 128K. Gives performance improvements. Increased the size of the buffer used in the communication between the Java task and the Streaming process to 128KB.
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s contrib/streaming [ 12310972 ]

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development