Chukwa
  1. Chukwa
  2. CHUKWA-583

Copying data from inputStream to OuputStream needs appropriate buffer size

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Trivial Trivial
    • Resolution: Unresolved
    • Affects Version/s: 0.4.0
    • Fix Version/s: None
    • Component/s: Data Processors
    • Labels:
      None

      Description

      In the file
      ./chukwa-0.4.0/src/test/org/apache/hadoop/chukwa/validationframework/util/DataOperations.java line: 54-58

      In the function copyFile, the buffer size is fixed as 4096 bytes. With the size of the data varies, the performance can be damaged a lot.

      We need an appropriate buffer size which depends on the size of the data to be copied.

      This is the same as the Appache Bug (https://issues.apache.org/bugzilla/show_bug.cgi?id=32546)

        Activity

        Hide
        Eric Yang added a comment -

        This bug is in the test code, and we don't push large amount of data over the validation framework. Hence, it is a minor bug. Patch to increase the size is welcome.

        Show
        Eric Yang added a comment - This bug is in the test code, and we don't push large amount of data over the validation framework. Hence, it is a minor bug. Patch to increase the size is welcome.
        Hide
        Xiaoming Shi added a comment -

        To copy 10M data with 1024-byte buffer it takes 338,591,180 nano-second, while it takes 105,384,689 nano-second with a 10240-byte buffer.

        Maybe you can check the parent bug report, if you have time.

        Show
        Xiaoming Shi added a comment - To copy 10M data with 1024-byte buffer it takes 338,591,180 nano-second, while it takes 105,384,689 nano-second with a 10240-byte buffer. Maybe you can check the parent bug report, if you have time.
        Hide
        Ari Rabkin added a comment -

        This looks like an autogenerated bug report. I would caution you against this. The code you're flagging is in the test suite. It is NOT performance critical, nor is it run on substantial data volumes. It's probably pointless to change. The human time invested in generating and committing a patch is probably more than the "fix" would ever save.

        Show
        Ari Rabkin added a comment - This looks like an autogenerated bug report. I would caution you against this. The code you're flagging is in the test suite. It is NOT performance critical, nor is it run on substantial data volumes. It's probably pointless to change. The human time invested in generating and committing a patch is probably more than the "fix" would ever save.

          People

          • Assignee:
            Unassigned
            Reporter:
            Xiaoming Shi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development