Chukwa
  1. Chukwa
  2. CHUKWA-313

Remove the 10 sec ack from SeqWriter

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.2.0, 0.3.0
    • Fix Version/s: 0.2.0, 0.3.0
    • Component/s: None
    • Labels:
      None

      Description

      SeqWriter.add is waiting 10 sec in a worse case before calling a flush. This was there to guarantee that the data has been written to HDFS before returning the ack.
      It used to work well with 0.18 but in 0.20 the flush is a nohup so there's no point in waiting 10 sec for nothing.

      The real flush will give us the ability to write redo logs directly to HDFS so the ack implementation will have to change anyway.
      So I vote for removing the 10 sec ack from the add method.

      1. CHUKWA-313.patch
        11 kB
        Jerome Boulon

        Issue Links

          Activity

          Jerome Boulon created issue -
          Hide
          Eric Yang added a comment -

          +1 on removing flush.

          Show
          Eric Yang added a comment - +1 on removing flush.
          Jerome Boulon made changes -
          Field Original Value New Value
          Link This issue incorporates CHUKWA-303 [ CHUKWA-303 ]
          Hide
          Jerome Boulon added a comment -
          • Remove the ClientAck since flush is a noHup operation in H.20
          • Refractor the rotate mechanism
          • Compute TimePeriod once an hour
          Show
          Jerome Boulon added a comment - Remove the ClientAck since flush is a noHup operation in H.20 Refractor the rotate mechanism Compute TimePeriod once an hour
          Jerome Boulon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.2.0 [ 12313934 ]
          Affects Version/s 0.3.0 [ 12314032 ]
          Fix Version/s 0.2.0 [ 12313934 ]
          Fix Version/s 0.3.0 [ 12314032 ]
          Jerome Boulon made changes -
          Attachment CHUKWA-313.patch [ 12411166 ]
          Hide
          Jerome Boulon added a comment -

          TestChukwaWriters is the test case that I have used to make sure that the dataSink file is still correct

          [junit] Running org.apache.hadoop.chukwa.datacollection.writer.TestChukwaWriters
          [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 11.458 sec

          Show
          Jerome Boulon added a comment - TestChukwaWriters is the test case that I have used to make sure that the dataSink file is still correct [junit] Running org.apache.hadoop.chukwa.datacollection.writer.TestChukwaWriters [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 11.458 sec
          Hide
          Eric Yang added a comment -

          +1 Looks good.

          Show
          Eric Yang added a comment - +1 Looks good.
          Hide
          Eric Yang added a comment -

          I just committed this, thanks Jerome.

          Show
          Eric Yang added a comment - I just committed this, thanks Jerome.
          Eric Yang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Chukwa-trunk #61 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/61/ )

            People

            • Assignee:
              Jerome Boulon
              Reporter:
              Jerome Boulon
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development