Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-313

Remove the 10 sec ack from SeqWriter

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2.0, 0.3.0
    • Fix Version/s: 0.2.0, 0.3.0
    • Component/s: None
    • Labels:
      None

      Description

      SeqWriter.add is waiting 10 sec in a worse case before calling a flush. This was there to guarantee that the data has been written to HDFS before returning the ack.
      It used to work well with 0.18 but in 0.20 the flush is a nohup so there's no point in waiting 10 sec for nothing.

      The real flush will give us the ability to write redo logs directly to HDFS so the ack implementation will have to change anyway.
      So I vote for removing the 10 sec ack from the add method.

      1. CHUKWA-313.patch
        11 kB
        Jerome Boulon

        Issue Links

          Activity

          Hide
          eyang Eric Yang added a comment -

          +1 on removing flush.

          Show
          eyang Eric Yang added a comment - +1 on removing flush.
          Hide
          jboulon Jerome Boulon added a comment -
          • Remove the ClientAck since flush is a noHup operation in H.20
          • Refractor the rotate mechanism
          • Compute TimePeriod once an hour
          Show
          jboulon Jerome Boulon added a comment - Remove the ClientAck since flush is a noHup operation in H.20 Refractor the rotate mechanism Compute TimePeriod once an hour
          Hide
          jboulon Jerome Boulon added a comment -

          TestChukwaWriters is the test case that I have used to make sure that the dataSink file is still correct

          [junit] Running org.apache.hadoop.chukwa.datacollection.writer.TestChukwaWriters
          [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 11.458 sec

          Show
          jboulon Jerome Boulon added a comment - TestChukwaWriters is the test case that I have used to make sure that the dataSink file is still correct [junit] Running org.apache.hadoop.chukwa.datacollection.writer.TestChukwaWriters [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 11.458 sec
          Hide
          eyang Eric Yang added a comment -

          +1 Looks good.

          Show
          eyang Eric Yang added a comment - +1 Looks good.
          Hide
          eyang Eric Yang added a comment -

          I just committed this, thanks Jerome.

          Show
          eyang Eric Yang added a comment - I just committed this, thanks Jerome.
          Hide
          hudson Hudson added a comment -
          Show
          hudson Hudson added a comment - Integrated in Chukwa-trunk #61 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/61/ )

            People

            • Assignee:
              jboulon Jerome Boulon
              Reporter:
              jboulon Jerome Boulon
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development