Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      File Channel has scaled so well that people now run channels with sizes in 100's of millions of events. Turns out, replay can be crazy slow even between checkpoints at this scale - because of the remove() method in FlumeEventQueue moving every pointer that follows the one being removed (1 remove causes 99 million+ moves for a channel of 100 million!). There are several ways of improving - one being move at the end of replay - sort of like a compaction. Another is to use the fact that all removes happen from the top of the queue, so move the first "k" events out to hashset and remove from there - we can find k using the write id of the last checkpoint and the current one.

        Attachments

        1. 700000-710000
          5.70 MB
          Hari Shreedharan
        2. 300000-310000
          5.70 MB
          Hari Shreedharan
        3. 10000-20000
          5.69 MB
          Hari Shreedharan
        4. 100000-110000
          5.70 MB
          Hari Shreedharan
        5. fc-test.patch
          12 kB
          Hari Shreedharan
        6. SmartReplay.pdf
          72 kB
          Hari Shreedharan
        7. SmartReplay1.1.pdf
          86 kB
          Hari Shreedharan
        8. FLUME-2155-initial.patch
          2 kB
          Hari Shreedharan
        9. FLUME-2155.patch
          12 kB
          Hari Shreedharan
        10. FLUME-FC-SLOW-REPLAY-1.patch
          11 kB
          Brock Noland
        11. FLUME-FC-SLOW-REPLAY-FIX-1.patch
          4 kB
          Brock Noland
        12. FLUME-2155.2.patch
          34 kB
          Brock Noland
        13. FLUME-2155.4.patch
          35 kB
          Brock Noland
        14. FLUME-2155.5.patch
          37 kB
          Brock Noland

          Issue Links

            Activity

              People

              • Assignee:
                brocknoland Brock Noland
                Reporter:
                hshreedharan Hari Shreedharan
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: