• Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: timelineserver
    • Labels:
    • Hadoop Flags:


      For large applications, the majority of the time in LeveldbTimelineStore is spent deleting old entities record at a time. An exclusive write lock is held during the entire deletion phase which in practice can be hours. If we are to relax some of the consistency constraints, other performance enhancing techniques can be employed to maximize the throughput and minimize locking time.

      Split the 5 sections of the leveldb database (domain, owner, start time, entity, index) into 5 separate databases. This allows each database to maximize the read cache effectiveness based on the unique usage patterns of each database. With 5 separate databases each lookup is much faster. This can also help with I/O to have the entity and index databases on separate disks.

      Rolling DBs for entity and index DBs. 99.9% of the data are in these two sections 4:1 ration (index to entity) at least for tez. We replace DB record removal with file system removal if we create a rolling set of databases that age out and can be efficiently removed. To do this we must place a constraint to always place an entity's events into it's correct rolling db instance based on start time. This allows us to stitching the data back together while reading and artificial paging.

      Relax the synchronous writes constraints. If we are willing to accept losing some records that we not flushed in the operating system during a crash, we can use async writes that can be much faster.

      Prefer Sequential writes. sequential writes can be several times faster than random writes. Spend some small effort arranging the writes in such a way that will trend towards sequential write performance over random write performance.


        1. YARN-3448.17.patch
          135 kB
          Jonathan Turner Eagles
        2. YARN-3448.16.patch
          134 kB
          Jonathan Turner Eagles
        3. YARN-3448.15.patch
          134 kB
          Jonathan Turner Eagles
        4. YARN-3448.14.patch
          133 kB
          Jonathan Turner Eagles
        5. YARN-3448.13.patch
          127 kB
          Jonathan Turner Eagles
        6. YARN-3448.12.patch
          119 kB
          Jonathan Turner Eagles
        7. YARN-3448.10.patch
          112 kB
          Jonathan Turner Eagles
        8. YARN-3448.9.patch
          116 kB
          Jonathan Turner Eagles
        9. YARN-3448.8.patch
          115 kB
          Jonathan Turner Eagles
        10. YARN-3448.7.patch
          96 kB
          Jonathan Turner Eagles
        11. YARN-3448.5.patch
          96 kB
          Jonathan Turner Eagles
        12. YARN-3448.4.patch
          98 kB
          Jonathan Turner Eagles
        13. YARN-3448.3.patch
          99 kB
          Jonathan Turner Eagles
        14. YARN-3448.2.patch
          94 kB
          Jonathan Turner Eagles
        15. YARN-3448.1.patch
          92 kB
          Jonathan Turner Eagles

          Issue Links



              • Assignee:
                jeagles Jonathan Turner Eagles
                jeagles Jonathan Turner Eagles
              • Votes:
                0 Vote for this issue
                16 Start watching this issue


                • Created: