Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1526

LoadingCache for TezTaskID slow for large jobs

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
    • Target Version/s:

      Description

      Using the LoadingCache with default builder settings. 100,000 TezTaskIDs are created in 10 seconds on my setup. With a LoadingCache initialCapacity of 10,000 they are created in 300 ms. With no LoadingCache, they are created in 10 ms. A test case in attached to illustrate the condition I would like to be sped up.

        Attachments

        1. hamlet.txt
          206 kB
          Jonathan Eagles
        2. TEZ-1526.8.patch
          21 kB
          Jonathan Eagles
        3. TEZ-1526.memory.test.patch
          3 kB
          Jonathan Eagles
        4. TEZ-1526.7.patch
          22 kB
          Jonathan Eagles
        5. TEZ-1526.6.patch
          19 kB
          Jonathan Eagles
        6. TEZ-1526.5.patch
          13 kB
          Jonathan Eagles
        7. TEZ-1526.4.patch
          5 kB
          Jonathan Eagles
        8. TEZ-1526.3.patch
          2 kB
          Jonathan Eagles
        9. TEZ-1526-v2.patch
          10 kB
          Jonathan Eagles
        10. TEZ-1526-v1.patch
          6 kB
          Jonathan Eagles
        11. 100000-TezTaskIDs.patch
          0.8 kB
          Jonathan Eagles

          Activity

            People

            • Assignee:
              jeagles Jonathan Eagles
              Reporter:
              jeagles Jonathan Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: