XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 4.0.0
    • Backend
    • None
    • ghx-label-2

    Description

      RuntimeProfile trees can potentially stress the memory allocator and use up a lot more memory and cache than is really necessary:

      • std::map is used throughout, and allocates a node per map entry. We do depend on the counters being displayed in-order, but we would probably be better of storing the counters in a vector and lazily sorting when needed (since the set of counters is generally static after Prepare()).
      • We store the same counter names redundantly all over the place. We'd probably be best off using a pool of constant counter names (we could just require registering them upfront).

      There may be a small gain from switching thrift to using unordered_map, e.g. for the info strings that appear with some frequency in profiles.

      However, I think we need to restructure the thrift representation and in-memory representation to get significant gains.

      Attachments

        1. profile_504b379400cba9f2_2d2cf00700000000
          58 kB
          Tim Armstrong
        2. tpcds_q10_profile_v1.txt
          547 kB
          Tim Armstrong
        3. tpcds_q10_profile_v2.txt
          356 kB
          Tim Armstrong
        4. tpcds_q10_profile_v2.txt
          408 kB
          Tim Armstrong

        Issue Links

          Activity

            People

              tarmstrong Tim Armstrong
              tarmstrong Tim Armstrong
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: