Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4341

Include metadata loading time in planner timeline

    Details

      Description

      Because missing metadata loading happens outside of analysis / planning (see AnalysisContext.analyze(), it's not included in the EventSequence that's created in Analyzer.GlobalState. We should move the event sequence into the analysis context, and mark when all metadata was successfully loaded.

        Activity

        Hide
        joemcdonnell Joe McDonnell added a comment -

        commit 5d028d93b9b796af448c72396d99ccba43790f72
        Author: Joe McDonnell <joemcdonnell@cloudera.com>
        Date: Wed Jan 11 13:41:00 2017 -0800

        IMPALA-4341: Add metadata load to planner timeline

        This moves the timeline from the Analyzer GlobalState to the AnalysisContext
        and AnalysisContext.AnalysisResult. When analysis needs to load metadata
        about missing tables, it marks an event noting the start of metadata load.
        Then, when metadata load completes (or times out), it marks an
        event noting that metadata load completed (or timed out). Keeping the
        timeline on the AnalysisContext means that it persists across attempts at
        analysis. AnalysisContext.AnalysisResult has a reference to the timeline,
        so that it persists past analyzeStmt and can be used for the rest of
        the planning.

        Here is an example output of the planner timeline after this change:
        Planner Timeline: 4s371ms

        • Metadata load started: 41.388ms (41.388ms)
        • Metadata load finished: 4s260ms (4s219ms)
        • Analysis finished: 4s296ms (35.693ms)
        • Equivalence classes computed: 4s315ms (19.062ms)
        • Single node plan created: 4s323ms (7.812ms)
        • Runtime filters computed: 4s323ms (777.010us)
        • Distributed plan created: 4s325ms (1.464ms)
        • Planning finished: 4s371ms (46.697ms)

        When there is no need to load metadata, the timeline looks like:
        Planner Timeline: 13.695ms

        • Analysis finished: 2.411ms (2.411ms)
        • Equivalence classes computed: 2.653ms (241.733us)
        • Single node plan created: 5.641ms (2.987ms)
        • Runtime filters computed: 5.726ms (85.204us)
        • Distributed plan created: 6.548ms (821.722us)
        • Planning finished: 13.695ms (7.147ms)

        Change-Id: I6f01a35e5f9f5007a0298acfc8e16da00ef99c6c
        Reviewed-on: http://gerrit.cloudera.org:8080/5685
        Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        joemcdonnell Joe McDonnell added a comment - commit 5d028d93b9b796af448c72396d99ccba43790f72 Author: Joe McDonnell <joemcdonnell@cloudera.com> Date: Wed Jan 11 13:41:00 2017 -0800 IMPALA-4341 : Add metadata load to planner timeline This moves the timeline from the Analyzer GlobalState to the AnalysisContext and AnalysisContext.AnalysisResult. When analysis needs to load metadata about missing tables, it marks an event noting the start of metadata load. Then, when metadata load completes (or times out), it marks an event noting that metadata load completed (or timed out). Keeping the timeline on the AnalysisContext means that it persists across attempts at analysis. AnalysisContext.AnalysisResult has a reference to the timeline, so that it persists past analyzeStmt and can be used for the rest of the planning. Here is an example output of the planner timeline after this change: Planner Timeline: 4s371ms Metadata load started: 41.388ms (41.388ms) Metadata load finished: 4s260ms (4s219ms) Analysis finished: 4s296ms (35.693ms) Equivalence classes computed: 4s315ms (19.062ms) Single node plan created: 4s323ms (7.812ms) Runtime filters computed: 4s323ms (777.010us) Distributed plan created: 4s325ms (1.464ms) Planning finished: 4s371ms (46.697ms) When there is no need to load metadata, the timeline looks like: Planner Timeline: 13.695ms Analysis finished: 2.411ms (2.411ms) Equivalence classes computed: 2.653ms (241.733us) Single node plan created: 5.641ms (2.987ms) Runtime filters computed: 5.726ms (85.204us) Distributed plan created: 6.548ms (821.722us) Planning finished: 13.695ms (7.147ms) Change-Id: I6f01a35e5f9f5007a0298acfc8e16da00ef99c6c Reviewed-on: http://gerrit.cloudera.org:8080/5685 Reviewed-by: Marcel Kornacker <marcel@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            joemcdonnell Joe McDonnell
            Reporter:
            henryr Henry Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development