Hive
  1. Hive
  2. HIVE-3946

Make it possible to configure for each stage

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Configuration
    • Labels:
      None

      Description

      Some MR related configurations like "mapred.reduce.tasks" or "hive.exec.reducers.bytes.per.reducer" needed to be configured for each stage.

        Activity

        Hide
        Phabricator added a comment -

        navis has commented on the revision "HIVE-3946 [jira] Make it possible to configure for each stage".

        'x' = 10, which is default. HiveConf is created based on SessionState.conf + overriding configs for the stage.

        REVISION DETAIL
        https://reviews.facebook.net/D8181

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - navis has commented on the revision " HIVE-3946 [jira] Make it possible to configure for each stage". 'x' = 10, which is default. HiveConf is created based on SessionState.conf + overriding configs for the stage. REVISION DETAIL https://reviews.facebook.net/D8181 To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-3946 [jira] Make it possible to configure for each stage".

        What are the exact semantics ?

        If Stage 4 depends on Stage 3, which depends on Stage 2, which depends on Stage 1.

        Say, a parameter 'x' = 10
        Say, stage-1.x = 20

        What is 'x' for stage-2 (is it supposed to be 10 or 20) ?

        REVISION DETAIL
        https://reviews.facebook.net/D8181

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-3946 [jira] Make it possible to configure for each stage". What are the exact semantics ? If Stage 4 depends on Stage 3, which depends on Stage 2, which depends on Stage 1. Say, a parameter 'x' = 10 Say, stage-1.x = 20 What is 'x' for stage-2 (is it supposed to be 10 or 20) ? REVISION DETAIL https://reviews.facebook.net/D8181 To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        navis has commented on the revision "HIVE-3946 [jira] Make it possible to configure for each stage".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java:1351 ok.
        ql/src/test/queries/clientpositive/config_per_stage.q:1 We are thinking of using this for the managed query of which plan and configuration is stored with it (and check plan before execution). If it's not, plan changes by optimization would make it invalid as you stated.

        Would it be possible to support this by providing generic expressions or hint? It seemed hard and might be inaccurate.
        ql/src/test/queries/clientpositive/config_per_stage.q:4 I cannot understand this. Why stage-2 should be overridden by defaults? stage-1 will be run with 2 reducer and stage-2 will be run by 3 reducer regardless of adding one more stage.

        REVISION DETAIL
        https://reviews.facebook.net/D8181

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - navis has commented on the revision " HIVE-3946 [jira] Make it possible to configure for each stage". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/Driver.java:1351 ok. ql/src/test/queries/clientpositive/config_per_stage.q:1 We are thinking of using this for the managed query of which plan and configuration is stored with it (and check plan before execution). If it's not, plan changes by optimization would make it invalid as you stated. Would it be possible to support this by providing generic expressions or hint? It seemed hard and might be inaccurate. ql/src/test/queries/clientpositive/config_per_stage.q:4 I cannot understand this. Why stage-2 should be overridden by defaults? stage-1 will be run with 2 reducer and stage-2 will be run by 3 reducer regardless of adding one more stage. REVISION DETAIL https://reviews.facebook.net/D8181 To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-3946 [jira] Make it possible to configure for each stage".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java:1351 Can you add a lot of comments ?
        ql/src/test/queries/clientpositive/config_per_stage.q:1 This might be fairly difficult to maintain for an application.
        I mean, the query plans change so many times due to random optimizations,
        because of which old stage numbers will become invalid.

        It is a very good idea, but can you think of a better naming scheme.
        I dont have any suggestion right now.
        ql/src/test/queries/clientpositive/config_per_stage.q:4 Can you change it to have stage-3 ?

        I mean, the stage-2. parameters should be overridden by defaults ?

        REVISION DETAIL
        https://reviews.facebook.net/D8181

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-3946 [jira] Make it possible to configure for each stage". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/Driver.java:1351 Can you add a lot of comments ? ql/src/test/queries/clientpositive/config_per_stage.q:1 This might be fairly difficult to maintain for an application. I mean, the query plans change so many times due to random optimizations, because of which old stage numbers will become invalid. It is a very good idea, but can you think of a better naming scheme. I dont have any suggestion right now. ql/src/test/queries/clientpositive/config_per_stage.q:4 Can you change it to have stage-3 ? I mean, the stage-2. parameters should be overridden by defaults ? REVISION DETAIL https://reviews.facebook.net/D8181 To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        navis requested code review of "HIVE-3946 [jira] Make it possible to configure for each stage".
        Reviewers: JIRA

        DPAL-1971 Make it possible to configure for each stage

        Some MR related configurations like "mapred.reduce.tasks" or "hive.exec.reducers.bytes.per.reducer" needed to be configured for each stage.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D8181

        AFFECTED FILES
        build-common.xml
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        ql/src/test/queries/clientpositive/config_per_stage.q
        ql/src/test/results/clientpositive/config_per_stage.q.out

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/19797/

        To: JIRA, navis

        Show
        Phabricator added a comment - navis requested code review of " HIVE-3946 [jira] Make it possible to configure for each stage". Reviewers: JIRA DPAL-1971 Make it possible to configure for each stage Some MR related configurations like "mapred.reduce.tasks" or "hive.exec.reducers.bytes.per.reducer" needed to be configured for each stage. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D8181 AFFECTED FILES build-common.xml ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/test/queries/clientpositive/config_per_stage.q ql/src/test/results/clientpositive/config_per_stage.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/19797/ To: JIRA, navis

          People

          • Assignee:
            Navis
            Reporter:
            Navis
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development