Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

      The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

      The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

      MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

      For more details see the design doc here: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Tez

        Issue Links

          Activity

          Gunther Hagleitner created issue -
          Gunther Hagleitner made changes -
          Field Original Value New Value
          Attachment HiveonTez.pdf [ 12586263 ]
          Gunther Hagleitner made changes -
          Link This issue is related to TEZ-135 [ TEZ-135 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4810 [ HIVE-4810 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4811 [ HIVE-4811 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4812 [ HIVE-4812 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4825 [ HIVE-4825 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4826 [ HIVE-4826 ]
          Gunther Hagleitner made changes -
          Summary Let there be Tez (aka mrr ftw) Let there be Tez
          Gunther Hagleitner made changes -
          Attachment HiveonTez.pdf [ 12586263 ]
          Gunther Hagleitner made changes -
          Remote Link This issue links to "Design document/Spec (Web Link)" [ 12431 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4843 [ HIVE-4843 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4916 [ HIVE-4916 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4917 [ HIVE-4917 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-4918 [ HIVE-4918 ]
          Gunther Hagleitner made changes -
          Description Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

          The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

          The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

          MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

          For more details see the attached design doc.
          Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

          The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

          The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

          MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

          For more details see the design doc here:
          Gunther Hagleitner made changes -
          Description Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

          The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

          The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

          MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

          For more details see the design doc here:
          Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

          The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

          The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

          MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

          For more details see the design doc here: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Tez
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5003 [ HIVE-5003 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5004 [ HIVE-5004 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5005 [ HIVE-5005 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5007 [ HIVE-5007 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5008 [ HIVE-5008 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5040 [ HIVE-5040 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5041 [ HIVE-5041 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5042 [ HIVE-5042 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5043 [ HIVE-5043 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5045 [ HIVE-5045 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5052 [ HIVE-5052 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5053 [ HIVE-5053 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5058 [ HIVE-5058 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5065 [ HIVE-5065 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5073 [ HIVE-5073 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5076 [ HIVE-5076 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5080 [ HIVE-5080 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-5095 [ HIVE-5095 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5097 [ HIVE-5097 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5103 [ HIVE-5103 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5108 [ HIVE-5108 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5148 [ HIVE-5148 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5151 [ HIVE-5151 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5183 [ HIVE-5183 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5184 [ HIVE-5184 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5367 [ HIVE-5367 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5271 [ HIVE-5271 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5368 [ HIVE-5368 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-5378 [ HIVE-5378 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5386 [ HIVE-5386 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-5387 [ HIVE-5387 ]
          Thejas M Nair made changes -
          Link This issue is blocked by HIVE-5388 [ HIVE-5388 ]
          Thejas M Nair made changes -
          Link This issue is blocked by HIVE-5389 [ HIVE-5389 ]
          Thejas M Nair made changes -
          Link This issue is blocked by HIVE-5390 [ HIVE-5390 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5270 [ HIVE-5270 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5409 [ HIVE-5409 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5437 [ HIVE-5437 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5439 [ HIVE-5439 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5442 [ HIVE-5442 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5451 [ HIVE-5451 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5505 [ HIVE-5505 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5522 [ HIVE-5522 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5533 [ HIVE-5533 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-5543 [ HIVE-5543 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5544 [ HIVE-5544 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5551 [ HIVE-5551 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5553 [ HIVE-5553 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5561 [ HIVE-5561 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5586 [ HIVE-5586 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5587 [ HIVE-5587 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5585 [ HIVE-5585 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5591 [ HIVE-5591 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5608 [ HIVE-5608 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5620 [ HIVE-5620 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5638 [ HIVE-5638 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5639 [ HIVE-5639 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5645 [ HIVE-5645 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5647 [ HIVE-5647 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5650 [ HIVE-5650 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5651 [ HIVE-5651 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5689 [ HIVE-5689 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-5688 [ HIVE-5688 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-5404 [ HIVE-5404 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-5703 [ HIVE-5703 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5719 [ HIVE-5719 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5734 [ HIVE-5734 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5735 [ HIVE-5735 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5736 [ HIVE-5736 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5738 [ HIVE-5738 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5766 [ HIVE-5766 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5770 [ HIVE-5770 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5772 [ HIVE-5772 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5778 [ HIVE-5778 ]
          Thejas M Nair made changes -
          Link This issue relates to HIVE-5808 [ HIVE-5808 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5808 [ HIVE-5808 ]
          Gunther Hagleitner made changes -
          Link This issue relates to HIVE-5808 [ HIVE-5808 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5832 [ HIVE-5832 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-5862 [ HIVE-5862 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5882 [ HIVE-5882 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5889 [ HIVE-5889 ]
          Hitesh Shah made changes -
          Link This issue is blocked by HIVE-5948 [ HIVE-5948 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-5984 [ HIVE-5984 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6001 [ HIVE-6001 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6011 [ HIVE-6011 ]
          Vikram Dixit K made changes -
          Link This issue is blocked by HIVE-6014 [ HIVE-6014 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6019 [ HIVE-6019 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6038 [ HIVE-6038 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6055 [ HIVE-6055 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6077 [ HIVE-6077 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6079 [ HIVE-6079 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6078 [ HIVE-6078 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6081 [ HIVE-6081 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6085 [ HIVE-6085 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6080 [ HIVE-6080 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6097 [ HIVE-6097 ]
          Gunther Hagleitner made changes -
          Link This issue relates to HIVE-6098 [ HIVE-6098 ]
          Gunther Hagleitner made changes -
          Link This issue relates to HIVE-6098 [ HIVE-6098 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6098 [ HIVE-6098 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6101 [ HIVE-6101 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6102 [ HIVE-6102 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6103 [ HIVE-6103 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6106 [ HIVE-6106 ]
          Gunther Hagleitner made changes -
          Link This issue relates to HIVE-6128 [ HIVE-6128 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6135 [ HIVE-6135 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6138 [ HIVE-6138 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6168 [ HIVE-6168 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6169 [ HIVE-6169 ]
          Gunther Hagleitner made changes -
          Link This issue is blocked by HIVE-6172 [ HIVE-6172 ]
          Gunther Hagleitner made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Resolution Fixed [ 1 ]
          Chao Sun made changes -
          Description Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

          The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

          The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

          MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

          For more details see the design doc here: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Tez
          Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

          The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

          The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

          MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

          For more details see the design doc here
          Chao Sun made changes -
          Description Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

          The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

          The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

          MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

          For more details see the design doc here
          Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

          The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

          The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

          MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

          For more details see the design doc here: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Tez

            People

            • Assignee:
              Gunther Hagleitner
              Reporter:
              Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              36 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development