Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4660

Let there be Tez

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.13.0
    • None
    • None

    Description

      Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html

      The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm.

      The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern.

      MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by)

      For more details see the design doc here: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Tez

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5065 Create proper (i.e.: non .q file based) junit tests for DagUtils and TezTask

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4916 Add TezWork

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4917 Tez Job Monitoring

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4918 Tez job submission

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5003 Localize hive exec jar for tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5004 Localize user jars/files/archives

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5005 Localize hash tables for mapjoin/local work

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5007 Add TezTaskCompiler

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5008 Reuse MapRedUtils to generate Map/ReduceWork

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5040 Yarn resource names cannot contain slashes

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5041 Retrieve and display diagnostic information when TezTask fails on cluster

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5042 Allow MiniMr tests to be run on MiniTezCluster

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5043 Re-enable analyze command for Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5045 Make ctrl-c work with Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5052 Set parallelism when generating the tez tasks

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5053 Let user override the parallelism of each tez task

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5058 Fix NPE issue with DAG submission in TEZ

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5073 Fix problem with multiple root tasks in tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5076 Subsequent reduce stages fail when executed in tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5095 Hive needs new operator walker for parallelization/optimization for tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5097 Update DagUtils to reflect changes in Tez API

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5103 Job numbers are incorrectly displayed in Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5108 Join on Tez fails in certain cases

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5148 Jam sessions w/ Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5151 Going green: Container re-cycling in Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5183 Tez EdgeProperty class has changed

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5184 Load filesystem, ugi, metastore client at tez session startup

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5270 Enable hash joins using tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5271 Convert join op to a map join op in the planning phase

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5367 fix hive-tez build after tez updates

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5368 Changes to work creation for tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5386 Update DagUtils/Tez task to reflect tez api changes

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5387 Need to create edge properties for hive on tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5388 Use a custom LogicalIOProcessor in Tez vertex

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5389 custom LogicalIOProcessor - map record processor

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5390 custom LogicalIOProcessor - reduce record processor

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5404 Remove changes from HIVE-5184

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5409 Enable vectorization for Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5437 Add map/reduce input map to MapWork/ReduceWork for multi input

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5439 Set input edge map for map join operator in Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5442 Plumbing for map join in tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5451 Use IOContext instead of HADOOPMAPFILENAME in MapOperator setup for Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5505 PerfLogger statements for Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5522 Move split generation into the AM

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5533 Re-connect Tez session after AM timeout

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5543 Running the mini tez cluster for tez unit tests

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5544 Print task totals instead of percentages on Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5551 Create perf logger statements for orc init/split creation

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5553 Fix vertex start logging for tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5561 Clear work map for container reuse on tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5585 Fix "empty hashtable" problem with container reuse on tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5586 Call createSplits with multiple paths if partition specs match on Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5587 Change MapRecordProcessor to use VectorMapOperator when necessary on Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5591 Use TezGroupedSplits to combine splits based on headroom in Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5608 tez AM should be able to serialize orc footer in splits

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5620 Tez job progress printing stops after a specific amount of time

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5638 NPE in ConvertJoinMapJoin on Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5645 Cannot compile tests on tez branch

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5647 Fix failing mapreduce tests on the tez branch

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5650 Print yarn app id when running Tez dag

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5651 union_view.q is failing on tez branch

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5688 TestCliDriver compilation fails on tez branch.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5689 Add some simple MRR tests

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5703 While using tez, Qtest needs to close session before creating a new one

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5719 Remove some overly noisy perflogger statements from Tez codepath

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5734 Enable merge/move tasks for Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5735 Enable noscan/partialscan on Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5736 Fix assertion in Operator.java

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5738 Need to clear global work in ExecMapper/ExecReducer for container reuse in tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5766 Update call to Tez DAG status to reflect updated API

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5770 Switch merge tasks to use tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5772 Print message for union operators

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5778 mapjoin hints on Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5808 broadcast join in tez discards duplicate records from the broadcasted table

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5832 Add shutdown hook to stop tez dag/session if jvm dies

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5862 While running some queries on large data using tez, we OOM.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5889 Add counter based stats aggregator for tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5984 Multi insert statement fails on Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6001 Tez: UDFs are not properly localized

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6011 correlation optimizer unit tests are failing on tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6014 Stage ids differ in the tez branch

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6019 Tez: Analyze command fails with dbclass=counter

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6038 Fix Tez branch to properly compile against hadoop-1 profile

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6055 Cleanup aisle tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6077 Fixing a couple of orc unit tests on tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6078 Choosing conditional task for merging files is not deterministic in tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6079 Hadoop 1 tests fail in tez branch

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6081 Dag utils in tez has incorrect dependency on Hadoop20 shims

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6085 Tez changed test parse tests output

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6097 Sessions on Tez NPE when quitting CLI

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6101 Classpath is incorrect for hadoop-1 tests on tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6102 Update golden files for extended explain on mapjoin

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6103 Change hive.optimize.tez to hive.execution.engine with [mr, tez] values

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6106 update golden files for tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6135 Fix merge error on tez branch (TestCompareCliDriver)

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6138 Tez: Add some additional comments to clarify intent

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6168 Fix some javadoc issues on Tez branch

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6169 Update tez specific golden files after merge

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6172 Whitespaces and comments on Tez

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5080 Add hook to do additional optimization on the operator plan in tez

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5378 Need to move SetReducerParallelism to the optimize package.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5882 Reduce logging verbosity on Tez

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5948 Output file name is random when using Tez with "insert overwrite local directory"

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6080 Non-deterministic stage dependencies in tez

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4810 Refactor exec package

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4811 (Slightly) break up the SemanticAnalyzer monstrosity

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4812 Logical explain plan

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4843 Refactoring MapRedTask and ExecDriver for better re-usability (for tez) and readability

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HIVE-4826 Setup build infrastructure for tez

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HIVE-6098 Merge Tez branch into trunk

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-5639 Allow caching of Orc footers in Tez AM

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-4825 Separate MapredWork into MapWork and ReduceWork

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. TEZ-4370 Apache Tez adoption umbrella

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. TEZ-135 Let there be hive

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6128 Add tez variables to hive-default.xml

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Design document/Spec

          Activity

            People

              hagleitn Gunther Hagleitner
              hagleitn Gunther Hagleitner
              Votes:
              0 Vote for this issue
              Watchers:
              35 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: