[PIG-3446] Umbrella jira for Pig on Tez

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tez-branch
• Fix Version/s: 0.14.0
• Component/s: tez
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 31/Aug/13 22:42
• Updated: 12/Jan/22 12:52
• Resolved: 13/Mar/15 21:38

Description

This is a umbrella jira for Pig on Tez. More detailed subtasks will be added.

More information can be found on the following wiki page:
https://cwiki.apache.org/confluence/display/PIG/Pig+on+Tez

How to set up your development environment-

1. Check out Tez trunk.
2. Install protobuf 2.5.0.
3. Build Tez with Hadoop 2.2.0.(By default, it builds against Hadoop trunk, which is 3.0.0.)
4. Install Tez jars on local maven repository with "mvn install -DskipTests".
5. Check out Pig Tez branch.
6. Build Pig running "ant jar-withouthadoop".
7. Set up a single-node (or multi-node) Hadoop 2.2 cluster.
8. Install Tez following the instructions on the Tez homepage.
9. Run Pig with "-x tez" option.

How to run Tez tests-

• unit test

  ant test-tez
  

  By default, exectype is tez, and hadoopversion is 23 in tez branch. But you can run unit tests in mr mode as follows:

  ant test -Dexectype=mr -Dhadoopversion=20
  

• e2e tests

  ant -Dharness.old.pig=$PIG_HOME -Dharness.hadoop.home=$HADOOP_HOME -Dharness.cluster.conf=$HADOOP_CONF -Dharness.cluster.bin=$HADOOP_BIN test-e2e-tez -Dhadoopversion=23
  

Issue Links

• is related to: TEZ-388 Pig on Tez
• is related to: TEZ-4370 Apache Tez adoption umbrella
• relates to: PIG-3890 Global sort is not working (order by) Pig over Tez
• relates to: PIG-3839 Umbrella jira for Pig on Tez Performance Improvements
• relates to: PIG-4059 Pig on Spark
• relates to: PIG-3840 Umbrella jira for Pig on Tez Unit Test porting
• requires: PIG-3520 Provide backward compatibility for PigRunner and PPNL after PIG-3419
• requires: PIG-3525 PigStats.get() and ScriptState.get() shouldn't return MR-specific objects
• requires: PIG-3591 Refactor POPackage to separate MR specific code from packaging
• requires: PIG-3508 'explain' now showing logical plan BEFORE the necessary optimization (ImplicitSplitInserter, DuplicateForEachColumnRewrite,etc)
• requires: PIG-3419 Pluggable Execution Engine
• requires: TEZ-591 Provide mode specific diagnostic information to the Tez client
• supercedes: PIG-1734 Pig needs a more efficient DAG execution

Sub-Tasks

• Move JobCreationException to org.apache.pig.backend.hadoop.executionengine: Closed
• Tez backend layout: Closed
• Add a base abstract class for ExecutionEngine: Closed
• Initial implementation of TezCompiler: Closed
• Initial implementation of TezJobControlCompiler: Closed
• Initial implementation of TezLauncher: Closed
• Initial implementation of TezStats: Closed
• Initial Implementation of PigProcessor: Closed
• Bump hadoop version to 2.2.0: Closed
• Allow PigProcessor to handle multiple inputs: Closed
• Add TezMiniCluster for unit tests: Closed
• Empty plan fails to run: Closed
• Make register work: Closed
• Make order by work: Closed
• Add test-tez target to build.xml: Closed
• Make distinct work: Closed
• Make limit work: Closed
• Pig should be able to submit multiple DAG: Closed
• e2e test for tez: Closed
• Add diagnostic information to TezStats: Closed
• Fix tez Checkin_2: Closed
• Initial implementation of combiner optimization: Closed
• Fix tez branch compilation with Hadoop 1.0: Closed
• Implement optimizations for LIMIT: Closed
• UniqueTez staging dir should be used for different users: Resolved
• Implement combiner optimizations for DISTINCT: Closed
• Make Tez work with security: Closed
• Make split work with Tez: Closed
• Make union work with tez: Closed
• Fix dependencies in ivy.xml: Closed
• Port Package refactoring to Tez branch: Closed
• Fix e2e Operator_1, 5, Checkin_3, and Join_1: Closed
• Move POSimpleTezLoad under tez package: Closed
• Tear down TezSessions when Pig exits: Closed
• Add counters to TezStats: Closed
• Implement replicated join in Tez: Closed
• Fix e2e tests Operators_3, Operators_5: Closed
• Add order by string, descending order e2e tests: Closed
• Replace broadcast edges with scatter/gather edges in union: Closed
• TezCompiler adds duplicate predecessors of blocking operators to TezPlan: Closed
• Fix intermittent test failure Join_1: Closed
• Make combiners, custom partitioners and secondary key sort work for multiple outputs: Closed
• Implement STREAM in Tez: Closed
• Improve performance of order-by: Closed
• Make accumulator UDF work in Tez: Closed
• Implement skewed join in Tez: Closed
• Implement merge join in Tez: Closed
• Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map: Closed
• TEZ-41 break pig-tez: Closed
• Fix store after load: Closed
• Fix skewed join e2e tests: Closed
• Change tez version dependency as a result of TEZ-739: Closed
• Fix split + skewed join: Resolved
• Fix TestSkewedJoin in tez mode: Closed
• Use ONE_TO_ONE edge and IdentityInOut in orderby intermediate vertex: Closed
• Set MR runtime settings on tez runtime: Closed
• Use VertexGroup and Alias vertex for union: Closed
• Support for multiquery off in Tez: Closed
• Add support for non-Java UDF's: Closed
• Make scalar work: Closed
• Fix desc order by in Tez: Closed
• CombinerOptimizer should not optimize cogroup case in tez: Closed
• Outer join fail on tez: Closed
• Properties aren't propagated to edges or vertices in Tez: Resolved
• Use ONE_TO_ONE edge and IdentityInOut in skewed join intermediate vertex: Closed
• Work with TEZ-668 which allows starting and closing of inputs and outputs: Closed
• TezCompiler.visitUnion() doesn't add compiled TezOp to phyToTezOpMap: Closed
• Scripting UDF is broken after PIG-3629: Closed
• Tez mini cluster tests run for a very long time with TezSession reuse on: Closed
• Fix TestTezCompiler#testReplicatedJoinInReducer: Closed
• TezResourceManager should not be a singleton: Closed
• POReservoirSample should handle endOfAllInput flag: Closed
• Multiquery with FRJoin fail: Closed
• NPE when POStream is not in the leaf vertex: Closed
• tuple in POStream binaryInputQueue keep changing: Closed
• Several changes in Tez e2e: Closed
• POValueInputTez should handle getNextTuple even after reader.next() returns null: Closed
• Parallelism specified by user is not honored if default parallelism is set to a higher value: Closed
• Fix some memory leaks affecting container reuse: Closed
• TestCustomPartitioner is broken in tez branch: Closed
• POPoissonSample should handle endOfAllInput flag: Closed
• Implement CROSS in Tez: Closed
• Implement RANK in Tez: Closed
• Remove reference to BroadcastKVReader as it is removed in TEZ-911: Closed
• Make custom counter work: Closed
• Implement mapside cogroup in Tez: Resolved
• Organize tez code into subpackages: Closed
• Honor Mapreduce Distributed Cache settings and localize resources in Tez: Closed
• Pig on tez job hangs when AM has a failure and Multiquery fixes: Closed
• Pig script encounters error with Tez MemoryDistributor: Resolved
• e2e test Rank_9 fail: Closed
• e2e tests run all tests even execonly flag does not match: Closed
• Fix MergeJoin_8 failure: Closed
• UdfDistributedCache_1 fails in tez branch: Closed
• Global sort is not working (order by) Pig over Tez: Resolved
• Hash join followed by replicated join fails in Tez mode: Closed
• Fix memory leak with PigTezLogger: Closed
• Fix UnionOptimizer bug with expressions and MR compressions settings not honored: Closed
• Implement PPNL for Tez mode (Pig side changes): Closed
• PigRecordWriter throws exception in Tez mode: Closed
• Fix e2e test failure CastScalar_11: Closed
• Skewed join followed by replicated join fails in Tez: Closed
• Fix MR unit tests on tez branch: Closed
• Pig on tez fails to run in Oozie in secure cluster: Closed
• Get TezStats working for Oozie: Closed
• Make the interval of DAGStatus report configurable: Closed
• New interface for resetting static variables for jvm reuse: Closed
• Fix compilation failure due in Pig on Tez due to TEZ-1127 change: Resolved
• Refactor TezJob and TezLauncher: Closed
• Make Streaming UDF work in Tez: Closed
• ObjectCache cause ClassCastException: Closed
• Change from TezJobConfig to TezRuntimeConfiguration: Closed
• Accumulator UDF throws OOM in Tez: Closed
• NPE in packager when union + group-by followed by replicated join in Tez: Closed
• Implement merge cogroup in Tez: Closed
• Add Native operator to tez: Closed
• Create a target to run mr and tez unit test in one shot: Closed
• Pin Tez to 0.5.0 release: Closed
• Intermediate reducer parallelism in Tez should be higher: Closed
• Mapreduce ACLs should be translated to Tez ACLs: Closed
• Reset UDFContext state before OutputCommitter invocations in Tez: Closed
• Fix few issues related to Union, CROSS and auto parallelism in Tez: Closed
• PigProcessor does not set pig.datetime.default.tz: Closed
• ObjectCache should use ProcessorContext.getObjectRegistry(): Closed