Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark
    • Labels:
      None

      Description

      SparkTask handles the execution of SparkWork. It will execute a graph of map and reduce work using a SparkClient instance.

      1. HIVE-7330.1-spark.patch
        12 kB
        Chinna Rao Lalam
      2. HIVE-7330-spark.patch
        10 kB
        Chinna Rao Lalam

        Issue Links

          Activity

          Xuefu Zhang made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Hide
          Xuefu Zhang added a comment -

          Close this as SparkTask is already in place and the work contained in the partch is covered in other JIRAs.

          Show
          Xuefu Zhang added a comment - Close this as SparkTask is already in place and the work contained in the partch is covered in other JIRAs.
          Brock Noland made changes -
          Summary Create SparkTask Create SparkTask [Spark Branch]
          Hide
          Brock Noland added a comment -

          Linking to HIVE-7606.

          Show
          Brock Noland added a comment - Linking to HIVE-7606 .
          Brock Noland made changes -
          Link This issue depends upon HIVE-7606 [ HIVE-7606 ]
          Hide
          Brock Noland added a comment -

          I was just re-reading the design doc, I think it makes sense to break this JIRA down. I don't think the first part of this JIRA is work on SparkTask. Let's create a sub-task which defines our "Session" interface and provides a basic impl as a first step. I think we can do that in a separate JIRA. I've created HIVE-7606 for that task. Chinna Rao Lalam do you want to start on that JIRA?

          Show
          Brock Noland added a comment - I was just re-reading the design doc, I think it makes sense to break this JIRA down. I don't think the first part of this JIRA is work on SparkTask. Let's create a sub-task which defines our "Session" interface and provides a basic impl as a first step. I think we can do that in a separate JIRA. I've created HIVE-7606 for that task. Chinna Rao Lalam do you want to start on that JIRA?
          Hide
          Brock Noland added a comment -

          Thank you Chinna Rao Lalam!

          Would you mind adding TODO items for aforementioned refactoring, rebasing on trunk (precommit failed to apply the patch), and add a Review Board item (reviews.apache.org)?

          Show
          Brock Noland added a comment - Thank you Chinna Rao Lalam ! Would you mind adding TODO items for aforementioned refactoring, rebasing on trunk (precommit failed to apply the patch), and add a Review Board item (reviews.apache.org)?
          Chinna Rao Lalam made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Chinna Rao Lalam added a comment -

          Thanks for the inputs Chengxiang Li and Xuefu Zhang, I will keep spark client as it is. Refactoring we can do later stage.
          As part of this Jira i will add missing functionality.

          Show
          Chinna Rao Lalam added a comment - Thanks for the inputs Chengxiang Li and Xuefu Zhang, I will keep spark client as it is. Refactoring we can do later stage. As part of this Jira i will add missing functionality.
          Hide
          Xuefu Zhang added a comment -

          Chengxiang Li and Chinna Rao Lalam, many thanks for the inputs. I can see merits on either side of argument, but in essence they pretty much do the same thing. I created this JIRA to add new functionality that was missing, such as session management, capacity control, etc. Since this patch doesn't seem adding any missing functionality, I think we can keep the refactoring at a later stage of the project.

          In general I'd like to see implementation following design, though I'm open to design changes if problems are found in the original design.

          Show
          Xuefu Zhang added a comment - Chengxiang Li and Chinna Rao Lalam , many thanks for the inputs. I can see merits on either side of argument, but in essence they pretty much do the same thing. I created this JIRA to add new functionality that was missing, such as session management, capacity control, etc. Since this patch doesn't seem adding any missing functionality, I think we can keep the refactoring at a later stage of the project. In general I'd like to see implementation following design, though I'm open to design changes if problems are found in the original design.
          Hide
          Chengxiang Li added a comment -

          Chinna Rao Lalam I agree with you that we could refresh local resource, monitoring job outside SparkClient, As a wrapper class of SparkContext, SparkClient would be just in charge of submit spark job. One concerns is that, in your patch, spark job is submitted in SparkTask instead of SparkClient, and SparkContext is exposed and used in SparkTask. For better responsibility division, should we just use SparkClient to submit spark job and load environment information in SparkTask? what do you think?

          Show
          Chengxiang Li added a comment - Chinna Rao Lalam I agree with you that we could refresh local resource, monitoring job outside SparkClient, As a wrapper class of SparkContext, SparkClient would be just in charge of submit spark job. One concerns is that, in your patch, spark job is submitted in SparkTask instead of SparkClient, and SparkContext is exposed and used in SparkTask. For better responsibility division, should we just use SparkClient to submit spark job and load environment information in SparkTask? what do you think?
          Hide
          Chinna Rao Lalam added a comment -

          Hi Chengxiang Li, my thoughts are

          Task.execute() method is overridden by other classes and corresponding sub classes will be responsible for executing the work.
          In spark task situation for executing spark job required resource loading and submission of job need to be done here by using spark client.

          Spark client will be a wrapper class for providing the sparks functionality.

          I have added one TODO for job monitoring in sprak task, here also we can use the class which provides functionality of the spark job monitoring and get the job status in spark task class itself.

          Similar approach i have seen in ExecDriver.java and TezTask.java.

          Show
          Chinna Rao Lalam added a comment - Hi Chengxiang Li, my thoughts are Task.execute() method is overridden by other classes and corresponding sub classes will be responsible for executing the work. In spark task situation for executing spark job required resource loading and submission of job need to be done here by using spark client. Spark client will be a wrapper class for providing the sparks functionality. I have added one TODO for job monitoring in sprak task, here also we can use the class which provides functionality of the spark job monitoring and get the job status in spark task class itself. Similar approach i have seen in ExecDriver.java and TezTask.java.
          Hide
          Chengxiang Li added a comment -

          Hi, Chinna Rao Lalam, From the design doc, SparkTask should be a quite thin layer which take SparkWork as input and use SparkClient to execute it. SparkClient is the one who would load configurations, ship local resources, submit spark job, and monitor job status. It seems your patch only use SparkClient to load configurations, and move others functions to SparkTask, do you have any good reasons for this?

          Show
          Chengxiang Li added a comment - Hi, Chinna Rao Lalam , From the design doc, SparkTask should be a quite thin layer which take SparkWork as input and use SparkClient to execute it. SparkClient is the one who would load configurations, ship local resources, submit spark job, and monitor job status. It seems your patch only use SparkClient to load configurations, and move others functions to SparkTask, do you have any good reasons for this?
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12658902/HIVE-7330.1-spark.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/6/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/6/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-6/

          Messages:

          **** This message was trimmed, see log for full details ****
          U    ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UDFCurrentDB.java
          U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
          U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java
          U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java
          U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java
          U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java
          A    ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
          A    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
          A    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanOptimizer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedMergeJoinProc.java
          A    ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/IntraQueryCorrelation.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/QueryPlanTreeTransformation.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
          A    ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java
          U    ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java
          U    ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java
          A    ql/src/java/org/apache/hadoop/hive/ql/plan/ShowConfDesc.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/RevokeDesc.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/TezEdgeProperty.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
          U    ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
          U    ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          A    ql/src/java/org/apache/hadoop/hive/ql/parse/StorageFormat.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
          U    ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
          A    ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
          U    ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
          U    ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
          U    ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          U    ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          U    ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java
          U    ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java
          U    ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssignFactory.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpressionWriterFactory.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorElt.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/ConstantVectorExpression.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkShuffler.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ChainedTran.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveVoidFunction.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/GroupByShuffler.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkCollector.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ReduceTran.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapTran.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
          A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTran.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
          A    ql/src/main/resources/META-INF
          A    ql/src/main/resources/META-INF/services
          A    ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor
          U    ql/pom.xml
          U    conf/hive-default.xml.template
          A    service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java
          U    service/src/java/org/apache/hive/service/server/HiveServer2.java
          U    service/src/java/org/apache/hive/service/cli/thrift/EmbeddedThriftBinaryCLIService.java
          U    service/src/java/org/apache/hive/service/cli/CLIService.java
          U    service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java
          U    service/src/java/org/apache/hive/service/cli/session/SessionManager.java
          U    service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
          U    service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
          U    contrib/src/test/results/clientnegative/udtf_explode2.q.out
          U    contrib/src/test/results/clientnegative/case_with_row_sequence.q.out
          U    contrib/src/test/results/clientnegative/invalid_row_sequence.q.out
          U    contrib/src/test/results/clientpositive/udaf_example_avg.q.out
          U    contrib/src/test/results/clientpositive/udf_example_arraymapstruct.q.out
          U    contrib/src/test/results/clientpositive/udaf_example_group_concat.q.out
          U    contrib/src/test/results/clientpositive/udaf_example_min.q.out
          U    contrib/src/test/results/clientpositive/udf_row_sequence.q.out
          U    contrib/src/test/results/clientpositive/udaf_example_max.q.out
          U    contrib/src/test/results/clientpositive/udf_example_add.q.out
          U    contrib/src/test/results/clientpositive/udf_example_format.q.out
          U    contrib/src/test/results/clientpositive/udtf_output_on_close.q.out
          U    contrib/src/test/results/clientpositive/udtf_explode2.q.out
          U    contrib/src/test/results/clientpositive/dboutput.q.out
          U    contrib/src/test/results/clientpositive/udaf_example_min_n.q.out
          U    contrib/src/test/results/clientpositive/udaf_example_max_n.q.out
          U    contrib/src/test/results/clientpositive/lateral_view_explode2.q.out
          A    serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java
          A    serde/src/test/resources
          A    serde/src/test/resources/avro-nested-struct.avsc
          A    serde/src/test/resources/avro-struct.avsc
          U    serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
          U    serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java
          U    serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
          A    serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java
          U    jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java
          U    jdbc/pom.xml
          U    beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java
          U    beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java
          U    beeline/src/java/org/apache/hive/beeline/TableOutputFormat.java
          U    beeline/src/java/org/apache/hive/beeline/BeeLine.java
          U    beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
          U    beeline/src/main/resources/BeeLine.properties
          A    data/files/in_file.dat
          A    data/conf/hiveserver2-site.xml
          U    data/conf/hive-site.xml
          A    data/conf/hivemetastore-site.xml
          U    itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java
          U    itests/hive-unit/src/test/java/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java
          A    itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestServerSpecificConfig.java
          U    itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestAuthorizationApiAuthorizer.java
          U    itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java
          A    itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestHS2AuthzContext.java
          U    itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestJdbcWithSQLAuthorization.java
          U    itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java
          A    itests/custom-serde/src/main/java/org/apache/hadoop/hive/serde2/CustomTextStorageFormatDescriptor.java
          A    itests/custom-serde/src/main/java/org/apache/hadoop/hive/serde2/CustomTextSerDe.java
          A    itests/custom-serde/src/main/resources
          A    itests/custom-serde/src/main/resources/META-INF
          A    itests/custom-serde/src/main/resources/META-INF/services
          A    itests/custom-serde/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor
          U    itests/util/src/main/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizerFactoryForTest.java
          U    itests/util/src/main/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAccessControllerForTest.java
          A    itests/util/src/main/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidatorForTest.java
          U    itests/qtest/testconfiguration.properties
          U    hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java
          U    shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          A    common/src/java/org/apache/hadoop/hive/ant
          A    common/src/java/org/apache/hadoop/hive/ant/GenHiveTemplate.java
          A    common/src/java/org/apache/hadoop/hive/conf/SystemVariables.java
          A    common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java
          U    common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          A    common/src/java/org/apache/hadoop/hive/conf/Validator.java
          A    common/src/java/org/apache/hadoop/hive/common/cli/HiveFileProcessor.java
          A    common/src/java/org/apache/hadoop/hive/common/cli/IHiveFileProcessor.java
          A    common/src/java/org/apache/hive/common/util/AnnotationUtils.java
          U    common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java
          U    common/src/test/org/apache/hadoop/hive/conf/TestHiveLogging.java
          U    common/src/test/org/apache/hadoop/hive/conf/TestHiveConfRestrictList.java
          U    common/pom.xml
          U    pom.xml
           U   .
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          Updated external to revision 1614857.
          
          Updated to revision 1614857.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12658902

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12658902/HIVE-7330.1-spark.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/6/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/6/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-6/ Messages: **** This message was trimmed, see log for full details **** U ql/src/java/org/apache/hadoop/hive/ql/Driver.java U ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UDFCurrentDB.java U ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java U ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java U ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java U ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java U ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java A ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java A ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java A ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanOptimizer.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedMergeJoinProc.java A ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/IntraQueryCorrelation.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/QueryPlanTreeTransformation.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java U ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java A ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java U ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java U ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java U ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java U ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java A ql/src/java/org/apache/hadoop/hive/ql/plan/ShowConfDesc.java U ql/src/java/org/apache/hadoop/hive/ql/plan/RevokeDesc.java U ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java U ql/src/java/org/apache/hadoop/hive/ql/plan/TezEdgeProperty.java U ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java U ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java U ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java U ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java U ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java U ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java U ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java U ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g U ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java U ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java U ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g U ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java U ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java U ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java U ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java U ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java U ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java U ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java U ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g U ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java U ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java A ql/src/java/org/apache/hadoop/hive/ql/parse/StorageFormat.java U ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java U ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java A ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java U ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java U ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java U ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java U ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java U ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java U ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java U ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java U ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java U ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java U ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssignFactory.java U ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpressionWriterFactory.java A ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorElt.java U ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/ConstantVectorExpression.java U ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java U ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java U ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java U ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java U ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java U ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkShuffler.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ChainedTran.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java U ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveVoidFunction.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/GroupByShuffler.java U ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkCollector.java U ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ReduceTran.java U ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapTran.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java U ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java A ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTran.java U ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java U ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java U ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java U ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java U ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java U ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java U ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java U ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java U ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java U ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java U ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java U ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java U ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java A ql/src/main/resources/META-INF A ql/src/main/resources/META-INF/services A ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor U ql/pom.xml U conf/hive-default.xml.template A service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java U service/src/java/org/apache/hive/service/server/HiveServer2.java U service/src/java/org/apache/hive/service/cli/thrift/EmbeddedThriftBinaryCLIService.java U service/src/java/org/apache/hive/service/cli/CLIService.java U service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java U service/src/java/org/apache/hive/service/cli/session/SessionManager.java U service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java U service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java U contrib/src/test/results/clientnegative/udtf_explode2.q.out U contrib/src/test/results/clientnegative/case_with_row_sequence.q.out U contrib/src/test/results/clientnegative/invalid_row_sequence.q.out U contrib/src/test/results/clientpositive/udaf_example_avg.q.out U contrib/src/test/results/clientpositive/udf_example_arraymapstruct.q.out U contrib/src/test/results/clientpositive/udaf_example_group_concat.q.out U contrib/src/test/results/clientpositive/udaf_example_min.q.out U contrib/src/test/results/clientpositive/udf_row_sequence.q.out U contrib/src/test/results/clientpositive/udaf_example_max.q.out U contrib/src/test/results/clientpositive/udf_example_add.q.out U contrib/src/test/results/clientpositive/udf_example_format.q.out U contrib/src/test/results/clientpositive/udtf_output_on_close.q.out U contrib/src/test/results/clientpositive/udtf_explode2.q.out U contrib/src/test/results/clientpositive/dboutput.q.out U contrib/src/test/results/clientpositive/udaf_example_min_n.q.out U contrib/src/test/results/clientpositive/udaf_example_max_n.q.out U contrib/src/test/results/clientpositive/lateral_view_explode2.q.out A serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java A serde/src/test/resources A serde/src/test/resources/avro-nested-struct.avsc A serde/src/test/resources/avro-struct.avsc U serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java U serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java U serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java A serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java U jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java U jdbc/pom.xml U beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java U beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java U beeline/src/java/org/apache/hive/beeline/TableOutputFormat.java U beeline/src/java/org/apache/hive/beeline/BeeLine.java U beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java U beeline/src/main/resources/BeeLine.properties A data/files/in_file.dat A data/conf/hiveserver2-site.xml U data/conf/hive-site.xml A data/conf/hivemetastore-site.xml U itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java U itests/hive-unit/src/test/java/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java A itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestServerSpecificConfig.java U itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestAuthorizationApiAuthorizer.java U itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java A itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestHS2AuthzContext.java U itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestJdbcWithSQLAuthorization.java U itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java A itests/custom-serde/src/main/java/org/apache/hadoop/hive/serde2/CustomTextStorageFormatDescriptor.java A itests/custom-serde/src/main/java/org/apache/hadoop/hive/serde2/CustomTextSerDe.java A itests/custom-serde/src/main/resources A itests/custom-serde/src/main/resources/META-INF A itests/custom-serde/src/main/resources/META-INF/services A itests/custom-serde/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor U itests/util/src/main/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizerFactoryForTest.java U itests/util/src/main/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAccessControllerForTest.java A itests/util/src/main/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidatorForTest.java U itests/qtest/testconfiguration.properties U hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java U shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java A common/src/java/org/apache/hadoop/hive/ant A common/src/java/org/apache/hadoop/hive/ant/GenHiveTemplate.java A common/src/java/org/apache/hadoop/hive/conf/SystemVariables.java A common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java U common/src/java/org/apache/hadoop/hive/conf/HiveConf.java A common/src/java/org/apache/hadoop/hive/conf/Validator.java A common/src/java/org/apache/hadoop/hive/common/cli/HiveFileProcessor.java A common/src/java/org/apache/hadoop/hive/common/cli/IHiveFileProcessor.java A common/src/java/org/apache/hive/common/util/AnnotationUtils.java U common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java U common/src/test/org/apache/hadoop/hive/conf/TestHiveLogging.java U common/src/test/org/apache/hadoop/hive/conf/TestHiveConfRestrictList.java U common/pom.xml U pom.xml U . Fetching external item into 'hcatalog/src/test/e2e/harness' Updated external to revision 1614857. Updated to revision 1614857. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12658902
          Chinna Rao Lalam made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chinna Rao Lalam made changes -
          Attachment HIVE-7330.1-spark.patch [ 12658902 ]
          Hide
          Xuefu Zhang added a comment -

          Thanks for the patch, Chinna Rao Lalam. It seems that most of changes are about merging SparkClient and SparkTask, which I'm not sure of. Chengxiang Li, you're working on SparkClient. Could you take a look at the patch and share your thoughts?

          Show
          Xuefu Zhang added a comment - Thanks for the patch, Chinna Rao Lalam . It seems that most of changes are about merging SparkClient and SparkTask, which I'm not sure of. Chengxiang Li , you're working on SparkClient. Could you take a look at the patch and share your thoughts?
          Chinna Rao Lalam made changes -
          Attachment HIVE-7330-spark.patch [ 12658847 ]
          Chinna Rao Lalam made changes -
          Assignee Na Yang [ nyang ] Chinna Rao Lalam [ chinnalalam ]
          Hide
          Chinna Rao Lalam added a comment -

          Most of the work is completed for this issue.. I will upload the patch very shortly..

          Show
          Chinna Rao Lalam added a comment - Most of the work is completed for this issue.. I will upload the patch very shortly..
          Xuefu Zhang made changes -
          Assignee Chinna Rao Lalam [ chinnalalam ] Na Yang [ nyang ]
          Hide
          Na Yang added a comment -

          Xuefu, I will start to work on this task. Please re-assign it to me if Chinna is fine with this. Thanks.

          Show
          Na Yang added a comment - Xuefu, I will start to work on this task. Please re-assign it to me if Chinna is fine with this. Thanks.
          Hide
          Xuefu Zhang added a comment -

          Na Yang If Chinna is fine with this, please plan to work on this. The basic work is done, but we'd like to put it in a production quality. Please take TezTask as a reference.

          Show
          Xuefu Zhang added a comment - Na Yang If Chinna is fine with this, please plan to work on this. The basic work is done, but we'd like to put it in a production quality. Please take TezTask as a reference.
          Hide
          Xuefu Zhang added a comment - - edited

          Chinna Rao Lalam, I'm wondering if you're working on this. Hope you don't mind, but I'm assigning this to Na Yang as we need to make progress on this. Let me know if otherwise.

          Show
          Xuefu Zhang added a comment - - edited Chinna Rao Lalam , I'm wondering if you're working on this. Hope you don't mind, but I'm assigning this to Na Yang as we need to make progress on this. Let me know if otherwise.
          Chinna Rao Lalam made changes -
          Assignee Chinna Rao Lalam [ chinnalalam ]
          Xuefu Zhang made changes -
          Field Original Value New Value
          Component/s Spark [ 12323200 ]
          Xuefu Zhang created issue -

            People

            • Assignee:
              Chinna Rao Lalam
              Reporter:
              Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development