[PIG-4807] Fix test cases of "TestEvalPipelineLocal" test suite.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: spark-branch
• Fix Version/s: spark-branch
• Component/s: spark

Release Note

Following test cases have been fixed -
1. org.apache.pig.test.TestEvalPipelineLocal.testSetLocationCalledInFE
2. org.apache.pig.test.TestEvalPipelineLocal.testExplainInDotGraph
3. org.apache.pig.test.TestEvalPipelineLocal.testSortWithUDF

1 was failing because of not saving UDF_CONTEXT configuration in jobConf. This leads UDFContext.getUDFProperties() to return NULL.

public Properties getUDFProperties(Class c) {
    UDFContextKey k = generateKey(c, null);
    Properties p = udfConfs.get(k);
    if (p == null) {
        p = new Properties();
        udfConfs.put(k, p);
    }
    return p;
}

Here, udfConfs remains empty even when it was set while processing the pig query.
udf configuration in jobConf is getting lost while running the job.
In the code udf configuration is meant to be saved by serializing them in jobConf.

Currently, serialization is done before loading configuration in jobConf.
It is done in 'newJobConf(PigContext pigContext)'
It needs to be done after loading configuration.

JobConf jobConf = SparkUtil.newJobConf(pigContext);
configureLoader(physicalPlan, op, jobConf);
UDFContext.getUDFContext().serialize(jobConf);

2 was failing because of pig-spark not supporting 'explain' in dot format. I have added the DotSparkPrinter to fix the same.

3 was failing because instead of UDFSortComparator, SortConveter class was using SortComparator.

JavaPairRDD<Tuple, Object> sorted = r.sortByKey(
                sortOperator.new SortComparator(), true);

It should be using mComparator stored in POSort class. I have changed it to following

JavaPairRDD<Tuple, Object> sorted = r.sortByKey(
                sortOperator.getMComparator(), true);

• Assignee: Prateek Vaishnav
• Reporter: Prateek Vaishnav
• Created: 12/Feb/16 06:48
• Updated: 21/Jun/17 09:18
• Resolved: 24/Feb/16 14:19

Description

This jira is created to address the failure of test cases
org.apache.pig.test.TestEvalPipelineLocal.testSetLocationCalledInFE
org.apache.pig.test.TestEvalPipelineLocal.testExplainInDotGraph
org.apache.pig.test.TestEvalPipelineLocal.testSortWithUDF