[PIG-5068] Set SPARK_REDUCERS by pig.properties not by system configuration

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: liyunzhang
• Reporter: liyunzhang
• Created: 23/Nov/16 06:13
• Updated: 21/Jun/17 09:18
• Resolved: 29/Nov/16 04:02

Description

In SparkUtil.java, we set the SPARK_REDUCERS by system configuration

    public static int getParallelism(List<RDD<Tuple>> predecessors,
            PhysicalOperator physicalOperator) {

        String numReducers = System.getenv("SPARK_REDUCERS");
        if (numReducers != null) {
            return Integer.parseInt(numReducers);
        }

        int parallelism = physicalOperator.getRequestedParallelism();
        if (parallelism <= 0) {
            // Parallelism wasn't set in Pig, so set it to whatever Spark thinks
            // is reasonable.
            parallelism = predecessors.get(0).context().defaultParallelism();
        }

        return parallelism;
    }

It is better to set it by pig.properties