[PIG-2654] LoadFunc and StoreFunc implementations should be instanciated only once in the FE and once in the BE per call in a given script

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Julien Le Dem
• Created: 15/Apr/12 18:46
• Updated: 18/May/12 22:40
• Resolved: 18/May/12 22:39

Description

I'm running the following script:

A = LOAD 'foo' USING Loader();
STORE A INTO 'bar' USING Storer();

Using instrumented Loader an Storer I get the following result:

• 7 instances of Loader
• 6 instances of Storer

each instance gets an ID and logs the calls to its methods:

Loader(1).<init> called by org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:791)
Loader(2).<init> called by org.apache.pig.parser.LogicalPlanBuilder.getAbolutePathForLoad(LogicalPlanBuilder.java:417)
Loader(2).relativeToAbsolutePath called by org.apache.pig.parser.LogicalPlanBuilder.getAbolutePathForLoad(LogicalPlanBuilder.java:421)
Loader(3).<init> called by org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:791)
Loader(4).<init> called by org.apache.pig.parser.LogicalPlanBuilder.getAbolutePathForLoad(LogicalPlanBuilder.java:417)
Storer(1).<init> called by org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:791)
Storer(2).<init> called by org.apache.pig.parser.LogicalPlanBuilder.getAbolutePathForStore(LogicalPlanBuilder.java:478)
Storer(2).setStoreFuncUDFContextSignature called by org.apache.pig.parser.LogicalPlanBuilder.getAbolutePathForStore(LogicalPlanBuilder.java:481)
Storer(2).relToAbsPathForStoreLocation called by org.apache.pig.parser.LogicalPlanBuilder.getAbolutePathForStore(LogicalPlanBuilder.java:484)
Storer(3).<init> called by org.apache.pig.newplan.logical.relational.LOStore.<init>(LOStore.java:55)
Storer(3).setStoreFuncUDFContextSignature called by org.apache.pig.newplan.logical.relational.LOStore.setSignature(LOStore.java:129)
Loader(5).<init> called by org.apache.pig.newplan.logical.relational.LOLoad.getLoadFunc(LOLoad.java:77)
Loader(5).setUDFContextSignature called by org.apache.pig.newplan.logical.relational.LOLoad.getLoadFunc(LOLoad.java:78)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Storer(3).setStoreFuncUDFContextSignature called by org.apache.pig.newplan.logical.relational.LOStore.setSignature(LOStore.java:129)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Storer(3).setStoreLocation called by org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:68)
Storer(3).getOutputFormat called by org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
Loader(5).getSchema called by org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
Loader(5).setLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:389)
Storer(4).<init> called by org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
Storer(4).setStoreFuncUDFContextSignature called by org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:233)
Storer(4).setStoreLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
Storer(5).<init> called by org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
Storer(5).setStoreFuncUDFContextSignature called by org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:233)
Storer(5).setStoreLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
Storer(5).getOutputFormat called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecsHelper(PigOutputFormat.java:203)
Loader(6).<init> called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:254)
Loader(6).setUDFContextSignature called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.passLoadSignature(PigInputFormat.java:187)
Loader(6).setLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:266)
Loader(6).getInputFormat called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:272)
Storer(6).<init> called by org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
Storer(6).setStoreFuncUDFContextSignature called by org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:233)
Storer(6).setStoreLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
Storer(6).getOutputFormat called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:92)
Storer(6).setStoreLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
Loader(7).<init> called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java:158)
Loader(7).setUDFContextSignature called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.passLoadSignature(PigInputFormat.java:187)
Loader(7).setLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:138)
Loader(7).getInputFormat called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:117)
Storer(6).setStoreLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
Storer(6).getOutputFormat called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
Storer(6).prepareToWrite called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.<init>(PigOutputFormat.java:125)
Loader(7).setUDFContextSignature called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.passLoadSignature(PigInputFormat.java:187)
Loader(7).prepareToRead called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:182)
Loader(7).getNext called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:194)
Storer(6).putNext called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
Loader(7).getNext called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:194)
Storer(6).putNext called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
Loader(7).getNext called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:194)
Storer(6).putNext called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
Loader(7).getNext called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:194)
Storer(6).setStoreLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
Storer(6).setStoreLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
Storer(4).setStoreLocation called by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)

Issue Links

• duplicates: PIG-2699 Reduce the number of instances of Load and Store Funcs down to 2+1. It should be 1 in the front-end and 1 in the backend
• is related to: PIG-2344 UDF / LoadFunc / StoreFunc should be serializable