[PIG-2478] Using Illustrate with Javascript UDF throws NEP

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.11

Environment

Mac OSX Lion 10.7.2, Hadoop (0.20.205.0), Pig (Apache Pig version 0.11.0-SNAPSHOT (r1230844).

• Assignee: Unassigned
• Reporter: dan young
• Created: 17/Jan/12 22:23
• Updated: 20/Jun/12 16:59

Description

When I try to run illustrate while using a Javascript UDF, I always get a NEP....but if I do a dump/store, everything works fine.

dans-MacBook-Pro:pig danoyoung$ pig -x local
2012-01-12 22:16:57,131 [main] INFO org.apache.pig.Main - Logging error messages to: /Users/danoyoung/projects/pig/pig_1326431817128.log
2012-01-12 22:16:57,386 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2012-01-12 22:16:57.483 java[27809:1903] Unable to load realm info from SCDynamicStore
grunt> set io.sort.mb 500;
grunt> register '/Users/danoyoung/projects/pig/udf/udf.js' using org.apache.pig.scripting.js.JsScriptEngine as myfuncs;
2012-01-12 22:17:24,803 [main] INFO org.apache.pig.scripting.js.JsScriptEngine - Register scripting UDF: get_date_marker
2012-01-12 22:17:24,804 [main] INFO org.apache.pig.scripting.js.JsScriptEngine - Register scripting UDF: get_record
grunt> register '/usr/local/pig/piggybank.jar';
grunt>
grunt> a = LOAD '/Users/danoyoung/Downloads/adwords_KeywordDailyReport_1-1-2012_daily' USING org.apache.pig.piggybank.storage.XMLLoader('row') AS (doc:chararray);
grunt>
grunt> b = FOREACH a GENERATE FLATTEN(myfuncs.get_record(doc)) AS (dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float);
grunt> describe b;
b:

{dw_date_marker: int,ad_network_ad_group_key: long,ad_network_keyword_key: long,firstpagecpc: int,qualityscore: int,cost: float,position: float}

grunt>
grunt> illustrate b;
2012-01-12 22:17:25,642 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2012-01-12 22:17:25,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2012-01-12 22:17:25,860 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2012-01-12 22:17:25,860 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2012-01-12 22:17:25,870 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2012-01-12 22:17:25,887 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-01-12 22:17:26,161 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2012-01-12 22:17:26,721 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2012-01-12 22:17:26,723 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2012-01-12 22:17:26,723 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2012-01-12 22:17:26,724 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2012-01-12 22:17:26,725 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
java.lang.NullPointerException
at org.apache.pig.scripting.js.JsFunction.exec(JsFunction.java:200)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:225)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:262)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194)
at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
at org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
at org.apache.pig.PigServer.getExamples(PigServer.java:1202)
at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:523)
at org.apache.pig.Main.main(Main.java:148)
2012-01-12 22:17:26,767 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. Exception : null
Details at logfile: /Users/danoyoung/projects/pig/pig_1326431817128.log
grunt>

Here's the log details:
Pig Stack Trace
---------------
ERROR 2997: Encountered IOException. Exception : null

java.io.IOException: Exception : null
at org.apache.pig.PigServer.getExamples(PigServer.java:1208)
at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:523)
at org.apache.pig.Main.main(Main.java:148)
================================================================================

Here's the Javascript UDF:

get_record.outputSchema = "dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float";
function get_record(doc){
var rec = new Object;
rec.dt = doc.substring(doc.search(/date=/ig)+6,doc.search(/\" campaignid=/ig));
//rec.campaign_id = doc.substring(doc.search(/campaignid=/ig)+12,doc.search(/\" adgroupid=/ig));
rec.ad_network_ad_group_key = doc.substring(doc.search(/adgroupid=/ig)+11,doc.search(/\" keywordid=/ig));
rec.ad_network_keyword_key = doc.substring(doc.search(/keywordid=/ig)+11,doc.search(/\" keyword=/ig));
rec.firstpagecpc = doc.substring(doc.search(/firstpagecpc=/ig)+14,doc.search(/\" qualityscore=/ig))/1000000;
rec.qualityscore = doc.substring(doc.search(/qualityscore=/ig)+14,doc.search(/\" imps=/ig));
rec.cost = doc.substring(doc.search(/cost=/ig)+6,doc.search(/\" pos=/ig))/1000000;
rec.position = doc.substring(doc.search(/pos=/ig)+5,doc.search(/\"><\/row/ig));
rec.dw_date_marker = get_date_marker(rec.dt);

return

{dw_date_marker:rec.dw_date_marker, ad_network_ad_group_key:rec.ad_network_ad_group_key, ad_network_keyword_key:rec.ad_network_keyword_key, firstpagecpc:rec.firstpagecpc, qualityscore:rec.qualityscore, cost:rec.cost, position:rec.position}

;

}

function get_date_marker(dt){
var dateMarkers =

{ _2012_01_01:1204, _2012_01_02:1205, _2012_01_03:1206, _2012_01_04:1207, _2012_01_05:1208, _2012_01_06:1209, _2012_01_07:1210, _2012_01_08:1211, _2012_01_09:1212, _2012_01_10:1213, _2012_01_11:1214, _2012_01_12:1215, _2012_01_13:1216, _2012_01_14:1217, _2012_01_15:1218, _2012_01_16:1219, _2012_01_17:1220, _2012_01_18:1221, _2012_01_19:1222, _2012_01_20:1223, _2012_01_21:1224, _2012_01_22:1225, _2012_01_23:1226, _2012_01_24:1227, _2012_01_25:1228, _2012_01_26:1229, _2012_01_27:1230, _2012_01_28:1231, _2012_01_29:1232, _2012_01_30:1233, _2012_01_31:1234, _2012_02_01:1235, _2012_02_02:1236, _2012_02_03:1237, _2012_02_04:1238, _2012_02_05:1239, _2012_02_06:1240, _2012_02_07:1241, _2012_02_08:1242, _2012_02_09:1243, _2012_02_10:1244, _2012_02_11:1245, _2012_02_12:1246, _2012_02_13:1247, _2012_02_14:1248, _2012_02_15:1249, _2012_02_16:1250, _2012_02_17:1251, _2012_02_18:1252, _2012_02_19:1253, _2012_02_20:1254, _2012_02_21:1255, _2012_02_22:1256, _2012_02_23:1257, _2012_02_24:1258, _2012_02_25:1259, _2012_02_26:1260, _2012_02_27:1261, _2012_02_28:1262, _2012_02_29:1263, _2012_03_01:1264, _2012_03_02:1265, _2012_03_03:1266, _2012_03_04:1267, _2012_03_05:1268, _2012_03_06:1269, _2012_03_07:1270, _2012_03_08:1271, _2012_03_09:1272, _2012_03_10:1273, _2012_03_11:1274, _2012_03_12:1275, _2012_03_13:1276, _2012_03_14:1277, _2012_03_15:1278, _2012_03_16:1279, _2012_03_17:1280, _2012_03_18:1281, _2012_03_19:1282, _2012_03_20:1283, _2012_03_21:1284, _2012_03_22:1285, _2012_03_23:1286, _2012_03_24:1287, _2012_03_25:1288, _2012_03_26:1289, _2012_03_27:1290, _2012_03_28:1291, _2012_03_29:1292, _2012_03_30:1293, _2012_03_31:1294, _2012_04_01:1295, _2012_04_02:1296, _2012_04_03:1297, _2012_04_04:1298, _2012_04_05:1299, _2012_04_06:1300, _2012_04_07:1301, _2012_04_08:1302, _2012_04_09:1303, _2012_04_10:1304, _2012_04_11:1305, _2012_04_12:1306, _2012_04_13:1307, _2012_04_14:1308, _2012_04_15:1309, _2012_04_16:1310, _2012_04_17:1311, _2012_04_18:1312, _2012_04_19:1313, _2012_04_20:1314, _2012_04_21:1315, _2012_04_22:1316, _2012_04_23:1317, _2012_04_24:1318, _2012_04_25:1319, _2012_04_26:1320, _2012_04_27:1321, _2012_04_28:1322, _2012_04_29:1323, _2012_04_30:1324, _2012_05_01:1325, _2012_05_02:1326, _2012_05_03:1327, _2012_05_04:1328, _2012_05_05:1329, _2012_05_06:1330, _2012_05_07:1331, _2012_05_08:1332, _2012_05_09:1333, _2012_05_10:1334, _2012_05_11:1335, _2012_05_12:1336, _2012_05_13:1337, _2012_05_14:1338, _2012_05_15:1339, _2012_05_16:1340, _2012_05_17:1341, _2012_05_18:1342, _2012_05_19:1343, _2012_05_20:1344, _2012_05_21:1345, _2012_05_22:1346, _2012_05_23:1347, _2012_05_24:1348, _2012_05_25:1349, _2012_05_26:1350, _2012_05_27:1351, _2012_05_28:1352, _2012_05_29:1353, _2012_05_30:1354, _2012_05_31:1355, _2012_06_01:1356, _2012_06_02:1357, _2012_06_03:1358, _2012_06_04:1359, _2012_06_05:1360, _2012_06_06:1361, _2012_06_07:1362, _2012_06_08:1363, _2012_06_09:1364, _2012_06_10:1365, _2012_06_11:1366, _2012_06_12:1367, _2012_06_13:1368, _2012_06_14:1369, _2012_06_15:1370, _2012_06_16:1371, _2012_06_17:1372, _2012_06_18:1373, _2012_06_19:1374, _2012_06_20:1375, _2012_06_21:1376, _2012_06_22:1377, _2012_06_23:1378, _2012_06_24:1379, _2012_06_25:1380, _2012_06_26:1381, _2012_06_27:1382, _2012_06_28:1383, _2012_06_29:1384, _2012_06_30:1385, _2012_07_01:1386, _2012_07_02:1387, _2012_07_03:1388, _2012_07_04:1389, _2012_07_05:1390, _2012_07_06:1391, _2012_07_07:1392, _2012_07_08:1393, _2012_07_09:1394, _2012_07_10:1395, _2012_07_11:1396, _2012_07_12:1397, _2012_07_13:1398, _2012_07_14:1399, _2012_07_15:1400, _2012_07_16:1401, _2012_07_17:1402, _2012_07_18:1403, _2012_07_19:1404, _2012_07_20:1405, _2012_07_21:1406, _2012_07_22:1407, _2012_07_23:1408, _2012_07_24:1409, _2012_07_25:1410, _2012_07_26:1411, _2012_07_27:1412, _2012_07_28:1413, _2012_07_29:1414, _2012_07_30:1415, _2012_07_31:1416, _2012_08_01:1417, _2012_08_02:1418, _2012_08_03:1419, _2012_08_04:1420, _2012_08_05:1421, _2012_08_06:1422, _2012_08_07:1423, _2012_08_08:1424, _2012_08_09:1425, _2012_08_10:1426, _2012_08_11:1427, _2012_08_12:1428, _2012_08_13:1429, _2012_08_14:1430, _2012_08_15:1431, _2012_08_16:1432, _2012_08_17:1433, _2012_08_18:1434, _2012_08_19:1435, _2012_08_20:1436, _2012_08_21:1437, _2012_08_22:1438, _2012_08_23:1439, _2012_08_24:1440, _2012_08_25:1441, _2012_08_26:1442, _2012_08_27:1443, _2012_08_28:1444, _2012_08_29:1445, _2012_08_30:1446, _2012_08_31:1447, _2012_09_01:1448, _2012_09_02:1449, _2012_09_03:1450, _2012_09_04:1451, _2012_09_05:1452, _2012_09_06:1453, _2012_09_07:1454, _2012_09_08:1455, _2012_09_09:1456, _2012_09_10:1457, _2012_09_11:1458, _2012_09_12:1459, _2012_09_13:1460, _2012_09_14:1461, _2012_09_15:1462, _2012_09_16:1463, _2012_09_17:1464, _2012_09_18:1465, _2012_09_19:1466, _2012_09_20:1467, _2012_09_21:1468, _2012_09_22:1469, _2012_09_23:1470, _2012_09_24:1471, _2012_09_25:1472, _2012_09_26:1473, _2012_09_27:1474, _2012_09_28:1475, _2012_09_29:1476, _2012_09_30:1477, _2012_10_01:1478, _2012_10_02:1479, _2012_10_03:1480, _2012_10_04:1481, _2012_10_05:1482, _2012_10_06:1483, _2012_10_07:1484, _2012_10_08:1485, _2012_10_09:1486, _2012_10_10:1487, _2012_10_11:1488, _2012_10_12:1489, _2012_10_13:1490, _2012_10_14:1491, _2012_10_15:1492, _2012_10_16:1493, _2012_10_17:1494, _2012_10_18:1495, _2012_10_19:1496, _2012_10_20:1497, _2012_10_21:1498, _2012_10_22:1499, _2012_10_23:1500, _2012_10_24:1501, _2012_10_25:1502, _2012_10_26:1503, _2012_10_27:1504, _2012_10_28:1505, _2012_10_29:1506, _2012_10_30:1507, _2012_10_31:1508, _2012_11_01:1509, _2012_11_02:1510, _2012_11_03:1511, _2012_11_04:1512, _2012_11_05:1513, _2012_11_06:1514, _2012_11_07:1515, _2012_11_08:1516, _2012_11_09:1517, _2012_11_10:1518, _2012_11_11:1519, _2012_11_12:1520, _2012_11_13:1521, _2012_11_14:1522, _2012_11_15:1523, _2012_11_16:1524, _2012_11_17:1525, _2012_11_18:1526, _2012_11_19:1527, _2012_11_20:1528, _2012_11_21:1529, _2012_11_22:1530, _2012_11_23:1531, _2012_11_24:1532, _2012_11_25:1533, _2012_11_26:1534, _2012_11_27:1535, _2012_11_28:1536, _2012_11_29:1537, _2012_11_30:1538, _2012_12_01:1539, _2012_12_02:1540, _2012_12_03:1541, _2012_12_04:1542, _2012_12_05:1543, _2012_12_06:1544, _2012_12_07:1545, _2012_12_08:1546, _2012_12_09:1547, _2012_12_10:1548, _2012_12_11:1549, _2012_12_12:1550, _2012_12_13:1551, _2012_12_14:1552, _2012_12_15:1553, _2012_12_16:1554, _2012_12_17:1555, _2012_12_18:1556, _2012_12_19:1557, _2012_12_20:1558, _2012_12_21:1559, _2012_12_22:1560, _2012_12_23:1561, _2012_12_24:1562, _2012_12_25:1563, _2012_12_26:1564, _2012_12_27:1565, _2012_12_28:1566, _2012_12_29:1567, _2012_12_30:1568, _2012_12_31:1569}

;

var rec = new Object;
rec.dt = dt.replace(/-/g, "_");
rec.date_marker = dateMarkers["_" + rec.dt];
return rec.date_marker;
}