Affects Version/s: None
Fix Version/s: 0.11
Currently Pig doesn't support lazy see/de, so as one of the best practices, we recommend users not to declare types in the schema so that Pig will guess the right types and cast them lazily. However, if Pig guesses a wrong type, especially mistakes a double field as an integer field, the overhead of casting is tremendous due to the exception handling.
See Utf8StorageConverter#bytesToIntege. It first casts bytes to Integer by Integer.parseInt(), and if exception occurs, it tries to cast it to Double by Double.parseDouble() and convert it back to Integer. The problem is that the exception handling can be 10x slower than the actual casting. bytesToLong has the same problem. Below is a mini-benchmark:
And the results:
|Integer.parseInt(i + ".0");||118|
|Integer.parseInt(i + "");||13|
|Double.parseDouble(i + "");||16|
We can see Integer.parseInt(i + ".0") is 10x slower than the other due to the exception handling.
This issue was found when I benchmark TPC-H Query 1, for which Pig was terribly slower than Hive:
After declaring three double fields as double, the performance was boosted.
|pig without types||pig with three doubles||hive|
|76 min||34 min||16 min|
Besides recommending users to declare actual double fields as double, we can also improve the casting to avoid this happening. Maybe the easiest way is to remove the Integer.parseInt and only use the Double.parseDouble and convert back to Integer. The mini benchmark above shows Double.parseDouble + range checking + Integer.valueOf(Double.intValue()) takes about 17 seconds. I think the small percent of extra overhead (3 seconds compared to Integer.parseInt()) is acceptable as it won't be the dominant bottleneck?
|Status||Open [ 1 ]||Patch Available [ 10002 ]|
|Status||Patch Available [ 10002 ]||Resolved [ 5 ]|
|Fix Version/s||0.11 [ 12318878 ]|
|Resolution||Fixed [ 1 ]|
|Assignee||Jie Li [ jay23jack ]|
|Status||Resolved [ 5 ]||Closed [ 6 ]|
|Transition||Time In Source Status||Execution Times||Last Executer||Last Execution Date|
|19h 20m||1||Jie Li||24/Jul/12 17:34|
|40d 5h 3m||1||Dmitriy V. Ryaboy||02/Sep/12 22:38|
|172d 6h 16m||1||Bill Graham||22/Feb/13 04:54|