[HIVE-20839] "Cannot find field" error during dynamically partitioned hash join

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Query Processor
• Assignee: Jason Dere
• Reporter: Jason Dere
• Created: 30/Oct/18 17:19
• Updated: 17/Nov/22 08:55
• Resolved: 05/Nov/18 21:44

Description

Occurs in some cases in the non-CBO optimized queries, either if CBO is disabled or has failed due to error.

2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2 (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error while closing operators - failing tree
2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2 (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor: java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1, 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9, 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17, 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24, 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31, 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38, 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45, 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52, 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59, 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66, 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75, 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82, 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89, 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96, 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102, 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108, 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114, 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120, 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126, 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132, 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138, 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144, 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150, 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156, 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162, 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168, 166:_col169, 167:_col170, 168:_col171, 169:_col318]
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
at org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
at org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)

Issue Links

• is blocked by: HIVE-20833 package.jdo needs to be updated to conform with HIVE-20221 changes
• is related to: HIVE-10673 Dynamically partitioned hash join for Tez
• relates to: HIVE-21107 Cannot find field" error during dynamically partitioned hash join