[SPARK-24176] The hdfs file path with wildcard can not be identified when loading data

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.3.0
• Component/s: SQL

Environment

OS: SUSE11

Spark Version:2.3

• Assignee: Unassigned
• Reporter: ABHISHEK KUMAR GUPTA
• Created: 04/May/18 05:22
• Updated: 28/Nov/18 04:55
• Resolved: 28/Nov/18 04:55

Description

1. Launch spark-sql
2. create table wild1 (time timestamp, name string, isright boolean, datetoday date, num binary, height double, score float, decimaler decimal(10,0), id tinyint, age int, license bigint, length smallint) row format delimited fields terminated by ',' stored as textfile;
3. loaded data in table as below and it failed some cases not consistent
4. load data inpath '/user/testdemo1/user1/?ype* ' into table wild1; - Success
   load data inpath '/user/testdemo1/user1/t??eddata60.txt' into table wild1; - Failed
   load data inpath '/user/testdemo1/user1/?ypeddata60.txt' into table wild1; - Success

Exception as below

> load data inpath '/user/testdemo1/user1/t??eddata61.txt' into table wild1;
2018-05-04 13:16:25 INFO HiveMetaStore:746 - 0: get_database: one
2018-05-04 13:16:25 INFO audit:371 - ugi=spark/[e-mail] ip=unknown-ip-addr cmd=get_database: one
2018-05-04 13:16:25 INFO HiveMetaStore:746 - 0: get_table : db=one tbl=wild1
2018-05-04 13:16:25 INFO audit:371 - ugi=spark/[e-mail] ip=unknown-ip-addr cmd=get_table : db=one tbl=wild1
2018-05-04 13:16:25 INFO HiveMetaStore:746 - 0: get_table : db=one tbl=wild1
2018-05-04 13:16:25 INFO audit:371 - ugi=spark/[e-mail] ip=unknown-ip-addr cmd=get_table : db=one tbl=wild1
Error in query: LOAD DATA input path does not exist: /user/testdemo1/user1/t??eddata61.txt;
spark-sql>

Behavior is not consistent. Need to fix with all combination of wild card char as it is not consistent.

Issue Links

• links to: [Github] Pull Request #21285 (kevinyu98)