[HIVE-18066] Unable to insert data into S3 backed table from Hive

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Hive

Environment

Hive 1.2.0
Hadoop 2.6.0
HDFS
AWS- S3A
Debian jessie

• Assignee: Unassigned
• Reporter: Shreyas Joshi
• Created: 14/Nov/17 21:48
• Updated: 14/Nov/17 21:48

Description

I am trying to create a table and insert data into a table on S3 from Hive like:

create table my_db.test_table (
userid int
) location   's3a://username:pass@bucket/s3-test/test-table'

This works, but when I try inserting data into this table like:

INSERT INTO my_db.test_table VALUES (1);

It fails, and here's what I see:

hive> insert into test values (1);
-chgrp: '' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
Failed with exception Wrong FS: s3a://username:pass@bucket/s3-test/test-table/.hive-staging_hive_2017-11-13_17-37-21_376_4109060838187898328-1/-ext-10002, expected: hdfs://<ip-address>:<port>
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

The hdfs://<ip-address>:<port> in the error seems to be coming from the fs.default.name setting in core-site.xml

If I insert data into the table via other other means, I can successfully query the table from Hive. So, insertion not selection seems to be the problem. What might be happening here?