[SQOOP-2161] Incremental append on to Hive Parquet tables doesn't work

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.5
• Fix Version/s: no-release
• Component/s: hive-integration
• Assignee: Qian Xu
• Reporter: Mark Grover
• Created: 01/Mar/15 06:23
• Updated: 15/Jun/16 19:28
• Resolved: 27/May/15 08:51

Description

I have some code that does incremental append to a Hive parquet table. I am able to get the first time automatic table creation in Hive to work, but subsequent appends return errors. Also, ideally, I would like to create the Hive table explicitly myself and only just do appends to it.

More concretely, the code I am using is at https://gist.github.com/markgrover/86f54663ece0943bc8ed

I am also attaching two verbose error logs. Each of them contain, at the top, the command ran.

new_pre_created_verbose.log contains the error which occurs if I try to import data into an empty Hive parquet table that have been created in Hive. I ran the create table statement in the above gist and it matches the source schema one-to-one.

In order to get past the above error, I don't run the hive create table command explicitly. Sqoop then successfully creates the table and adds data to it. However, on the next run when I want to append more data, I get another error which is detailed in append_pre_created_verbose.log

Issue Links

• relates to: SQOOP-2295 Hive import with Parquet should append automatically