Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2161

Incremental append on to Hive Parquet tables doesn't work

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.5
    • Fix Version/s: no-release
    • Component/s: hive-integration
    • Labels:
      None

      Description

      I have some code that does incremental append to a Hive parquet table. I am able to get the first time automatic table creation in Hive to work, but subsequent appends return errors. Also, ideally, I would like to create the Hive table explicitly myself and only just do appends to it.

      More concretely, the code I am using is at https://gist.github.com/markgrover/86f54663ece0943bc8ed

      I am also attaching two verbose error logs. Each of them contain, at the top, the command ran.

      new_pre_created_verbose.log contains the error which occurs if I try to import data into an empty Hive parquet table that have been created in Hive. I ran the create table statement in the above gist and it matches the source schema one-to-one.

      In order to get past the above error, I don't run the hive create table command explicitly. Sqoop then successfully creates the table and adds data to it. However, on the next run when I want to append more data, I get another error which is detailed in append_pre_created_verbose.log

        Attachments

        1. append_pre_created_verbose.log
          199 kB
          Mark Grover
        2. new_pre_created_verbose.log
          174 kB
          Mark Grover

          Issue Links

            Activity

              People

              • Assignee:
                stanleyxu2005 Qian Xu
                Reporter:
                mgrover Mark Grover
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: