Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-409

hive-drop-import-delims doesn't work with MySQL direct import

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0-incubating
    • Fix Version/s: 1.4.1-incubating
    • Component/s: hive-integration
    • Labels:
      None

      Description

      When using hive-drop-import-delims together with direct on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.

      I tried looking at the code but it wasn't easy to follow.

      There seem to be at least two problems:

      • The logic in MySQLDumpMapper.map only looks for MySQLUtils.outputDelimsAreMySQL(conf) to decide if it needs to be reparsed or not, this should also include the hive options
      • This is just a guess at the real problem: The ReparsingStreamThread calls a RecordParser directly without using the created ORM classes and thus circumventing the calls to hiveStringDropDelims

        Attachments

        1. SQOOP-409.patch
          1 kB
          Jarek Jarcec Cecho
        2. SQOOP-409.patch
          1 kB
          Jarek Jarcec Cecho

          Activity

            People

            • Assignee:
              jarcec Jarek Jarcec Cecho
              Reporter:
              lars_francke Lars Francke
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: