Sqoop
  1. Sqoop
  2. SQOOP-409

hive-drop-import-delims doesn't work with MySQL direct import

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0-incubating
    • Fix Version/s: 1.4.1-incubating
    • Component/s: hive-integration
    • Labels:
      None

      Description

      When using hive-drop-import-delims together with direct on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.

      I tried looking at the code but it wasn't easy to follow.

      There seem to be at least two problems:

      • The logic in MySQLDumpMapper.map only looks for MySQLUtils.outputDelimsAreMySQL(conf) to decide if it needs to be reparsed or not, this should also include the hive options
      • This is just a guess at the real problem: The ReparsingStreamThread calls a RecordParser directly without using the created ORM classes and thus circumventing the calls to hiveStringDropDelims
      1. SQOOP-409.patch
        1 kB
        Jarek Jarcec Cecho
      2. SQOOP-409.patch
        1 kB
        Jarek Jarcec Cecho

        Activity

          People

          • Assignee:
            Jarek Jarcec Cecho
            Reporter:
            Lars Francke
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development