Sqoop
  1. Sqoop
  2. SQOOP-824

Sqoop code generation in 'update' export mode incompatible with '--columns' option

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.3
    • Component/s: codegen, tools
    • Labels:
    • Environment:

      Description

      When using sqoop export with --update-key and --columns the options are incompatible. For example, given table T1

        +------+
        | T1   |
        +------+
        | id   |
        | name |
        | mail |
        | cell |
        +-------
      

      When using:

      $ sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name
      

      the code generation utility creates a Java file which fails to compile.

      This seems to happen because ConnManager appends all table columns, even those not mentioned on the command-line (see ConnManager.configureDbOutputColumns). As a result, the generated class contains a write method which references class members which do not exist.

      I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

      I may be missing something as that code is there for a reason, but I can't see why?

      1. bugSQOOP-824.patch
        5 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Stan Angeloff created issue -
          Stan Angeloff made changes -
          Field Original Value New Value
          Description When using the {{sqoop export}} with {{\-\-update-key}} and {{--columns}} the options are incompatible. For example, given table {{T1}}

          {code}
          +------+
          | T1 |
          +------+
          | id |
          | name |
          | mail |
          | cell |
          +-------
          {code}

          When using {{sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name}} the code generation utility creates a Java file which fails to compile.

          This seems to happen because {{ConnManager}} appends all table columns, even those not mentioned on the command-line (see {{ConnManager.configureDbOutputColumns}}. As a result, the generated class contains a {{write}} method which references class members which do not exist.

          I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

          I may be missing something as that code is there for a reason, but I can't see why?
          When using {{sqoop export}} with {{\-\-update-key}} and {{--columns}} the options are incompatible. For example, given table {{T1}}

          {code}
          +------+
          | T1 |
          +------+
          | id |
          | name |
          | mail |
          | cell |
          +-------
          {code}

          When using {{sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name}} the code generation utility creates a Java file which fails to compile.

          This seems to happen because {{ConnManager}} appends all table columns, even those not mentioned on the command-line (see {{ConnManager.configureDbOutputColumns}}. As a result, the generated class contains a {{write}} method which references class members which do not exist.

          I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

          I may be missing something as that code is there for a reason, but I can't see why?
          Stan Angeloff made changes -
          Description When using {{sqoop export}} with {{\-\-update-key}} and {{--columns}} the options are incompatible. For example, given table {{T1}}

          {code}
          +------+
          | T1 |
          +------+
          | id |
          | name |
          | mail |
          | cell |
          +-------
          {code}

          When using {{sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name}} the code generation utility creates a Java file which fails to compile.

          This seems to happen because {{ConnManager}} appends all table columns, even those not mentioned on the command-line (see {{ConnManager.configureDbOutputColumns}}. As a result, the generated class contains a {{write}} method which references class members which do not exist.

          I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

          I may be missing something as that code is there for a reason, but I can't see why?
          When using {{sqoop export}} with {{\-\-update-key}} and {{--columns}} the options are incompatible. For example, given table {{T1}}

          {code}
            +------+
            | T1 |
            +------+
            | id |
            | name |
            | mail |
            | cell |
            +-------
          {code}

          When using {{sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name}} the code generation utility creates a Java file which fails to compile.

          This seems to happen because {{ConnManager}} appends all table columns, even those not mentioned on the command-line (see {{ConnManager.configureDbOutputColumns}}. As a result, the generated class contains a {{write}} method which references class members which do not exist.

          I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

          I may be missing something as that code is there for a reason, but I can't see why?
          Stan Angeloff made changes -
          Description When using {{sqoop export}} with {{\-\-update-key}} and {{--columns}} the options are incompatible. For example, given table {{T1}}

          {code}
            +------+
            | T1 |
            +------+
            | id |
            | name |
            | mail |
            | cell |
            +-------
          {code}

          When using {{sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name}} the code generation utility creates a Java file which fails to compile.

          This seems to happen because {{ConnManager}} appends all table columns, even those not mentioned on the command-line (see {{ConnManager.configureDbOutputColumns}}. As a result, the generated class contains a {{write}} method which references class members which do not exist.

          I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

          I may be missing something as that code is there for a reason, but I can't see why?
          When using {{sqoop export}} with {{\-\-update-key}} and {{--columns}} the options are incompatible. For example, given table {{T1}}

          {code}
            +------+
            | T1 |
            +------+
            | id |
            | name |
            | mail |
            | cell |
            +-------
          {code}

          When using:

          {code}
          $ sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name
          {code}

          the code generation utility creates a Java file which fails to compile.

          This seems to happen because {{ConnManager}} appends all table columns, even those not mentioned on the command-line (see {{ConnManager.configureDbOutputColumns}}. As a result, the generated class contains a {{write}} method which references class members which do not exist.

          I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

          I may be missing something as that code is there for a reason, but I can't see why?
          Stan Angeloff made changes -
          Description When using {{sqoop export}} with {{\-\-update-key}} and {{--columns}} the options are incompatible. For example, given table {{T1}}

          {code}
            +------+
            | T1 |
            +------+
            | id |
            | name |
            | mail |
            | cell |
            +-------
          {code}

          When using:

          {code}
          $ sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name
          {code}

          the code generation utility creates a Java file which fails to compile.

          This seems to happen because {{ConnManager}} appends all table columns, even those not mentioned on the command-line (see {{ConnManager.configureDbOutputColumns}}. As a result, the generated class contains a {{write}} method which references class members which do not exist.

          I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

          I may be missing something as that code is there for a reason, but I can't see why?
          When using {{sqoop export}} with {{\-\-update-key}} and {{--columns}} the options are incompatible. For example, given table {{T1}}

          {code}
            +------+
            | T1 |
            +------+
            | id |
            | name |
            | mail |
            | cell |
            +-------
          {code}

          When using:

          {code}
          $ sqoop export [ .. ] --update-key id --update-mode allowinsert --columns id,name
          {code}

          the code generation utility creates a Java file which fails to compile.

          This seems to happen because {{ConnManager}} appends all table columns, even those not mentioned on the command-line (see {{ConnManager.configureDbOutputColumns}}). As a result, the generated class contains a {{write}} method which references class members which do not exist.

          I could suggest two possible solutions: 1) either make this case an exception or 2) make the code that appends all remaining table columns optional.

          I may be missing something as that code is there for a reason, but I can't see why?
          Jarek Jarcec Cecho made changes -
          Remote Link This issue links to "Review board (Web Link)" [ 11963 ]
          Jarek Jarcec Cecho made changes -
          Attachment bugSQOOP-824.patch [ 12564163 ]
          Jarek Jarcec Cecho made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 1.4.3 [ 12322644 ]
          Jarek Jarcec Cecho made changes -
          Assignee Jarek Jarcec Cecho [ jarcec ]
          Abhijeet Gaikwad made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Stan Angeloff
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development