Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.4
    • Component/s: None
    • Labels:
      None

      Description

      Need to provide Windows version of scripts to run Sqoop on Windows. This includes sqoop.cmd and configure-sqooop.cmd

      1. SQOOP-954.trunk.4.patch
        7 kB
        Ahmed El Baz
      2. SQOOP-954.trunk.2.patch
        13 kB
        Ahmed El Baz
      3. SQOOP-954.trunk.1.patch
        13 kB
        Ahmed El Baz

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #632 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/632/)
          SQOOP-954: Create Sqoop runtime scripts to run Sqoop on Windows (Revision ed2a9036345d03c35cad9071d7e19c8d825f00c3)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed2a9036345d03c35cad9071d7e19c8d825f00c3
          Files :

          • conf/sqoop-env-template.cmd
          • bin/sqoop.cmd
          • bin/configure-sqoop.cmd
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #632 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/632/ ) SQOOP-954 : Create Sqoop runtime scripts to run Sqoop on Windows (Revision ed2a9036345d03c35cad9071d7e19c8d825f00c3) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed2a9036345d03c35cad9071d7e19c8d825f00c3 Files : conf/sqoop-env-template.cmd bin/sqoop.cmd bin/configure-sqoop.cmd
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #833 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/833/)
          SQOOP-954: Create Sqoop runtime scripts to run Sqoop on Windows (Revision ed2a9036345d03c35cad9071d7e19c8d825f00c3)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed2a9036345d03c35cad9071d7e19c8d825f00c3
          Files :

          • bin/sqoop.cmd
          • conf/sqoop-env-template.cmd
          • bin/configure-sqoop.cmd
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #833 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/833/ ) SQOOP-954 : Create Sqoop runtime scripts to run Sqoop on Windows (Revision ed2a9036345d03c35cad9071d7e19c8d825f00c3) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed2a9036345d03c35cad9071d7e19c8d825f00c3 Files : bin/sqoop.cmd conf/sqoop-env-template.cmd bin/configure-sqoop.cmd
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #624 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/624/)
          SQOOP-954: Create Sqoop runtime scripts to run Sqoop on Windows (Revision ed2a9036345d03c35cad9071d7e19c8d825f00c3)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed2a9036345d03c35cad9071d7e19c8d825f00c3
          Files :

          • conf/sqoop-env-template.cmd
          • bin/configure-sqoop.cmd
          • bin/sqoop.cmd
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #624 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/624/ ) SQOOP-954 : Create Sqoop runtime scripts to run Sqoop on Windows (Revision ed2a9036345d03c35cad9071d7e19c8d825f00c3) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed2a9036345d03c35cad9071d7e19c8d825f00c3 Files : conf/sqoop-env-template.cmd bin/configure-sqoop.cmd bin/sqoop.cmd
          Hide
          Jarek Jarcec Cecho added a comment -

          The failure in profile 200 seems to be some sort of issue on Jenkins side, will rerun the job.

          Show
          Jarek Jarcec Cecho added a comment - The failure in profile 200 seems to be some sort of issue on Jenkins side, will rerun the job.
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #625 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/625/)
          SQOOP-954: Create Sqoop runtime scripts to run Sqoop on Windows (Revision ed2a9036345d03c35cad9071d7e19c8d825f00c3)

          Result = FAILURE
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed2a9036345d03c35cad9071d7e19c8d825f00c3
          Files :

          • bin/sqoop.cmd
          • conf/sqoop-env-template.cmd
          • bin/configure-sqoop.cmd
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #625 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/625/ ) SQOOP-954 : Create Sqoop runtime scripts to run Sqoop on Windows (Revision ed2a9036345d03c35cad9071d7e19c8d825f00c3) Result = FAILURE jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed2a9036345d03c35cad9071d7e19c8d825f00c3 Files : bin/sqoop.cmd conf/sqoop-env-template.cmd bin/configure-sqoop.cmd
          Hide
          Jarek Jarcec Cecho added a comment -
          Show
          Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=ed2a9036345d03c35cad9071d7e19c8d825f00c3 Thank you Ahmed for your contribution! Jarcec
          Hide
          Ahmed El Baz added a comment -

          Uploading the patch after addressing all comments from the review board.

          Thanks,
          Ahmed

          Show
          Ahmed El Baz added a comment - Uploading the patch after addressing all comments from the review board. Thanks, Ahmed
          Hide
          Ahmed El Baz added a comment -

          Responding to review comments:
          1) Removed trailing spaces
          2) Made an early check for the JDK path and exiting with error if not found

          Show
          Ahmed El Baz added a comment - Responding to review comments: 1) Removed trailing spaces 2) Made an early check for the JDK path and exiting with error if not found
          Hide
          Ahmed El Baz added a comment -

          Here is the link for review board for the patch

          Show
          Ahmed El Baz added a comment - Here is the link for review board for the patch
          Hide
          Ahmed El Baz added a comment -

          Thank you Venkat.
          Yes this is correct. This will require documentation. I have added a sub-task for updating Sqoop docs for Windows.

          Please let me know if you have any comments on this patch.

          Thanks,

          Ahmed

          Show
          Ahmed El Baz added a comment - Thank you Venkat. Yes this is correct. This will require documentation. I have added a sub-task for updating Sqoop docs for Windows. Please let me know if you have any comments on this patch. Thanks, Ahmed
          Hide
          Jarek Jarcec Cecho added a comment -

          Ahmed El Baz, would you mind uploading the patch to review board?

          Show
          Jarek Jarcec Cecho added a comment - Ahmed El Baz , would you mind uploading the patch to review board?
          Hide
          Venkat Ranganathan added a comment -

          Hi Ahmed El Baz

          I had a quick question. You are using Powershell scripts. Is this an additional requirement for users on Pre-windows 7 and Windows Server 2008 systems?

          It looks like there will be some documentation impact on creation of the jar file here. I was wondering if it is somehow possible to provide some of the options in an option file that Sqoop supports. In either case, I think we may need one additional JIRA for documentation of Sqoop on windows.

          Thanks

          Venkat

          Show
          Venkat Ranganathan added a comment - Hi Ahmed El Baz I had a quick question. You are using Powershell scripts. Is this an additional requirement for users on Pre-windows 7 and Windows Server 2008 systems? It looks like there will be some documentation impact on creation of the jar file here. I was wondering if it is somehow possible to provide some of the options in an option file that Sqoop supports. In either case, I think we may need one additional JIRA for documentation of Sqoop on windows. Thanks Venkat
          Hide
          Ahmed El Baz added a comment -

          Attaching a patch implementing the Windows version of Sqoop run scripts. The scripts follow the same logic as there .sh counterparts.
          One difference is to create a Jar which references all classpath elements in its Manifest, and provide that jar as the single jar needed for Sqoop. The reason here is that in some cases if the number of classpath elements is large, HADOOP_CLASSPATH gets very long which causes failures in Windows since there is a limit to command lines.
          As a workaround, I added a step to wrap all jars in the classpath in a single jar, and then use that generated jar (this is also done in hadoop for Windows to handle similar issues)
          I did this in a utility script "BuildJar" which can be used for other components as well.
          This change is specific to Windows scripts, Linux scripts are not affected.

          Show
          Ahmed El Baz added a comment - Attaching a patch implementing the Windows version of Sqoop run scripts. The scripts follow the same logic as there .sh counterparts. One difference is to create a Jar which references all classpath elements in its Manifest, and provide that jar as the single jar needed for Sqoop. The reason here is that in some cases if the number of classpath elements is large, HADOOP_CLASSPATH gets very long which causes failures in Windows since there is a limit to command lines. As a workaround, I added a step to wrap all jars in the classpath in a single jar, and then use that generated jar (this is also done in hadoop for Windows to handle similar issues) I did this in a utility script "BuildJar" which can be used for other components as well. This change is specific to Windows scripts, Linux scripts are not affected.

            People

            • Assignee:
              Ahmed El Baz
              Reporter:
              Ahmed El Baz
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development