Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.4
    • Component/s: build
    • Labels:
      None

      Description

      Sqoop fails to build on Windows due to dependencies on Linux scripts like write-version-info.sh and create-tool-scripts.sh which fail to build on Windows without Cygwin.

      1. SQOOP-949.trunk.1.patch
        11 kB
        Ahmed El Baz
      2. SQOOP-949.trunk.2.patch
        11 kB
        Ahmed El Baz

        Issue Links

          Activity

          Hide
          Ahmed El Baz added a comment -

          Attaching a patch with propsed fix which does the following:

          • provide Windows scripts src/scripts/create-tool-scripts.cmd and src/scripts/write-version-info.cmd which perform the same logic as the .sh scripts on Linux.
          • Update build.xml to invoke the appropriate script based on the host OS.
          • Skip real-docs target on Windows OS builds since as it invokes Linux specific "make" executable

          Verified builds are successful on both Linux and Windows, and Unit tests are fully passing

          Show
          Ahmed El Baz added a comment - Attaching a patch with propsed fix which does the following: provide Windows scripts src/scripts/create-tool-scripts.cmd and src/scripts/write-version-info.cmd which perform the same logic as the .sh scripts on Linux. Update build.xml to invoke the appropriate script based on the host OS. Skip real-docs target on Windows OS builds since as it invokes Linux specific "make" executable Verified builds are successful on both Linux and Windows, and Unit tests are fully passing
          Hide
          Jarek Jarcec Cecho added a comment -

          Changing ticket to "Patch available" so that it will show in the review queue.

          Show
          Jarek Jarcec Cecho added a comment - Changing ticket to "Patch available" so that it will show in the review queue.
          Hide
          Venkat Ranganathan added a comment -

          Ahmed,

          Thanks for the patch. Can you please upload it to the review board (reviews.apache.org). You can create a new review request and add a link to this bug

          Thanks

          Show
          Venkat Ranganathan added a comment - Ahmed, Thanks for the patch. Can you please upload it to the review board (reviews.apache.org). You can create a new review request and add a link to this bug Thanks
          Hide
          Ahmed El Baz added a comment -
          Show
          Ahmed El Baz added a comment - Done. https://reviews.apache.org/r/9961/ Thank you
          Hide
          Ahmed El Baz added a comment -

          Updating the patch with feedback received from Jarek and Venkat.

          • Remove trailing space
          • Redirect error output of the generate-tool script to NUL in windows (equivalent to /dev/null in Linux.
          Show
          Ahmed El Baz added a comment - Updating the patch with feedback received from Jarek and Venkat. Remove trailing space Redirect error output of the generate-tool script to NUL in windows (equivalent to /dev/null in Linux.
          Hide
          Ahmed El Baz added a comment -

          Attaching the link to the review board on this change
          Thanks,

          Ahmed

          Show
          Ahmed El Baz added a comment - Attaching the link to the review board on this change Thanks, Ahmed
          Hide
          Abhijeet Gaikwad added a comment -

          I'm trying to understand few things here:
          1. It doesn't seem like a bug, but an enhancement.
          2. As far as I know, the ultimate use of sqoop would be on a linux box with Hadoop on Linux. If windows I believe currently these things can run only with Cygwin. Now, why would a developer want to build sqoop on Windows? Getting windows hadoop up is a work in progress, are we also planning on porting sqoop to windows? Then this commit would make sense.
          3. With this commit, does that mean we can create a sqoop binary distributable package from a windows machine?

          Pardon my ignorance, if any

          Show
          Abhijeet Gaikwad added a comment - I'm trying to understand few things here: 1. It doesn't seem like a bug, but an enhancement. 2. As far as I know, the ultimate use of sqoop would be on a linux box with Hadoop on Linux. If windows I believe currently these things can run only with Cygwin. Now, why would a developer want to build sqoop on Windows? Getting windows hadoop up is a work in progress, are we also planning on porting sqoop to windows? Then this commit would make sense. 3. With this commit, does that mean we can create a sqoop binary distributable package from a windows machine? Pardon my ignorance, if any
          Hide
          Venkat Ranganathan added a comment -

          Hi Abhijeet Gaikwad

          Good questions. Please see below. Ahmed, others, feel free to clarify/edit/correct as needed.

          1. Yes, it could be termed as an improvement
          2. That is something that was true, but with Microsft HDInsight platform, there is an effort to get Hadoop run natively on Windows. Microsoft Hadoop platform is something we can download and use from Microsoft. Please see (1). Regarding running on windows, Ahmed mentioned that there will be patches to run on Windows with windows command scripts and fix any Windows specific bugs if any.
          3. Unfortunately no. Except for the creation of the documentation we should be able to build and (with subsequent patches) run Sqoop on windows.

          [1] http://hortonworks.com/about-us/news/hortonworks-brings-apache-hadoop-to-windows/

          Show
          Venkat Ranganathan added a comment - Hi Abhijeet Gaikwad Good questions. Please see below. Ahmed, others, feel free to clarify/edit/correct as needed. 1. Yes, it could be termed as an improvement 2. That is something that was true, but with Microsft HDInsight platform, there is an effort to get Hadoop run natively on Windows. Microsoft Hadoop platform is something we can download and use from Microsoft. Please see (1). Regarding running on windows, Ahmed mentioned that there will be patches to run on Windows with windows command scripts and fix any Windows specific bugs if any. 3. Unfortunately no. Except for the creation of the documentation we should be able to build and (with subsequent patches) run Sqoop on windows. [1] http://hortonworks.com/about-us/news/hortonworks-brings-apache-hadoop-to-windows/
          Hide
          Jarek Jarcec Cecho added a comment -

          I believe that allowing Sqoop to run on more environments will allow the project even brother adoption. Which is very good thing that can help Sqoop be more successful, so I definitely support the idea to natively support Windows platform.

          I do have concerns about destabilizing our stable branch, as the patch seems to be referencing files that are not inside the patch and it seems that we're planning to add much more related changes in near future. I was thinking about our options and I would like to propose using feature branch in this case. I know that we did not do any feature branches in the past, but this task seems to be a good fit. Would you guys be open to the idea of putting all windows related changes to a special branch before merging them to trunk? I can put together more formalish workflow if this is feasible to everyone.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - I believe that allowing Sqoop to run on more environments will allow the project even brother adoption. Which is very good thing that can help Sqoop be more successful, so I definitely support the idea to natively support Windows platform. I do have concerns about destabilizing our stable branch, as the patch seems to be referencing files that are not inside the patch and it seems that we're planning to add much more related changes in near future. I was thinking about our options and I would like to propose using feature branch in this case. I know that we did not do any feature branches in the past, but this task seems to be a good fit. Would you guys be open to the idea of putting all windows related changes to a special branch before merging them to trunk? I can put together more formalish workflow if this is feasible to everyone. Jarcec
          Hide
          Venkat Ranganathan added a comment -

          Hi Jarek Jarcec Cecho

          I share your concerns on destabilizing by having patches come one a time. I agree that the one patch at a time does not seem to be good. I was under the impression that there are three patches (to fix build/provide run scripts and some unit test fixes). We should probably have one umbrella JIRA and add subtasks to it and provide complete patches to evaluate it. I understand having a separate branch makes sense for many changes but not sure if the scope of changes is big in this case. May be we should collect all the patches and decide if it makes sense to create a branch? I am OK with either of the approaches

          Thanks

          Venkat

          Show
          Venkat Ranganathan added a comment - Hi Jarek Jarcec Cecho I share your concerns on destabilizing by having patches come one a time. I agree that the one patch at a time does not seem to be good. I was under the impression that there are three patches (to fix build/provide run scripts and some unit test fixes). We should probably have one umbrella JIRA and add subtasks to it and provide complete patches to evaluate it. I understand having a separate branch makes sense for many changes but not sure if the scope of changes is big in this case. May be we should collect all the patches and decide if it makes sense to create a branch? I am OK with either of the approaches Thanks Venkat
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you Venkat Ranganathan for your quick feedback, much appreciated.

          Having umbrella jira "Windows support" makes complete sense. I wasn't sure how many patches are required to get Windows support or how long it will take to finish them, so I was more thinking about "long term" branch. However if there are just three patches that are more or less ready and it's just matter of upload and review, I'm pretty much fine with doing that on trunk.

          Ahmed El Baz, what do you think?

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Thank you Venkat Ranganathan for your quick feedback, much appreciated. Having umbrella jira "Windows support" makes complete sense. I wasn't sure how many patches are required to get Windows support or how long it will take to finish them, so I was more thinking about "long term" branch. However if there are just three patches that are more or less ready and it's just matter of upload and review, I'm pretty much fine with doing that on trunk. Ahmed El Baz , what do you think? Jarcec
          Hide
          Abhijeet Gaikwad added a comment -

          It would be really nice to see Sqoop for Windows. HDInsights effort is commendable.

          I believe Porting Sqoop to windows wouldn't be easy. I was thinking of having a separate branch track this. We may want to fix some bugs in near future before stability. Have we thought of features like hive and hbase import? We would need to have hudson integration for windows related builds. It would be a totally different package that we would be distributing. I think merge can wait until we decide on distributing stable Sqoop package for Windows.
          I would like PMC take the decision on merge via VOTE after we are ready.

          I'm sorry this may be a naive or a mean question but do we have any community obligations with Sqoop being legally tested and supported for a third-party hadoop distro? Do we need to wait until Apache Hadoop announces native windows support? I personally don't care.

          I agree on umbrella JIRA tracking these changes.

          Show
          Abhijeet Gaikwad added a comment - It would be really nice to see Sqoop for Windows. HDInsights effort is commendable. I believe Porting Sqoop to windows wouldn't be easy. I was thinking of having a separate branch track this. We may want to fix some bugs in near future before stability. Have we thought of features like hive and hbase import? We would need to have hudson integration for windows related builds. It would be a totally different package that we would be distributing. I think merge can wait until we decide on distributing stable Sqoop package for Windows. I would like PMC take the decision on merge via VOTE after we are ready. I'm sorry this may be a naive or a mean question but do we have any community obligations with Sqoop being legally tested and supported for a third-party hadoop distro? Do we need to wait until Apache Hadoop announces native windows support? I personally don't care. I agree on umbrella JIRA tracking these changes.
          Hide
          Venkat Ranganathan added a comment -

          Hi Abhijeet Gaikwad
          Sorry - I had an eye appt and had my eyes dilated this afternoon so did not want to respond with blurry eyes
          You raise some good questions on getting Sqoop to work on Windows and I think the patches from Ahmed El Baz would be a good one to review with respect to those.

          One of the curious statements I saw above was
          bq I'm sorry this may be a naive or a mean question but do we have any community obligations with Sqoop being legally tested and supported for a third-party hadoop distro? Do we need to wait until Apache Hadoop announces native windows support eq

          I am not sure I parse that correctly. As you have mentioned about community and Jarek Jarcec Cecho did mention about community development and expansion and hopefully that is the viewpoint of the Sqoop community in general, this effort is more geared towards community expansion much like supporting many proprietary databases to work with Sqoop. I don't know what is the intersection of legal requirement and I am also not sure support for third party distro support.

          I agree we should get this in an umbrella JIRA with sub tasks and decide as a community where we want to take them

          Thanks
          Venkat

          Show
          Venkat Ranganathan added a comment - Hi Abhijeet Gaikwad Sorry - I had an eye appt and had my eyes dilated this afternoon so did not want to respond with blurry eyes You raise some good questions on getting Sqoop to work on Windows and I think the patches from Ahmed El Baz would be a good one to review with respect to those. One of the curious statements I saw above was bq I'm sorry this may be a naive or a mean question but do we have any community obligations with Sqoop being legally tested and supported for a third-party hadoop distro? Do we need to wait until Apache Hadoop announces native windows support eq I am not sure I parse that correctly. As you have mentioned about community and Jarek Jarcec Cecho did mention about community development and expansion and hopefully that is the viewpoint of the Sqoop community in general, this effort is more geared towards community expansion much like supporting many proprietary databases to work with Sqoop. I don't know what is the intersection of legal requirement and I am also not sure support for third party distro support. I agree we should get this in an umbrella JIRA with sub tasks and decide as a community where we want to take them Thanks Venkat
          Hide
          Ahmed El Baz added a comment -

          Thank you Venkat, Abhijeet, and Jarek for the valuable comments.

          A few things to clarify:
          1) As I mentioned, and Venkat has confirmed. There a few other patches that I will be sending shortly for review which complete the picture for Sqoop on Windows.
          a) Build Sqoop on Windows (Compile). Which is this one
          b) Run Sqoop on Windows (Runtime scripts)
          c) Fix unit test issues on Windows (Fixes are all Windows specific).
          d) Introdice a Winpkg distro target for Sqoop on Windows (Similar to tar on Linux)

          With every commit, as this one, we make sure that nothing is currently impacted/destabilized.

          2) This patch does not actually reference files that do not exist. It generates the sqoop-tool scripts. I agree they would not actually work for now if invoked until Runtime support for Sqoop on Windows is provided, which is coming next.

          I agree that making an umbrella Jira with subtasks would provide more context. I will be creating this one and linking to this Jira.

          Thank you again
          Ahmed

          Show
          Ahmed El Baz added a comment - Thank you Venkat, Abhijeet, and Jarek for the valuable comments. A few things to clarify: 1) As I mentioned, and Venkat has confirmed. There a few other patches that I will be sending shortly for review which complete the picture for Sqoop on Windows. a) Build Sqoop on Windows (Compile). Which is this one b) Run Sqoop on Windows (Runtime scripts) c) Fix unit test issues on Windows (Fixes are all Windows specific). d) Introdice a Winpkg distro target for Sqoop on Windows (Similar to tar on Linux) With every commit, as this one, we make sure that nothing is currently impacted/destabilized. 2) This patch does not actually reference files that do not exist. It generates the sqoop-tool scripts. I agree they would not actually work for now if invoked until Runtime support for Sqoop on Windows is provided, which is coming next. I agree that making an umbrella Jira with subtasks would provide more context. I will be creating this one and linking to this Jira. Thank you again Ahmed
          Hide
          Ahmed El Baz added a comment -

          I have created an umbrella Jira for Sqoop on Wondows support SQOOP-957
          https://issues.apache.org/jira/browse/SQOOP-957

          Many thanks,
          Ahmed

          Show
          Ahmed El Baz added a comment - I have created an umbrella Jira for Sqoop on Wondows support SQOOP-957 https://issues.apache.org/jira/browse/SQOOP-957 Many thanks, Ahmed
          Hide
          Abhijeet Gaikwad added a comment -

          Venkat:
          I'm sorry I missed Jarcec's statement that clarifies things. Thanks.

          Let us be fair here in saying that only one hadoop distributor supports Windows. So, indirectly when we claim Sqoop would run on Windows we know what hadoop distro would it be working with. Now, I was under an impression that we would be building the code and documenting things with respect to this distro. But that doesn't seem to be the case.

          If we would build the code with Apache Hadoop, I am more worried about the fact that we expect users to use it with some other distribution. But from the comments above it doesn't seem like an issue.

          Ahmed:
          If we are sure we could get a stable connector out for windows after these four commits, I don't think there is need to create a new branch. We should also think of creating Hudson configuration for windows related builds. Thanks.

          I'm sorry to keep the commit waiting, but wanted to be sure about few things before we commit. +1 on all the related commits.

          Good to see windows support for Sqoop. Thanks for the efforts Ahmed.

          Show
          Abhijeet Gaikwad added a comment - Venkat: I'm sorry I missed Jarcec's statement that clarifies things. Thanks. Let us be fair here in saying that only one hadoop distributor supports Windows. So, indirectly when we claim Sqoop would run on Windows we know what hadoop distro would it be working with. Now, I was under an impression that we would be building the code and documenting things with respect to this distro. But that doesn't seem to be the case. If we would build the code with Apache Hadoop, I am more worried about the fact that we expect users to use it with some other distribution. But from the comments above it doesn't seem like an issue. Ahmed: If we are sure we could get a stable connector out for windows after these four commits, I don't think there is need to create a new branch. We should also think of creating Hudson configuration for windows related builds. Thanks. I'm sorry to keep the commit waiting, but wanted to be sure about few things before we commit. +1 on all the related commits. Good to see windows support for Sqoop. Thanks for the efforts Ahmed.
          Hide
          Ahmed El Baz added a comment -

          Thank you Abhijeet,

          Yes after these 4 patches, Sqoop should be ready to run on Windows. I am planning to submit future patches to handle errors specific to the MSSQlServer Connector, but they are not Windows specific.

          Venkat, Jarek:
          I have submitted a patch to SQOOP-954. The changes for SQOOP-955 and SQOOP-956 depend on the first two patches.
          I can make the patches for the those Jira incremental to the first 2 Jiras if we do not want to commit them separately.

          Thanks,

          Ahmed

          Show
          Ahmed El Baz added a comment - Thank you Abhijeet, Yes after these 4 patches, Sqoop should be ready to run on Windows. I am planning to submit future patches to handle errors specific to the MSSQlServer Connector, but they are not Windows specific. Venkat, Jarek: I have submitted a patch to SQOOP-954 . The changes for SQOOP-955 and SQOOP-956 depend on the first two patches. I can make the patches for the those Jira incremental to the first 2 Jiras if we do not want to commit them separately. Thanks, Ahmed
          Hide
          Jarek Jarcec Cecho added a comment -

          It seems that we have agreement on this patch. As the changes are not intrusive and are mainly with adding new files, I've committed it.

          Thank you Ahmed for your time and effort!

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - It seems that we have agreement on this patch. As the changes are not intrusive and are mainly with adding new files, I've committed it . Thank you Ahmed for your time and effort! Jarcec
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #827 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/827/)
          SQOOP-949: Allow Sqoop to build on Windows OS (Revision c4b0eac2d0bd954d1c6cae2db158124908fbf055)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4b0eac2d0bd954d1c6cae2db158124908fbf055
          Files :

          • src/scripts/create-tool-scripts.cmd
          • src/scripts/tool-script.cmd.template
          • build.xml
          • src/scripts/write-version-info.cmd
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #827 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/827/ ) SQOOP-949 : Allow Sqoop to build on Windows OS (Revision c4b0eac2d0bd954d1c6cae2db158124908fbf055) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4b0eac2d0bd954d1c6cae2db158124908fbf055 Files : src/scripts/create-tool-scripts.cmd src/scripts/tool-script.cmd.template build.xml src/scripts/write-version-info.cmd
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #619 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/619/)
          SQOOP-949: Allow Sqoop to build on Windows OS (Revision c4b0eac2d0bd954d1c6cae2db158124908fbf055)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4b0eac2d0bd954d1c6cae2db158124908fbf055
          Files :

          • src/scripts/create-tool-scripts.cmd
          • build.xml
          • src/scripts/write-version-info.cmd
          • src/scripts/tool-script.cmd.template
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #619 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/619/ ) SQOOP-949 : Allow Sqoop to build on Windows OS (Revision c4b0eac2d0bd954d1c6cae2db158124908fbf055) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4b0eac2d0bd954d1c6cae2db158124908fbf055 Files : src/scripts/create-tool-scripts.cmd build.xml src/scripts/write-version-info.cmd src/scripts/tool-script.cmd.template
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #619 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/619/)
          SQOOP-949: Allow Sqoop to build on Windows OS (Revision c4b0eac2d0bd954d1c6cae2db158124908fbf055)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4b0eac2d0bd954d1c6cae2db158124908fbf055
          Files :

          • src/scripts/tool-script.cmd.template
          • build.xml
          • src/scripts/create-tool-scripts.cmd
          • src/scripts/write-version-info.cmd
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #619 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/619/ ) SQOOP-949 : Allow Sqoop to build on Windows OS (Revision c4b0eac2d0bd954d1c6cae2db158124908fbf055) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4b0eac2d0bd954d1c6cae2db158124908fbf055 Files : src/scripts/tool-script.cmd.template build.xml src/scripts/create-tool-scripts.cmd src/scripts/write-version-info.cmd
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #625 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/625/)
          SQOOP-949: Allow Sqoop to build on Windows OS (Revision c4b0eac2d0bd954d1c6cae2db158124908fbf055)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4b0eac2d0bd954d1c6cae2db158124908fbf055
          Files :

          • src/scripts/write-version-info.cmd
          • src/scripts/create-tool-scripts.cmd
          • src/scripts/tool-script.cmd.template
          • build.xml
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #625 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/625/ ) SQOOP-949 : Allow Sqoop to build on Windows OS (Revision c4b0eac2d0bd954d1c6cae2db158124908fbf055) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4b0eac2d0bd954d1c6cae2db158124908fbf055 Files : src/scripts/write-version-info.cmd src/scripts/create-tool-scripts.cmd src/scripts/tool-script.cmd.template build.xml

            People

            • Assignee:
              Ahmed El Baz
              Reporter:
              Ahmed El Baz
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development