Hive
  1. Hive
  2. HIVE-344

Fix the Hive CLI to run on cygwin under windows

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      windows xp

    • Hadoop Flags:
      Reviewed
    • Release Note:
      * Fixes for running hive under cygwin

      Description

      Yes, I am guilty as charged, I don't use mac like the rest of you

      The Hive CLI currently doesn't work with cygwin under windows, this is due to the dual path model that cygwin employs (there are file paths relative to the cygwin virtual root, and file paths relative to the windows root)

      Since Sun's JDK is installed under the windows environment, if the paths are not converted to windows format before being passed along then java will be at a loss to where the files are. The solution is to use the cygpath command to convert the paths to windows format before passing along to java world.

      I have a fix for this already, still doing some further testing (to make sure it works under both unix and windows environments), then I will submit patch to this bug (should submit before end of this week).

      Cheers,

      – amr

      1. HIVE-344-v2.patch
        3 kB
        Amr Awadallah
      2. HIVE-344.patch
        2 kB
        Amr Awadallah

        Activity

        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Rim Moussa made changes -
        Environment windows xp
        Component/s Clients [ 12312587 ]
        Hide
        Rim Moussa added a comment -

        Hi
        I've installed hadoop-0.19.2 and hive under cygwin
        am trying to follow the tutorial http://wiki.apache.org/hadoop/Hive/GettingStarted#Downloading_and_building
        everything is working fine in the tutorial until I tried to execute an sql stmt
        after performing following settings
        hive>SET mapred.job.tracker=myhost.mycompany.com:50030;
        hive>SET -v ;
        then I get "Number of reduce tasks is set to 0 since there's no reduce operator"
        first does the patch apply to this error
        am a new to linux, how to apply the patch
        isn't worth to work with co-linux?
        thankx

        Show
        Rim Moussa added a comment - Hi I've installed hadoop-0.19.2 and hive under cygwin am trying to follow the tutorial http://wiki.apache.org/hadoop/Hive/GettingStarted#Downloading_and_building everything is working fine in the tutorial until I tried to execute an sql stmt after performing following settings hive>SET mapred.job.tracker=myhost.mycompany.com:50030; hive>SET -v ; then I get "Number of reduce tasks is set to 0 since there's no reduce operator" first does the patch apply to this error am a new to linux, how to apply the patch isn't worth to work with co-linux? thankx
        Hide
        Prasad Chakka added a comment -

        what was the problem?

        Show
        Prasad Chakka added a comment - what was the problem?
        Ashish Thusoo made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.4.0 [ 12313714 ]
        Fix Version/s 0.3.1 [ 12313845 ]
        Resolution Fixed [ 1 ]
        Hide
        Ashish Thusoo added a comment -

        committed. Thanks Amr!!

        Show
        Ashish Thusoo added a comment - committed. Thanks Amr!!
        Hide
        Ashish Thusoo added a comment -

        Got it working. Looks all good to me.

        +1

        Will checkin once the unit tests pass.

        Show
        Ashish Thusoo added a comment - Got it working. Looks all good to me. +1 Will checkin once the unit tests pass.
        Hide
        Ashish Thusoo added a comment -

        I am having problems with show tables.

        hive> show tables;
        FAILED: Error in metadata: javax.jdo.JDODataStoreException: Exception thrown performing schema operation : Add classes to Catalog "", Schema "APP"
        NestedThrowables:
        java.sql.SQLNonTransientConnectionException: No current connection.
        FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

        Prasad can you comment on this?

        Show
        Ashish Thusoo added a comment - I am having problems with show tables. hive> show tables; FAILED: Error in metadata: javax.jdo.JDODataStoreException: Exception thrown performing schema operation : Add classes to Catalog "", Schema "APP" NestedThrowables: java.sql.SQLNonTransientConnectionException: No current connection. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask Prasad can you comment on this?
        Hide
        Amr Awadallah added a comment -

        unfortunately HIVE_LIB is checked for validity inside these files using normal bash commands, so the cygpath translation for HIVE_LIB has to be done this late as opposed to in the main hive script (otherwise the verification in sub-scripts would fail).

        Show
        Amr Awadallah added a comment - unfortunately HIVE_LIB is checked for validity inside these files using normal bash commands, so the cygpath translation for HIVE_LIB has to be done this late as opposed to in the main hive script (otherwise the verification in sub-scripts would fail).
        Hide
        Ashish Thusoo added a comment -

        The first tests seem to be ok. Will do a bit more today and checkin if it works. I have one question though about the patch. The cygpath thingy is done in the cli.sh and lineage.sh files after HIVE_LIB is used. Should it not be before HIVE_LIB is used?

        Show
        Ashish Thusoo added a comment - The first tests seem to be ok. Will do a bit more today and checkin if it works. I have one question though about the patch. The cygpath thingy is done in the cli.sh and lineage.sh files after HIVE_LIB is used. Should it not be before HIVE_LIB is used?
        Hide
        Amr Awadallah added a comment -

        ashish, worked?

        Show
        Amr Awadallah added a comment - ashish, worked?
        Hide
        Ashish Thusoo added a comment -

        Thanks Amr. Will try it out today.

        Show
        Ashish Thusoo added a comment - Thanks Amr. Will try it out today.
        Johan Oskarsson made changes -
        Fix Version/s 0.3.1 [ 12313845 ]
        Fix Version/s 0.3.2 [ 12314124 ]
        Amr Awadallah made changes -
        Attachment HIVE-344-v2.patch [ 12414326 ]
        Hide
        Amr Awadallah added a comment -

        @Ashish

        Can't believe it has been two months already, but please find a new patch attached (HIVE-344-v2.patch).

        I think I tracked down all places where AUX_PARAM and AUX_CLASSPATH get set.

        Please give a try and let me know.

        Cheers,

        – amr

        Show
        Amr Awadallah added a comment - @Ashish Can't believe it has been two months already, but please find a new patch attached ( HIVE-344 -v2.patch). I think I tracked down all places where AUX_PARAM and AUX_CLASSPATH get set. Please give a try and let me know. Cheers, – amr
        Johan Oskarsson made changes -
        Fix Version/s 0.3.2 [ 12314124 ]
        Fix Version/s 0.3.1 [ 12313845 ]
        Hide
        Johan Oskarsson added a comment -

        Since there's been no activity on this ticket for a while I suggest we move it to 0.3.2 so that 0.3.1 can be released.

        Show
        Johan Oskarsson added a comment - Since there's been no activity on this ticket for a while I suggest we move it to 0.3.2 so that 0.3.1 can be released.
        Hide
        Amr Awadallah added a comment -

        @Ashish, sorry for delay replying and thanks for testing. I thought I tracked down all of the path params, I will take a closer look at AUX_PARAMS next week and submit another patch.

        – amr

        Show
        Amr Awadallah added a comment - @Ashish, sorry for delay replying and thanks for testing. I thought I tracked down all of the path params, I will take a closer look at AUX_PARAMS next week and submit another patch. – amr
        Hide
        Ashish Thusoo added a comment -

        I tried this but it does not seem to work with auxilary jars. I get the following error

        bash-3.2$ hive/bin/hive
        java.net.URISyntaxException: Illegal character in scheme name at index 4: file;c:/Users/Ashish/dev/hive/hive/auxlib/fbhive_hooks.jar

        AUX_PARAM has URIs in it so cygpath will probably cause problems there. The cygpath code for AUX_PARAMS needs to go at all the places where aux params are getting created from file names as opposed to at the end. I think that should fix this issue.

        Show
        Ashish Thusoo added a comment - I tried this but it does not seem to work with auxilary jars. I get the following error bash-3.2$ hive/bin/hive java.net.URISyntaxException: Illegal character in scheme name at index 4: file;c:/Users/Ashish/dev/hive/hive/auxlib/fbhive_hooks.jar AUX_PARAM has URIs in it so cygpath will probably cause problems there. The cygpath code for AUX_PARAMS needs to go at all the places where aux params are getting created from file names as opposed to at the end. I think that should fix this issue.
        Johan Oskarsson made changes -
        Fix Version/s 0.3.1 [ 12313845 ]
        Fix Version/s 0.3.0 [ 12313637 ]
        Hide
        Johan Oskarsson added a comment -

        Moving to 0.3.1 as 0.3.0 is released

        Show
        Johan Oskarsson added a comment - Moving to 0.3.1 as 0.3.0 is released
        Ashish Thusoo made changes -
        Assignee Amr Awadallah [ aaa ]
        Hide
        Ashish Thusoo added a comment -

        Thanks for the patch Amr. Like you I am a PC user as well. Will try it out.

        Sometime back I do remember seeing some email on the hadoop mailing list that they are going to deprecate support on cygwin. Don't know if that is true or not...

        Also you should try coLinux (it is a very lightweight linux vm that runs on windows). I have been able to do all my hive development on the FC7 image with that.

        Show
        Ashish Thusoo added a comment - Thanks for the patch Amr. Like you I am a PC user as well. Will try it out. Sometime back I do remember seeing some email on the hadoop mailing list that they are going to deprecate support on cygwin. Don't know if that is true or not... Also you should try coLinux (it is a very lightweight linux vm that runs on windows). I have been able to do all my hive development on the FC7 image with that.
        Amr Awadallah made changes -
        Attachment HIVE-344.patch [ 12407452 ]
        Amr Awadallah made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note * Fixes for running hive under cygwin
        Affects Version/s 0.3.0 [ 12313637 ]
        Fix Version/s 0.3.0 [ 12313637 ]
        Hide
        Amr Awadallah added a comment -


        I upgraded the fix to work with Hive 0.3. Would be great if a couple of folks can try this under cygwin and let me know if it works fine. It shouldn't affect non-windows installations, and for windows installations it will change the relevant paths/dirs using cygpath (I added the fix to all services: cli.sh, hwi.sh, hiveserver.sh, and lineage.sh)

        Show
        Amr Awadallah added a comment - I upgraded the fix to work with Hive 0.3. Would be great if a couple of folks can try this under cygwin and let me know if it works fine. It shouldn't affect non-windows installations, and for windows installations it will change the relevant paths/dirs using cygpath (I added the fix to all services: cli.sh, hwi.sh, hiveserver.sh, and lineage.sh)
        Amr Awadallah created issue -

          People

          • Assignee:
            Amr Awadallah
            Reporter:
            Amr Awadallah
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development