[HIVE-2998] Making Hive run on Windows Server and Windows Azure environment

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.1, 0.8.1

Environment

Windows Server 2008 R2 and Windows Azure

• Tags: Windows,Azure,Native,Hive
• Assignee: Unassigned
• Reporter: Lengning Liu
• Created: 02/May/12 22:33
• Updated: 26/Jun/15 09:26

Description

This is the master JIRA for improvements to Hive that would enable it to run natively on Windows Server and Windows Azure environments. Microsoft has done the initial work here to have Hive (releases 0.7.1 and 0.8.1) running on Windows and would like to contribute this work back to the community. The end-to-end HiveQL query tests pass. We are currently working on investigating failed unit test cases. It is expected that we post the initial patches within a few weeks for review. Looking forward to the collaboration. Thanks!

Issue Links

• incorporates: HIVE-3130 Fix ODBC build errors (C++ Targets) in Hive on windows
• incorporates: HIVE-4349 Fix the Hive unit test failures when the Hive enlistment root path is longer than ~12 characters
• incorporates: HIVE-4445 Fix the Hive unit test failures on Windows when Linux scripts or commands are used in test cases
• incorporates: HIVE-4555 Run Hive unit test on windows without Cygwin
• incorporates: HIVE-4417 Make unit tests pass on Windows which use Linux script in query files
• incorporates: HIVE-4448 Fix metastore warehouse incorrect location on Windows in unit tests
• incorporates: HIVE-3126 Generate & build the velocity based Hive tests on windows by fixing the path issues
• incorporates: HIVE-3127 Pass hconf values as XML instead of command line arguments to child JVM
• incorporates: HIVE-3128 use commons-compress instead of forking tar process
• incorporates: HIVE-4347 Hcatalog build fail on Windows because javadoc command exceed length limit
• incorporates: HIVE-4348 Unit test compile fail at hbase-handler project on Windows becuase of illegal escape character
• incorporates: HIVE-4422 Test output need to be updated for Windows only unit test in TestCliDriver
• incorporates: HIVE-3129 Create windows native scripts (CMD files) to run hive on windows without Cygwin
• incorporates: HIVE-4469 Tests in TestCliDriver fail on Windows caused by hard coded file size in .q.out file
• incorporates: HIVE-4558 mapreduce_stack_trace_hadoop20 in TestNegativeMinimrCliDriver fails on Windows
• incorporates: HIVE-3132 Update the documentation with the instructions to run hive on windows without Cygwin
• is blocked by: HADOOP-8486 Resource leak - Close the open resource handles (File handles) before throwing the exception from the SequenceFile constructor
• requires: HIVE-3232 Resource Leak: Fix the File handle leak in EximUtil.java
• requires: HIVE-3131 Allow to run Hive unit tests against unreleased versions of Hadoop

Sub-Tasks

• Support external hive tables whose data are stored in Azure blob store/Azure Storage Volumes (ASV): Closed
• Remove the duplicate JAR entries from the (“test.classpath”) to avoid command line exceeding char limit on windows: Closed
• Fix “TestDosToUnix” unit tests on Windows by closing the leaking file handle in DosToUnix.java.: Closed
• Fix the “TestHiveHistory”, “TestHiveConf”, & “TestExecDriver” unit tests on Windows by fixing the path related issues.: Closed
• Handle “CRLF” line endings to avoid the extra spacing in generated test outputs in Windows. (Utilities.Java :: readColumn): Closed
• Remove the Unix specific absolute path of “Cat” utility in several .q files to make them run on Windows with CygWin in path.: Closed