Details
Description
Past few days, I am struggling with running my Hadoop application from windows client. and. I started with latest stable build 2.6.0. Installed HDFS on Linux.
My application is running on Windows which submits mapreduce jobs remotely. By default we don’t have windows distribution for Hadoop, So I need to build Hadoop on win32 platform. This turns out the non-trivial tasks as there are so many issue with windows build on both 2.6.0 and 2.2.0. By default Win32 support is not there.
I am sharing my efforts, so that It can be used by others struggling with same issue.
By default the targets to build on Windows platforms are missing in Hadoop 2.6.0 and 2.2.0 and other release.
So modify
modify native.sln and native.vcproj (Add targets for win32)
modify libwinutils.vcproj, winutils.sln and winutils.vcproj (Add targets for win32)
Sample for *.sln modification
353FDE506F33}.Release|Win32.ActiveCfg = Release|Win32
.Release|Win32.Build.0 = Release|Win32
{12131AA7-902E-4A6D-9CE3-043261D22A12}
.Release|Win32.Build.0 = Release|Win32
Sample for *.vcproj modification - modify properties with Win32 platform conditional conf.
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
Critical Issue - In native.vcproj the following file is not compiled.
This gives linker error.
<ClCompile Include="src\org\apache\hadoop\io\compress\lz4\lz4hc.c" />
We also need to document to setup the build environment for windows. As there are several issues creating Windows dev setup for Hadoop. May be we need to documents the steps.
Steps to set up you windows build environment-
1. Download and install Microsoft Windows SDK v7.1. From link http://www.microsoft.com/en-in/download/details.aspx?id=8279. Please follow the guidelines provided. You may have to uninstall .Net 4.5 before installing SDK version 7.x
2. You may get following error File Microsoft.Cpp.props not found, Patch SDK. Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1. Download from link http://www.microsoft.com/en-in/download/details.aspx?id=4422
3. You may also get following fatal error C1083: Cannot open include file: 'ammintrin.h': No such file or directory. This is mIcrosoft SDK bug. Get a patch to download ammintrin.h from https://connect.microsoft.com/VisualStudio/feedback/details/660584/
4. Now your Windows environment is ready to build. Hope there is no other errors reported on your system.
Attachments
Issue Links
- duplicates
-
HADOOP-9922 hadoop windows native build will fail in 32 bit machine
- Closed