Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-573

Porting libhdfs to Windows

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: libhdfs
    • Labels:
      None
    • Environment:

      Windows, Visual Studio 2008

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The libhdfs C API is now supported on Windows.

      Description

      The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.

      In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.

        Attachments

        1. HDFS-573.4.patch
          164 kB
          Chris Nauroth
        2. HDFS-573.3.patch
          165 kB
          Chris Nauroth
        3. HDFS-573.2.patch
          165 kB
          Chris Nauroth
        4. HDFS-573.1.patch
          144 kB
          Chris Nauroth

          Issue Links

            Activity

              People

              • Assignee:
                cnauroth Chris Nauroth
                Reporter:
                zwabbit Ziliang Guo
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 336h
                  336h
                  Remaining:
                  Remaining Estimate - 336h
                  336h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified