Details

    • Release Note:
      Hide
      Made Hadoop QFS plugin as part of Quantcast File System open-source project.
      Users can download the Hadoop QFS jar + libs and dynamically use it with Hadoop.
      Show
      Made Hadoop QFS plugin as part of Quantcast File System open-source project. Users can download the Hadoop QFS jar + libs and dynamically use it with Hadoop.

      Description

      Quantcast has released QFS 1.0 (http://quantcast.github.com/qfs), a C++ distributed filesystem based on Kosmos File System(KFS). QFS comes with various feature, performance, and stability improvements over KFS.

      A hadoop 'fs' shim needs be added to support QFS through 'qfs://' URIs.

        Issue Links

          Activity

          Hide
          thilee added a comment -

          Attached is the patch for branch-1.
          In addition to applying the patch, you would need to do two things:

          (1) To compile: you need to copy the qfs access JAR (qfs-<version>.jar) to the branch-1 lib directory.
          (2) To run: you need to have the directory of libqfs*.so in your LD_LIBRARY_PATH.

          The tarball containing the JAR file and the shared libraries can be downloaded here: (please download the one that matches your platform, untar and look inside lib directory)
          https://github.com/quantcast/qfs/wiki/Binary-Distributions

          The JAR file should eventually be integrated to Apache Hadoop, under lib directory, as qfs-1.0.0.jar.

          Show
          thilee added a comment - Attached is the patch for branch-1 . In addition to applying the patch, you would need to do two things: (1) To compile: you need to copy the qfs access JAR (qfs-<version>.jar) to the branch-1 lib directory. (2) To run: you need to have the directory of libqfs*.so in your LD_LIBRARY_PATH . The tarball containing the JAR file and the shared libraries can be downloaded here: (please download the one that matches your platform, untar and look inside lib directory) https://github.com/quantcast/qfs/wiki/Binary-Distributions The JAR file should eventually be integrated to Apache Hadoop, under lib directory, as qfs-1.0.0.jar.
          Hide
          thilee added a comment -

          Patch for trunk is attached as well now.

          Show
          thilee added a comment - Patch for trunk is attached as well now.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12548323/HADOOP-8885-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1589//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548323/HADOOP-8885-trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1589//console This message is automatically generated.
          Hide
          thilee added a comment -

          Based on the comments in the mailing list, I have pulled out the Quantcast File System implementation from Apache Hadoop to the QFS project. QFS wil be maintained by Quantcast. Please refer to http://quantcast.github.com/qfs/ and https://github.com/quantcast/qfs/wiki/Migration-Guide.

          Show
          thilee added a comment - Based on the comments in the mailing list, I have pulled out the Quantcast File System implementation from Apache Hadoop to the QFS project. QFS wil be maintained by Quantcast. Please refer to http://quantcast.github.com/qfs/ and https://github.com/quantcast/qfs/wiki/Migration-Guide .
          Hide
          thilee added a comment -

          Based on suggestions in the hadoop-common mailing list, we decided to take the QFS implementation out of Hadoop Common/core.

          QFS Hadoop plugin is now maintained at http://quantcast.github.com/qfs/

          Instructions for obtaining and using the QFS Hadoop plugin are at
          https://github.com/quantcast/qfs/wiki/Migration-Guide

          Show
          thilee added a comment - Based on suggestions in the hadoop-common mailing list, we decided to take the QFS implementation out of Hadoop Common/core. QFS Hadoop plugin is now maintained at http://quantcast.github.com/qfs/ Instructions for obtaining and using the QFS Hadoop plugin are at https://github.com/quantcast/qfs/wiki/Migration-Guide

            People

            • Assignee:
              thilee
              Reporter:
              thilee
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development