Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None
    • Release Note:
      Hide
      Made Hadoop QFS plugin as part of Quantcast File System open-source project.
      Users can download the Hadoop QFS jar + libs and dynamically use it with Hadoop.
      Show
      Made Hadoop QFS plugin as part of Quantcast File System open-source project. Users can download the Hadoop QFS jar + libs and dynamically use it with Hadoop.

      Description

      Quantcast has released QFS 1.0 (http://quantcast.github.com/qfs), a C++ distributed filesystem based on Kosmos File System(KFS). QFS comes with various feature, performance, and stability improvements over KFS.

      A hadoop 'fs' shim needs be added to support QFS through 'qfs://' URIs.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          3d 23h 58m 1 thilee 09/Oct/12 17:16
          Patch Available Patch Available Open Open
          16d 6h 35m 1 thilee 25/Oct/12 23:52
          Open Open In Progress In Progress
          40s 1 thilee 25/Oct/12 23:53
          In Progress In Progress Open Open
          17h 21m 1 thilee 26/Oct/12 17:14
          Open Open Resolved Resolved
          9m 13s 1 thilee 26/Oct/12 17:23
          Allen Wittenauer made changes -
          Fix Version/s 0.23.3 [ 12320059 ]
          Fix Version/s 1.0.2 [ 12320152 ]
          Fix Version/s 1.0.3 [ 12320248 ]
          Fix Version/s 2.0.0-alpha [ 12320352 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 2.0.1-alpha [ 12322467 ]
          thilee made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Release Note Made Hadoop QFS plugin as part of Quantcast File System open-source project.
          Users can download the Hadoop QFS jar + libs and dynamically use it with Hadoop.
          Resolution Fixed [ 1 ]
          Hide
          thilee added a comment -

          Based on suggestions in the hadoop-common mailing list, we decided to take the QFS implementation out of Hadoop Common/core.

          QFS Hadoop plugin is now maintained at http://quantcast.github.com/qfs/

          Instructions for obtaining and using the QFS Hadoop plugin are at
          https://github.com/quantcast/qfs/wiki/Migration-Guide

          Show
          thilee added a comment - Based on suggestions in the hadoop-common mailing list, we decided to take the QFS implementation out of Hadoop Common/core. QFS Hadoop plugin is now maintained at http://quantcast.github.com/qfs/ Instructions for obtaining and using the QFS Hadoop plugin are at https://github.com/quantcast/qfs/wiki/Migration-Guide
          thilee made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          Hide
          thilee added a comment -

          Based on the comments in the mailing list, I have pulled out the Quantcast File System implementation from Apache Hadoop to the QFS project. QFS wil be maintained by Quantcast. Please refer to http://quantcast.github.com/qfs/ and https://github.com/quantcast/qfs/wiki/Migration-Guide.

          Show
          thilee added a comment - Based on the comments in the mailing list, I have pulled out the Quantcast File System implementation from Apache Hadoop to the QFS project. QFS wil be maintained by Quantcast. Please refer to http://quantcast.github.com/qfs/ and https://github.com/quantcast/qfs/wiki/Migration-Guide .
          thilee made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          thilee made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12548323/HADOOP-8885-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1589//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548323/HADOOP-8885-trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1589//console This message is automatically generated.
          thilee made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.23.3 [ 12320059 ]
          Affects Version/s 1.0.2 [ 12320152 ]
          Affects Version/s 1.0.3 [ 12320248 ]
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Affects Version/s 2.0.1-alpha [ 12322467 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 2.0.1-alpha [ 12322467 ]
          Fix Version/s 2.0.0-alpha [ 12320352 ]
          Fix Version/s 0.23.3 [ 12320059 ]
          Fix Version/s 1.0.3 [ 12320248 ]
          Fix Version/s 1.0.2 [ 12320152 ]
          thilee made changes -
          Attachment HADOOP-8885-trunk.patch [ 12548323 ]
          Hide
          thilee added a comment -

          Patch for trunk is attached as well now.

          Show
          thilee added a comment - Patch for trunk is attached as well now.
          Aaron T. Myers made changes -
          Assignee thilee [ thilee ]
          thilee made changes -
          Attachment HADOOP-8885-branch-1.patch [ 12548072 ]
          Hide
          thilee added a comment -

          Attached is the patch for branch-1.
          In addition to applying the patch, you would need to do two things:

          (1) To compile: you need to copy the qfs access JAR (qfs-<version>.jar) to the branch-1 lib directory.
          (2) To run: you need to have the directory of libqfs*.so in your LD_LIBRARY_PATH.

          The tarball containing the JAR file and the shared libraries can be downloaded here: (please download the one that matches your platform, untar and look inside lib directory)
          https://github.com/quantcast/qfs/wiki/Binary-Distributions

          The JAR file should eventually be integrated to Apache Hadoop, under lib directory, as qfs-1.0.0.jar.

          Show
          thilee added a comment - Attached is the patch for branch-1 . In addition to applying the patch, you would need to do two things: (1) To compile: you need to copy the qfs access JAR (qfs-<version>.jar) to the branch-1 lib directory. (2) To run: you need to have the directory of libqfs*.so in your LD_LIBRARY_PATH . The tarball containing the JAR file and the shared libraries can be downloaded here: (please download the one that matches your platform, untar and look inside lib directory) https://github.com/quantcast/qfs/wiki/Binary-Distributions The JAR file should eventually be integrated to Apache Hadoop, under lib directory, as qfs-1.0.0.jar.
          Eli Collins made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-8886 [ HADOOP-8886 ]
          thilee created issue -

            People

            • Assignee:
              thilee
              Reporter:
              thilee
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development