Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:

      Activity

      Hide
      githubbot ASF GitHub Bot added a comment -

      GitHub user fsareshwala opened a pull request:

      https://github.com/apache/bigtop/pull/82

      BIGTOP-2293. Add puppet recipes for qfs components

      This change adds puppet recipes to the bigtop source code for qfs components:
      qfs-metaserver, qfs-chunkserver, and qfs-client. The puppet recipes install the
      necessary packages, create necessary files and directories, generate
      configuration with the correct ports and paths, and finally start the services
      so that they can talk to each other.

      I have tested to make sure that the systems work properly by logging in manually
      to the docker container and interacting with the chunkserver and metaserver
      using the qfs client. In my next commit, I will add testing to the bigtop
      codebase so that we don't need to use manual inspection to ensure that things
      are working correctly.

      You can merge this pull request into a Git repository by running:

      $ git pull https://github.com/fsareshwala/bigtop qfs

      Alternatively you can review and apply these changes as the patch at:

      https://github.com/apache/bigtop/pull/82.patch

      To close this pull request, make a commit to your master/trunk branch
      with (at least) the following in the commit message:

      This closes #82


      commit 72763415fffb58adfbf303cb1bac8e4f0a885da5
      Author: Faraaz Sareshwala <fsareshwala@quantcast.com>
      Date: 2016-02-02T22:19:27Z

      BIGTOP-2293. Add puppet recipes for qfs components

      This change adds puppet recipes to the bigtop source code for qfs components:
      qfs-metaserver, qfs-chunkserver, and qfs-client. The puppet recipes install the
      necessary packages, create necessary files and directories, generate
      configuration with the correct ports and paths, and finally start the services
      so that they can talk to each other.

      I have tested to make sure that the systems work properly by logging in manually
      to the docker container and interacting with the chunkserver and metaserver
      using the qfs client. In my next commit, I will add testing to the bigtop
      codebase so that we don't need to use manual inspection to ensure that things
      are working correctly.


      Show
      githubbot ASF GitHub Bot added a comment - GitHub user fsareshwala opened a pull request: https://github.com/apache/bigtop/pull/82 BIGTOP-2293 . Add puppet recipes for qfs components This change adds puppet recipes to the bigtop source code for qfs components: qfs-metaserver, qfs-chunkserver, and qfs-client. The puppet recipes install the necessary packages, create necessary files and directories, generate configuration with the correct ports and paths, and finally start the services so that they can talk to each other. I have tested to make sure that the systems work properly by logging in manually to the docker container and interacting with the chunkserver and metaserver using the qfs client. In my next commit, I will add testing to the bigtop codebase so that we don't need to use manual inspection to ensure that things are working correctly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fsareshwala/bigtop qfs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/82.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #82 commit 72763415fffb58adfbf303cb1bac8e4f0a885da5 Author: Faraaz Sareshwala <fsareshwala@quantcast.com> Date: 2016-02-02T22:19:27Z BIGTOP-2293 . Add puppet recipes for qfs components This change adds puppet recipes to the bigtop source code for qfs components: qfs-metaserver, qfs-chunkserver, and qfs-client. The puppet recipes install the necessary packages, create necessary files and directories, generate configuration with the correct ports and paths, and finally start the services so that they can talk to each other. I have tested to make sure that the systems work properly by logging in manually to the docker container and interacting with the chunkserver and metaserver using the qfs client. In my next commit, I will add testing to the bigtop codebase so that we don't need to use manual inspection to ensure that things are working correctly.
      Hide
      githubbot ASF GitHub Bot added a comment -

      Github user fsareshwala closed the pull request at:

      https://github.com/apache/bigtop/pull/82

      Show
      githubbot ASF GitHub Bot added a comment - Github user fsareshwala closed the pull request at: https://github.com/apache/bigtop/pull/82
      Hide
      githubbot ASF GitHub Bot added a comment -

      Github user fsareshwala commented on the pull request:

      https://github.com/apache/bigtop/pull/82#issuecomment-181655874

      Closing this pull request out so that I can fix the source branch.

      Show
      githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on the pull request: https://github.com/apache/bigtop/pull/82#issuecomment-181655874 Closing this pull request out so that I can fix the source branch.
      Hide
      githubbot ASF GitHub Bot added a comment -

      GitHub user fsareshwala opened a pull request:

      https://github.com/apache/bigtop/pull/84

      BIGTOP-2293. Add puppet recipes for qfs components

      This change adds puppet recipes to the bigtop source code for qfs components:
      qfs-metaserver, qfs-chunkserver, and qfs-client. The puppet recipes install the
      necessary packages, create necessary files and directories, generate
      configuration with the correct ports and paths, and finally start the services
      so that they can talk to each other.

      I have tested to make sure that the systems work properly by logging in manually
      to the docker container and interacting with the chunkserver and metaserver
      using the qfs client. In my next commit, I will add testing to the bigtop
      codebase so that we don't need to use manual inspection to ensure that things
      are working correctly.

      You can merge this pull request into a Git repository by running:

      $ git pull https://github.com/fsareshwala/bigtop qfs-puppet

      Alternatively you can review and apply these changes as the patch at:

      https://github.com/apache/bigtop/pull/84.patch

      To close this pull request, make a commit to your master/trunk branch
      with (at least) the following in the commit message:

      This closes #84


      commit 72763415fffb58adfbf303cb1bac8e4f0a885da5
      Author: Faraaz Sareshwala <fsareshwala@quantcast.com>
      Date: 2016-02-02T22:19:27Z

      BIGTOP-2293. Add puppet recipes for qfs components

      This change adds puppet recipes to the bigtop source code for qfs components:
      qfs-metaserver, qfs-chunkserver, and qfs-client. The puppet recipes install the
      necessary packages, create necessary files and directories, generate
      configuration with the correct ports and paths, and finally start the services
      so that they can talk to each other.

      I have tested to make sure that the systems work properly by logging in manually
      to the docker container and interacting with the chunkserver and metaserver
      using the qfs client. In my next commit, I will add testing to the bigtop
      codebase so that we don't need to use manual inspection to ensure that things
      are working correctly.


      Show
      githubbot ASF GitHub Bot added a comment - GitHub user fsareshwala opened a pull request: https://github.com/apache/bigtop/pull/84 BIGTOP-2293 . Add puppet recipes for qfs components This change adds puppet recipes to the bigtop source code for qfs components: qfs-metaserver, qfs-chunkserver, and qfs-client. The puppet recipes install the necessary packages, create necessary files and directories, generate configuration with the correct ports and paths, and finally start the services so that they can talk to each other. I have tested to make sure that the systems work properly by logging in manually to the docker container and interacting with the chunkserver and metaserver using the qfs client. In my next commit, I will add testing to the bigtop codebase so that we don't need to use manual inspection to ensure that things are working correctly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fsareshwala/bigtop qfs-puppet Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/84.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #84 commit 72763415fffb58adfbf303cb1bac8e4f0a885da5 Author: Faraaz Sareshwala <fsareshwala@quantcast.com> Date: 2016-02-02T22:19:27Z BIGTOP-2293 . Add puppet recipes for qfs components This change adds puppet recipes to the bigtop source code for qfs components: qfs-metaserver, qfs-chunkserver, and qfs-client. The puppet recipes install the necessary packages, create necessary files and directories, generate configuration with the correct ports and paths, and finally start the services so that they can talk to each other. I have tested to make sure that the systems work properly by logging in manually to the docker container and interacting with the chunkserver and metaserver using the qfs client. In my next commit, I will add testing to the bigtop codebase so that we don't need to use manual inspection to ensure that things are working correctly.
      Hide
      githubbot ASF GitHub Bot added a comment -

      Github user fsareshwala commented on the pull request:

      https://github.com/apache/bigtop/pull/82#issuecomment-181656247

      New puppet recipes pull request: https://github.com/apache/bigtop/pull/84

      Show
      githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on the pull request: https://github.com/apache/bigtop/pull/82#issuecomment-181656247 New puppet recipes pull request: https://github.com/apache/bigtop/pull/84
      Hide
      cos Konstantin Boudnik added a comment -
      • commit message is a way too vast (as elsewhere)
      • what would happen is someone enables hdfs and qfs simultaneously? Shall there be a way for QFS recipes to check if HDFS is enabled?
        Without running it, seems to be ok. But let's wait until the package code is ready, so we can validate this.
      Show
      cos Konstantin Boudnik added a comment - commit message is a way too vast (as elsewhere) what would happen is someone enables hdfs and qfs simultaneously? Shall there be a way for QFS recipes to check if HDFS is enabled? Without running it, seems to be ok. But let's wait until the package code is ready, so we can validate this.
      Hide
      fsareshwala Faraaz Sareshwala added a comment -

      Thanks Cos! I've updated the commit message to just be the first line and removed the extra description.

      HDFS and QFS simultaneously works just fine:

      [root@bigtop1 /]# hadoop fs -ls /
      Found 7 items
      drwxr-xr-x   - hdfs  hadoop          0 2016-02-17 00:00 /apps
      drwxrwxrwx   - hdfs  hadoop          0 2016-02-17 00:00 /benchmarks
      drwxr-xr-x   - hbase hbase           0 2016-02-17 00:00 /hbase
      drwxr-xr-x   - solr  solr            0 2016-02-17 00:00 /solr
      drwxrwxrwt   - hdfs  hadoop          0 2016-02-17 00:00 /tmp
      drwxr-xr-x   - hdfs  hadoop          0 2016-02-17 00:00 /user
      drwxr-xr-x   - hdfs  hadoop          0 2016-02-17 00:00 /var
      [root@bigtop1 /]# hadoop-qfs fs -ls /
      Found 1 items
      drwx------   - root root          0 2016-02-17 00:00 /dumpster
      

      To work with qfs, just use the hadoop-qfs command instead of the hadoop command. Jobs submitted with hadoop-qfs will run on hadoop, but will instruct hadoop to use qfs as the backend store instead of HDFS.

      Show
      fsareshwala Faraaz Sareshwala added a comment - Thanks Cos! I've updated the commit message to just be the first line and removed the extra description. HDFS and QFS simultaneously works just fine: [root@bigtop1 /]# hadoop fs -ls / Found 7 items drwxr-xr-x - hdfs hadoop 0 2016-02-17 00:00 /apps drwxrwxrwx - hdfs hadoop 0 2016-02-17 00:00 /benchmarks drwxr-xr-x - hbase hbase 0 2016-02-17 00:00 /hbase drwxr-xr-x - solr solr 0 2016-02-17 00:00 /solr drwxrwxrwt - hdfs hadoop 0 2016-02-17 00:00 /tmp drwxr-xr-x - hdfs hadoop 0 2016-02-17 00:00 /user drwxr-xr-x - hdfs hadoop 0 2016-02-17 00:00 / var [root@bigtop1 /]# hadoop-qfs fs -ls / Found 1 items drwx------ - root root 0 2016-02-17 00:00 /dumpster To work with qfs, just use the hadoop-qfs command instead of the hadoop command. Jobs submitted with hadoop-qfs will run on hadoop, but will instruct hadoop to use qfs as the backend store instead of HDFS.
      Hide
      cos Konstantin Boudnik added a comment -

      Oh, that's nice! I thought QFS is a drop-in replacement. Cool!

      Show
      cos Konstantin Boudnik added a comment - Oh, that's nice! I thought QFS is a drop-in replacement . Cool!
      Hide
      fsareshwala Faraaz Sareshwala added a comment -

      Technically, it is a drop-in replacement in that at the moment, you can't run a hadoop job that talks to both hdfs and qfs at the same time. We have that support internally inside Quantcast but I don't think it works within hadoop yet. Please correct me if I'm wrong.

      A job submitted using hadoop-qfs will only write to qfs and a job submitted using hadoop will only write to hdfs. However, both filesystems can run simultaneously and it is possible to copy files from hdfs to qfs using hadoop-qfs distcp.

      I want to make sure users have as good an experience using qfs as possible. I didn't want to make any assumptions for them that turning qfs on means that they want to also turn hdfs off. I just had qfs turn on and provide a command, hadoop-qfs, which does all the heavy lifting to get qfs interfacing with hadoop properly. I felt this would stay out of the user's way as best as possible and would be the smallest amount of change for them to get used to. Do you think this design is reasonable? Should I change it in any way to be more bigtop-like?

      Show
      fsareshwala Faraaz Sareshwala added a comment - Technically, it is a drop-in replacement in that at the moment, you can't run a hadoop job that talks to both hdfs and qfs at the same time. We have that support internally inside Quantcast but I don't think it works within hadoop yet. Please correct me if I'm wrong. A job submitted using hadoop-qfs will only write to qfs and a job submitted using hadoop will only write to hdfs. However, both filesystems can run simultaneously and it is possible to copy files from hdfs to qfs using hadoop-qfs distcp . I want to make sure users have as good an experience using qfs as possible. I didn't want to make any assumptions for them that turning qfs on means that they want to also turn hdfs off. I just had qfs turn on and provide a command, hadoop-qfs , which does all the heavy lifting to get qfs interfacing with hadoop properly. I felt this would stay out of the user's way as best as possible and would be the smallest amount of change for them to get used to. Do you think this design is reasonable? Should I change it in any way to be more bigtop-like?
      Hide
      cos Konstantin Boudnik added a comment -

      I think it all makes perfect sense - thanks for taking time explaining all this!

      Show
      cos Konstantin Boudnik added a comment - I think it all makes perfect sense - thanks for taking time explaining all this!
      Hide
      oflebbe Olaf Flebbe added a comment - - edited

      From the puppet side it seems not to affect other modules. I am not in the position to check if it is valid qfs configuration, but the author surely is. So +1 from me.

      Show
      oflebbe Olaf Flebbe added a comment - - edited From the puppet side it seems not to affect other modules. I am not in the position to check if it is valid qfs configuration, but the author surely is. So +1 from me.
      Hide
      oflebbe Olaf Flebbe added a comment -

      committed

      Show
      oflebbe Olaf Flebbe added a comment - committed
      Hide
      githubbot ASF GitHub Bot added a comment -

      Github user fsareshwala commented on the pull request:

      https://github.com/apache/bigtop/pull/84#issuecomment-201706294

      This has been committed by @oflebbe into the BigTop repository. I am closing this pull request.

      Show
      githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on the pull request: https://github.com/apache/bigtop/pull/84#issuecomment-201706294 This has been committed by @oflebbe into the BigTop repository. I am closing this pull request.
      Hide
      githubbot ASF GitHub Bot added a comment -

      Github user fsareshwala closed the pull request at:

      https://github.com/apache/bigtop/pull/84

      Show
      githubbot ASF GitHub Bot added a comment - Github user fsareshwala closed the pull request at: https://github.com/apache/bigtop/pull/84

        People

        • Assignee:
          fsareshwala Faraaz Sareshwala
          Reporter:
          fsareshwala Faraaz Sareshwala
        • Votes:
          0 Vote for this issue
          Watchers:
          4 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development