Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1970

Ignite IGFS now fully supports mutiltenancy: deployment should configure it with HDFS backing

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: deployment
    • Labels:
      None

      Description

      IGFS can work with HCFS as a secondary storage. Let's improve the puppet recipes to configure Ignite node in that way.

      1. BIGTOP-1970.patch
        10 kB
        Konstantin Boudnik

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        Pushed to the master. Thanks for the review, Evans!

        Show
        cos Konstantin Boudnik added a comment - Pushed to the master. Thanks for the review, Evans!
        Hide
        cos Konstantin Boudnik added a comment -

        ah... - you're right of course. Thanks for spotting. I will commit the patch now.

        Show
        cos Konstantin Boudnik added a comment - ah... - you're right of course. Thanks for spotting. I will commit the patch now.
        Hide
        evans_ye Evans Ye added a comment -

        The patch looks good. And my test works, too. So here's my +1.

        BTW(This is tangential), I think there's a minor error in the training script:

        # 7.4 Validate (optional)
        time hadoop jar $MR_JAR teravalidate /user/root/sort.out.imc \
             /user/root/validate.report.imc
        

        The validate output should be /user/root/validate.report.t instead of {{ /user/root/validate.report.imc}}.

        Show
        evans_ye Evans Ye added a comment - The patch looks good. And my test works, too. So here's my +1. BTW(This is tangential), I think there's a minor error in the training script: # 7.4 Validate (optional) time hadoop jar $MR_JAR teravalidate /user/root/sort.out.imc \ /user/root/validate.report.imc The validate output should be /user/root/validate.report.t instead of {{ /user/root/validate.report.imc}}.
        Hide
        cos Konstantin Boudnik added a comment - - edited

        you can quickly test it by running a teragen or something, using Ignite's MapReduce framework. You can do it as explained in out Bigtop workshop for Budapest, ie

        time HADOOP_CONFIG_DIR=/etc/hadoop/ignite.client.conf hadoop jar $MR_JAR \
             teragen 100000 /user/root/tera.out.imc
        

        where $MR is pointing to /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar

        Show
        cos Konstantin Boudnik added a comment - - edited you can quickly test it by running a teragen or something, using Ignite's MapReduce framework. You can do it as explained in out Bigtop workshop for Budapest, ie time HADOOP_CONFIG_DIR=/etc/hadoop/ignite.client.conf hadoop jar $MR_JAR \ teragen 100000 /user/root/tera.out.imc where $MR is pointing to /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar
        Hide
        evans_ye Evans Ye added a comment -

        Hmm, the deploy works. But I'm not sure whether or not the feature is working.
        Do I need to write some java code to cache data or there's another way to test it?

        Show
        evans_ye Evans Ye added a comment - Hmm, the deploy works. But I'm not sure whether or not the feature is working. Do I need to write some java code to cache data or there's another way to test it?
        Hide
        cos Konstantin Boudnik added a comment -

        1.2 is enough, actually.

        Show
        cos Konstantin Boudnik added a comment - 1.2 is enough, actually.
        Hide
        evans_ye Evans Ye added a comment -

        Does ignite 1.3.0 packages a requirement for this to work?

        Show
        evans_ye Evans Ye added a comment - Does ignite 1.3.0 packages a requirement for this to work?
        Hide
        cos Konstantin Boudnik added a comment -

        Here's the patch to provide for the functionality. Tested on official ubuntu-14.04 container and is working as expected.

        Show
        cos Konstantin Boudnik added a comment - Here's the patch to provide for the functionality. Tested on official ubuntu-14.04 container and is working as expected.

          People

          • Assignee:
            cos Konstantin Boudnik
            Reporter:
            cos Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development