Bigtop
  1. Bigtop
  2. BIGTOP-841

hadoop-conf-pseudo missed configuration for capacity scheduler and historyserver

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Component/s: General
    • Labels:
      None

      Description

      Configuration package for pseudo distributed cluster misses two important pieces: capacity scheduler config and needed properties for historyserver (from mapred-site.xml file).

      1. BIGTOP-841.patch
        4 kB
        Konstantin Boudnik
      2. BIGTOP-841-rvs.patch.txt
        2 kB
        Roman Shaposhnik

        Activity

        Hide
        Konstantin Boudnik added a comment -

        While missing capacity-scheduler.xml simply renders yarn unusable, the historyserver configuration provides just a minor annoyance: jobs take longer trying to connect to non-bound port 10020.

        Show
        Konstantin Boudnik added a comment - While missing capacity-scheduler.xml simply renders yarn unusable, the historyserver configuration provides just a minor annoyance: jobs take longer trying to connect to non-bound port 10020.
        Hide
        Konstantin Boudnik added a comment -

        Here's the fix for the configurations

        Show
        Konstantin Boudnik added a comment - Here's the fix for the configurations
        Hide
        Mark Grover added a comment -

        Cos, thanks for this!

        The mapreduce properties look good to me.

        I had a question about capacity-scheduler.xml. This file seems to be bundled in the hadoop tarball (atleast the 2.0.2-alpha one). Is there a particular reason we are forking the config instead of using the one from the tarball?

        Show
        Mark Grover added a comment - Cos, thanks for this! The mapreduce properties look good to me. I had a question about capacity-scheduler.xml. This file seems to be bundled in the hadoop tarball (atleast the 2.0.2-alpha one). Is there a particular reason we are forking the config instead of using the one from the tarball?
        Hide
        Konstantin Boudnik added a comment -

        hmm... good point. I guess I have just found a fastest way of doing this

        Show
        Konstantin Boudnik added a comment - hmm... good point. I guess I have just found a fastest way of doing this
        Hide
        Roman Shaposhnik added a comment -

        We may need to bump to branch-2 at this point and see whether capacity-scheduler.xml issue exists upstream.

        Show
        Roman Shaposhnik added a comment - We may need to bump to branch-2 at this point and see whether capacity-scheduler.xml issue exists upstream.
        Hide
        Konstantin Boudnik added a comment -

        well, the capacity-scheduler issue will exist because the default scheduler is capacity one and without this config it won't work.

        Show
        Konstantin Boudnik added a comment - well, the capacity-scheduler issue will exist because the default scheduler is capacity one and without this config it won't work.
        Hide
        Konstantin Boudnik added a comment -

        I had a question about capacity-scheduler.xml. This file seems to be bundled in the hadoop tarball

        Do we copy hdfs-site.xml and all others from the Hadoop tarball? I think this is a flawed argument, because conf-pseudo is something that is introduced by BT packaging. The patch fixes the real issue that just yet has been discovered by 2.0.3 rc testing, so we are ahead of the curve already

        If there's no objections I am going to commit it soon.

        Show
        Konstantin Boudnik added a comment - I had a question about capacity-scheduler.xml. This file seems to be bundled in the hadoop tarball Do we copy hdfs-site.xml and all others from the Hadoop tarball? I think this is a flawed argument, because conf-pseudo is something that is introduced by BT packaging. The patch fixes the real issue that just yet has been discovered by 2.0.3 rc testing, so we are ahead of the curve already If there's no objections I am going to commit it soon.
        Hide
        Roman Shaposhnik added a comment -

        Cos, can you take a look at the attached patch? I think it may be a bit better suited for our situation. What do you think?

        Show
        Roman Shaposhnik added a comment - Cos, can you take a look at the attached patch? I think it may be a bit better suited for our situation. What do you think?
        Hide
        Konstantin Boudnik added a comment - - edited

        Yeah, I suppose this is a better approach. Thanks for improving it!
        +1 apparently

        Show
        Konstantin Boudnik added a comment - - edited Yeah, I suppose this is a better approach. Thanks for improving it! +1 apparently
        Hide
        Mark Grover added a comment -

        +1 (non-committer) on the latest patch

        Show
        Mark Grover added a comment - +1 (non-committer) on the latest patch

          People

          • Assignee:
            Konstantin Boudnik
            Reporter:
            Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development