Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: flink, general
    • Labels:
      None

      Issue Links

        Activity

        Hide
        rmetzger Robert Metzger added a comment -

        What is the status of the puppet recipes for Flink?

        I'm happy to answer any Flink questions if they arise while creating the recipes.

        Show
        rmetzger Robert Metzger added a comment - What is the status of the puppet recipes for Flink? I'm happy to answer any Flink questions if they arise while creating the recipes.
        Hide
        Bhupendra Singh Bhupendra Singh added a comment -

        Working on it,almost done with the recipes.Facing a little issue in getting taskmanager talk to the jobmanager. Will have the patch ready by tomorrow.

        Show
        Bhupendra Singh Bhupendra Singh added a comment - Working on it,almost done with the recipes.Facing a little issue in getting taskmanager talk to the jobmanager. Will have the patch ready by tomorrow.
        Hide
        rmetzger Robert Metzger added a comment -

        Cool!
        What's the issue with the taskmanager <--> jobmanager? Maybe I can help.

        Show
        rmetzger Robert Metzger added a comment - Cool! What's the issue with the taskmanager <--> jobmanager? Maybe I can help.
        Hide
        Bhupendra Singh Bhupendra Singh added a comment -

        Logs for jobmanager and taskmanager

        Show
        Bhupendra Singh Bhupendra Singh added a comment - Logs for jobmanager and taskmanager
        Hide
        Bhupendra Singh Bhupendra Singh added a comment -

        Attached logs for your reference.

        Show
        Bhupendra Singh Bhupendra Singh added a comment - Attached logs for your reference.
        Hide
        rmetzger Robert Metzger added a comment -

        Thank you. Have you checked whether the virtual machine running the Taskmanager can connect to 10.10.10.11 ?
        10.10.10.12 (TM machine) tries to connect to 10.10.10.11:6123, but that connection is refused.

        I think there is an issue with the network connection setup.

        Show
        rmetzger Robert Metzger added a comment - Thank you. Have you checked whether the virtual machine running the Taskmanager can connect to 10.10.10.11 ? 10.10.10.12 (TM machine) tries to connect to 10.10.10.11:6123, but that connection is refused. I think there is an issue with the network connection setup.
        Hide
        Bhupendra Singh Bhupendra Singh added a comment -

        I am successfully able to ping the 10.10.10.11 from the TM Machine, but not sure if this is firewall issue or do we have to do port forwarding.?

        Show
        Bhupendra Singh Bhupendra Singh added a comment - I am successfully able to ping the 10.10.10.11 from the TM Machine, but not sure if this is firewall issue or do we have to do port forwarding.?
        Hide
        rmetzger Robert Metzger added a comment -

        I don't know your setup.
        You can also use telnet to try to connect from the TM machine to the JM port.

        Show
        rmetzger Robert Metzger added a comment - I don't know your setup. You can also use telnet to try to connect from the TM machine to the JM port.
        Hide
        rmetzger Robert Metzger added a comment -

        Disabling firewalld resolves the issue in the vagrant VMs.

        Show
        rmetzger Robert Metzger added a comment - Disabling firewalld resolves the issue in the vagrant VMs.
        Hide
        rmetzger Robert Metzger added a comment -

        I'm starting to work on this issue.

        Show
        rmetzger Robert Metzger added a comment - I'm starting to work on this issue.
        Hide
        Vikramnegi_87@yahoo.com Vikram (Inactive) added a comment -

        Who आर you bastered
        Sent via Micromax

        Show
        Vikramnegi_87@yahoo.com Vikram (Inactive) added a comment - Who आर you bastered Sent via Micromax
        Hide
        Bhupendra Singh Bhupendra Singh added a comment -

        Hi Robert,
        Apology for the delay, will upload the patch for recipes today.You can review the same if there are any changes required.

        Show
        Bhupendra Singh Bhupendra Singh added a comment - Hi Robert, Apology for the delay, will upload the patch for recipes today.You can review the same if there are any changes required.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user rmetzger opened a pull request:

        https://github.com/apache/bigtop/pull/111

        BIGTOP-2357: Add Apache Flink to Bigtop / Create puppet recipes

        With this change, I'm adding some basic puppet scripts for deploying flink.

        I've tested the change using the `bigtop-deploy/vm/vagrant-puppet-vm/`. Its the first time that I'm working with puppet, so any feedback is very welcome.

        While testing, I also came across https://issues.apache.org/jira/browse/BIGTOP-2157. Please let me know if the fix I'm suggesting in this PR is appropriate.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/rmetzger/bigtop bigtop2357

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/bigtop/pull/111.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #111


        commit ccdb13eaf420214abf0050dd2e38e91c0d037a1c
        Author: Robert Metzger <rmetzger@apache.org>
        Date: 2016-05-24T12:51:48Z

        BIGTOP-2357: Add Apache Flink to Bigtop / Create puppet recipes


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user rmetzger opened a pull request: https://github.com/apache/bigtop/pull/111 BIGTOP-2357 : Add Apache Flink to Bigtop / Create puppet recipes With this change, I'm adding some basic puppet scripts for deploying flink. I've tested the change using the `bigtop-deploy/vm/vagrant-puppet-vm/`. Its the first time that I'm working with puppet, so any feedback is very welcome. While testing, I also came across https://issues.apache.org/jira/browse/BIGTOP-2157 . Please let me know if the fix I'm suggesting in this PR is appropriate. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/bigtop bigtop2357 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/111.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #111 commit ccdb13eaf420214abf0050dd2e38e91c0d037a1c Author: Robert Metzger <rmetzger@apache.org> Date: 2016-05-24T12:51:48Z BIGTOP-2357 : Add Apache Flink to Bigtop / Create puppet recipes
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user ktsakalozos commented on the pull request:

        https://github.com/apache/bigtop/pull/111#issuecomment-222458643

        Thank you for doing this your work. It will be very helpful to us (juju team) since we plan to create a charm for Flink soon.

        Show
        githubbot ASF GitHub Bot added a comment - Github user ktsakalozos commented on the pull request: https://github.com/apache/bigtop/pull/111#issuecomment-222458643 Thank you for doing this your work. It will be very helpful to us (juju team) since we plan to create a charm for Flink soon.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rmetzger commented on the pull request:

        https://github.com/apache/bigtop/pull/111#issuecomment-222460271

        Great, looking forward to having a charm for Flink.

        Please let me know if you find any issues with these puppet recipes.

        Show
        githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the pull request: https://github.com/apache/bigtop/pull/111#issuecomment-222460271 Great, looking forward to having a charm for Flink. Please let me know if you find any issues with these puppet recipes.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rmetzger commented on the issue:

        https://github.com/apache/bigtop/pull/111

        @rvs any chances you can have a look or find somebody who can?

        Show
        githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/bigtop/pull/111 @rvs any chances you can have a look or find somebody who can?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rvs commented on the issue:

        https://github.com/apache/bigtop/pull/111

        Will definitely take a look over the weekend once I'm back in US.

        Show
        githubbot ASF GitHub Bot added a comment - Github user rvs commented on the issue: https://github.com/apache/bigtop/pull/111 Will definitely take a look over the weekend once I'm back in US.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rmetzger commented on the issue:

        https://github.com/apache/bigtop/pull/111

        Great, thank you

        Show
        githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/bigtop/pull/111 Great, thank you
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rmetzger commented on the issue:

        https://github.com/apache/bigtop/pull/111

        Ping

        Show
        githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/bigtop/pull/111 Ping
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rvs commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/111#discussion_r67065894

        — Diff: bigtop-deploy/puppet/modules/flink/manifests/init.pp —
        @@ -0,0 +1,64 @@
        +# The ASF licenses this file to You under the Apache License, Version 2.0
        +# (the "License"); you may not use this file except in compliance with
        +# the License. You may obtain a copy of the License at
        +#
        +# http://www.apache.org/licenses/LICENSE-2.0
        +#
        +# Unless required by applicable law or agreed to in writing, software
        +# distributed under the License is distributed on an "AS IS" BASIS,
        +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        +# See the License for the specific language governing permissions and
        +# limitations under the License.
        +class flink {
        +
        + class deploy ($roles) {
        + if ("flink-jobmanager" in $roles)

        { + include flink::jobmanager + }

        +
        + if ("flink-taskmanager" in $roles)

        { + include flink::taskmanager + }

        + }
        +
        + class common($jobmanager_host, $jobmanager_port, $ui_port) {
        + # make sure flink is installed
        + package

        { "flink": + ensure => latest + }

        +
        +
        + # set values in flink-conf.yaml
        + $tmp_dirs = hadoop::hadoop_storage_dirs
        — End diff –

        I'd be much more comfortable if this was passed in as a parameter from the outside of the puppet Flink class. Can you please do it that way?

        Show
        githubbot ASF GitHub Bot added a comment - Github user rvs commented on a diff in the pull request: https://github.com/apache/bigtop/pull/111#discussion_r67065894 — Diff: bigtop-deploy/puppet/modules/flink/manifests/init.pp — @@ -0,0 +1,64 @@ +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +class flink { + + class deploy ($roles) { + if ("flink-jobmanager" in $roles) { + include flink::jobmanager + } + + if ("flink-taskmanager" in $roles) { + include flink::taskmanager + } + } + + class common($jobmanager_host, $jobmanager_port, $ui_port) { + # make sure flink is installed + package { "flink": + ensure => latest + } + + + # set values in flink-conf.yaml + $tmp_dirs = hadoop::hadoop_storage_dirs — End diff – I'd be much more comfortable if this was passed in as a parameter from the outside of the puppet Flink class. Can you please do it that way?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rvs commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/111#discussion_r67066106

        — Diff: bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile —
        @@ -112,7 +113,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

        bigtop.vm.provider :virtualbox do |vb|
        vb.customize ["modifyvm", :id, "--memory", CONF['memory_size']]

        • vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']]
          + vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']]
            • End diff –

        This appears to be just a whitespace change - if you'll be fixing the comment above, can you please remove this spurious change as well?

        Show
        githubbot ASF GitHub Bot added a comment - Github user rvs commented on a diff in the pull request: https://github.com/apache/bigtop/pull/111#discussion_r67066106 — Diff: bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile — @@ -112,7 +113,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| bigtop.vm.provider :virtualbox do |vb| vb.customize ["modifyvm", :id, "--memory", CONF ['memory_size'] ] vb.customize ['modifyvm', :id, '--cpus', CONF ['number_cpus'] ] + vb.customize ['modifyvm', :id, '--cpus', CONF ['number_cpus'] ] End diff – This appears to be just a whitespace change - if you'll be fixing the comment above, can you please remove this spurious change as well?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rmetzger commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/111#discussion_r67134651

        — Diff: bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile —
        @@ -112,7 +113,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

        bigtop.vm.provider :virtualbox do |vb|
        vb.customize ["modifyvm", :id, "--memory", CONF['memory_size']]

        • vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']]
          + vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']]
            • End diff –

        Sure, I will undo the change. Its not an accidental change. The file is using spaces for indentation, and only this line uses tabs (which makes it look unaligned in some editors)

        Show
        githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on a diff in the pull request: https://github.com/apache/bigtop/pull/111#discussion_r67134651 — Diff: bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile — @@ -112,7 +113,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| bigtop.vm.provider :virtualbox do |vb| vb.customize ["modifyvm", :id, "--memory", CONF ['memory_size'] ] vb.customize ['modifyvm', :id, '--cpus', CONF ['number_cpus'] ] + vb.customize ['modifyvm', :id, '--cpus', CONF ['number_cpus'] ] End diff – Sure, I will undo the change. Its not an accidental change. The file is using spaces for indentation, and only this line uses tabs (which makes it look unaligned in some editors)
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rmetzger commented on the issue:

        https://github.com/apache/bigtop/pull/111

        Thank you for the review. I addressed your comments.

        Show
        githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/bigtop/pull/111 Thank you for the review. I addressed your comments.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rvs commented on the issue:

        https://github.com/apache/bigtop/pull/111

        Great! If nobody has any other concerns – I'll be merging it in tomorrow.

        Show
        githubbot ASF GitHub Bot added a comment - Github user rvs commented on the issue: https://github.com/apache/bigtop/pull/111 Great! If nobody has any other concerns – I'll be merging it in tomorrow.
        Hide
        rvs Roman Shaposhnik added a comment -

        This has been committed! Thanks a lot for the patch! You can close your PR now.

        Show
        rvs Roman Shaposhnik added a comment - This has been committed! Thanks a lot for the patch! You can close your PR now.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rmetzger closed the pull request at:

        https://github.com/apache/bigtop/pull/111

        Show
        githubbot ASF GitHub Bot added a comment - Github user rmetzger closed the pull request at: https://github.com/apache/bigtop/pull/111

          People

          • Assignee:
            rmetzger Robert Metzger
            Reporter:
            Bhupendra Singh Bhupendra Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development