Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Scripts
    • Labels:
      None
    • Environment:

      Redhat EL 4.x, Java 1.6

      Description

      Since JIRA-468, bin/chukwa stop no longer works for collectors and agents. Can we change kill -1 back to kill -TERM?

      1. CHUKWA-483.1.patch
        0.4 kB
        Bill Graham
      2. CHUKWA-483.threaddump.txt
        5 kB
        Bill Graham
      3. chukwa-agent-thread-dump.txt
        15 kB
        Eric Yang

        Activity

        Hide
        asrabkin Ari Rabkin added a comment -

        Huhm. I haven't been particularly paying attention to the script-side of things.

        Eric, can you comment on why the kill -1 isn't killing the process? Didn't think we were masking that signal.

        Show
        asrabkin Ari Rabkin added a comment - Huhm. I haven't been particularly paying attention to the script-side of things. Eric, can you comment on why the kill -1 isn't killing the process? Didn't think we were masking that signal.
        Hide
        eyang Eric Yang added a comment -

        kill -1 works fine for me with Java 6, and Redhat EL 5.1. Bill, could you do a kill -3 after kill -1 to see what the process is doing? Perhaps the existing process was taking a while to terminate?

        Show
        eyang Eric Yang added a comment - kill -1 works fine for me with Java 6, and Redhat EL 5.1. Bill, could you do a kill -3 after kill -1 to see what the process is doing? Perhaps the existing process was taking a while to terminate?
        Hide
        billgraham Bill Graham added a comment -

        Sure. Here's a thread dump of an agent with no adaptors. I start it, try to stop it with bin/chukwa (using kill -1), the run kill -3.

        Show
        billgraham Bill Graham added a comment - Sure. Here's a thread dump of an agent with no adaptors. I start it, try to stop it with bin/chukwa (using kill -1), the run kill -3.
        Hide
        eyang Eric Yang added a comment -

        Looks like DestroyJavaVM thread was running. This means it should be able to terminate the program. Maybe it take longer than expected to clean up.

        Show
        eyang Eric Yang added a comment - Looks like DestroyJavaVM thread was running. This means it should be able to terminate the program. Maybe it take longer than expected to clean up.
        Hide
        billgraham Bill Graham added a comment -

        The DestroyJavaVM thread is always running, before I even execute a 'kill -1' command. 'kill -1' just isn't stopping this process, even after I wait 5 minutes.

        Show
        billgraham Bill Graham added a comment - The DestroyJavaVM thread is always running, before I even execute a 'kill -1' command. 'kill -1' just isn't stopping this process, even after I wait 5 minutes.
        Hide
        billgraham Bill Graham added a comment -

        The DestroyJavaVM thread is always running, before I even execute a 'kill
        -1' command. 'kill -1' just isn't stopping this process, even after I wait 5
        minutes.

        Show
        billgraham Bill Graham added a comment - The DestroyJavaVM thread is always running, before I even execute a 'kill -1' command. 'kill -1' just isn't stopping this process, even after I wait 5 minutes.
        Hide
        eyang Eric Yang added a comment -

        The DestroyJavaVM thread shouldn't be created unless the main thread is prepared to exit. Do you have special agent adaptor and collector writer which is preventing the system to exit (i.e. forget to free jni resource, etc) I will enclose my thread dump for reference.

        Show
        eyang Eric Yang added a comment - The DestroyJavaVM thread shouldn't be created unless the main thread is prepared to exit. Do you have special agent adaptor and collector writer which is preventing the system to exit (i.e. forget to free jni resource, etc) I will enclose my thread dump for reference.
        Hide
        eyang Eric Yang added a comment -

        Thread dump, only UTF8CharFileTailing adaptor and ExecPluginAdaptor are running, before kill -1

        Show
        eyang Eric Yang added a comment - Thread dump, only UTF8CharFileTailing adaptor and ExecPluginAdaptor are running, before kill -1
        Hide
        billgraham Bill Graham added a comment -

        If I check out the trunk, package and run bin/startAgents.sh without any initial adaptors, then issue kill -3, I see the DestroyJavaVM thread. It's always there. I see the same thing if I kill -3 a collector process. I don't see it on a DemuxManager process for what it's worth.

        Show
        billgraham Bill Graham added a comment - If I check out the trunk, package and run bin/startAgents.sh without any initial adaptors, then issue kill -3, I see the DestroyJavaVM thread. It's always there. I see the same thing if I kill -3 a collector process. I don't see it on a DemuxManager process for what it's worth.
        Hide
        kirktrue Kirk True added a comment -

        I'm also seeing this behavior starting in 0.4.0 on Fedora. Let me know if there's any useful information I can add.

        Show
        kirktrue Kirk True added a comment - I'm also seeing this behavior starting in 0.4.0 on Fedora. Let me know if there's any useful information I can add.
        Hide
        eyang Eric Yang added a comment -

        Bill and Kirk, do you guys see core.[pid] files pile up in CHUKWA_HOME directory, if the script uses -TERM on your OS? If you don't see the core file, then I am ok with this patch.

        Show
        eyang Eric Yang added a comment - Bill and Kirk, do you guys see core. [pid] files pile up in CHUKWA_HOME directory, if the script uses -TERM on your OS? If you don't see the core file, then I am ok with this patch.
        Hide
        billgraham Bill Graham added a comment -

        I do not see core files on my OS when using -TERM.

        Show
        billgraham Bill Graham added a comment - I do not see core files on my OS when using -TERM.
        Hide
        asrabkin Ari Rabkin added a comment -

        As per eric's okay, I just committed this. Thanks Bill for the patch, also Kirk and Eric for testing.

        Show
        asrabkin Ari Rabkin added a comment - As per eric's okay, I just committed this. Thanks Bill for the patch, also Kirk and Eric for testing.

          People

          • Assignee:
            billgraham Bill Graham
            Reporter:
            billgraham Bill Graham
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development