Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0
    • Component/s: None
    • Labels:
      None

      Description

      Starting Flume generates the FLUME_CLASSPATH variable dynamically, adding jars one by one.
      In some cases, the classpath can grow too long, sometimes exceeding ARG_MAX, maximum length of arguments for a new process ( http://www.in-ulm.de/~mascheck/various/argmax/ )
      Thus, being unable to start Flume:
      flume-ng/bin/flume-ng: line 239: /usr/java/jdk1.7.0_67/bin/java: Argument list too long

      1. FLUME-2913.patch
        1 kB
        Bessenyei Balázs Donát

        Activity

        Hide
        bessbd Bessenyei Balázs Donát added a comment -

        This patch omits expanding of paths. Only the wildcard paths get appended after applying the changes. Tested on vanilla CDH 5.8.1.

        Show
        bessbd Bessenyei Balázs Donát added a comment - This patch omits expanding of paths. Only the wildcard paths get appended after applying the changes. Tested on vanilla CDH 5.8.1.
        Hide
        mpercy Mike Percy added a comment -

        Hi Bessenyei Balázs Donát, thanks for the patch. It looks like this patch primarily removes exclusion of slf4j jars from the hadoop and hbase classpaths. Did you verify whether including multiple versions of slf4j api / log4j connectors doesn't cause a problem? Or do you have any explanation about this? From my recollection (a hazy one, admittedly) that used to cause slf4j to refuse to function.

        Thanks,
        Mike

        Show
        mpercy Mike Percy added a comment - Hi Bessenyei Balázs Donát , thanks for the patch. It looks like this patch primarily removes exclusion of slf4j jars from the hadoop and hbase classpaths. Did you verify whether including multiple versions of slf4j api / log4j connectors doesn't cause a problem? Or do you have any explanation about this? From my recollection (a hazy one, admittedly) that used to cause slf4j to refuse to function. Thanks, Mike
        Hide
        mpercy Mike Percy added a comment -

        Bessenyei Balázs Donát and I worked together offline to test this and it looks fine. When multiple versions of an slf4j plugin are on the classpath, even if they are different versions, one "wins" despite printing a warning and the logging functions as expected.

        Here is a test repo I put together to verify: https://github.com/mpercy/slf4j-test

        Show
        mpercy Mike Percy added a comment - Bessenyei Balázs Donát and I worked together offline to test this and it looks fine. When multiple versions of an slf4j plugin are on the classpath, even if they are different versions, one "wins" despite printing a warning and the logging functions as expected. Here is a test repo I put together to verify: https://github.com/mpercy/slf4j-test
        Hide
        mpercy Mike Percy added a comment -

        +1. I am about to commit this.

        Show
        mpercy Mike Percy added a comment - +1. I am about to commit this.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit f4f15bee1e4ba109c6a21f639668911a13241014 in flume's branch refs/heads/trunk from Mike Percy
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f4f15be ]

        FLUME-2913. Don't strip SLF4J from imported classpaths

        It is no longer necessary to strip other versions of SLF4J from
        classpaths imported from hadoop or hbase. With recent versions of SLF4J,
        when multiple connectors are found, one of them "wins" and the logging
        does not break. A warning is printed once, however. This is acceptable
        because having too long of a classpath can cause major problems on
        systems with a low ARG_MAX.

        (Bessenyei Balázs Donát via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit f4f15bee1e4ba109c6a21f639668911a13241014 in flume's branch refs/heads/trunk from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f4f15be ] FLUME-2913 . Don't strip SLF4J from imported classpaths It is no longer necessary to strip other versions of SLF4J from classpaths imported from hadoop or hbase. With recent versions of SLF4J, when multiple connectors are found, one of them "wins" and the logging does not break. A warning is printed once, however. This is acceptable because having too long of a classpath can cause major problems on systems with a low ARG_MAX. (Bessenyei Balázs Donát via Mike Percy)
        Hide
        mpercy Mike Percy added a comment -

        I just pushed this to trunk. Thanks for the patch, Bessenyei Balázs Donát!

        Show
        mpercy Mike Percy added a comment - I just pushed this to trunk. Thanks for the patch, Bessenyei Balázs Donát !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Flume-trunk-hbase-1 #159 (See https://builds.apache.org/job/Flume-trunk-hbase-1/159/)
        FLUME-2913. Don't strip SLF4J from imported classpaths (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f4f15bee1e4ba109c6a21f639668911a13241014)

        • bin/flume-ng
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #159 (See https://builds.apache.org/job/Flume-trunk-hbase-1/159/ ) FLUME-2913 . Don't strip SLF4J from imported classpaths (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f4f15bee1e4ba109c6a21f639668911a13241014 ) bin/flume-ng

          People

          • Assignee:
            bessbd Bessenyei Balázs Donát
            Reporter:
            bessbd Bessenyei Balázs Donát
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development