Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2.0
    • Fix Version/s: 0.2.0
    • Component/s: Data Collection
    • Labels:
      None

      Description

      Chukwa should have a command line tools todo some backfilling process.

      Parameters should be:

      • cluster
      • machine
      • adaptor
      • recordType
      • file

      The idea is to call all component within the same process and the output should be a dataSink file ready to be processed by Demux.

      1. CHUKWA-194.patch
        39 kB
        Jerome Boulon
      2. CHUKWA-194-1.patch
        41 kB
        Jerome Boulon
      3. CHUKWA-194-2.patch
        41 kB
        Ari Rabkin

        Activity

        Hide
        jboulon Jerome Boulon added a comment -

        Backfilling tool:

        Class:

        • BackfillingLoader
          Input:
        • String cluster = args[0];
        • String machine = args[1];
        • String adaptorName = args[2];
        • String recordType = args[3];
        • String logFile = args[4];

        Major change API change: for backfilling I need to know when an adaptor has finished posting data to the queue. So instead of create yet another shutdown method I have deprecated the hardStop and the shutdown method to add a new one: public long shutdown(AdaptorShutdownPolicy shutdownPolicy) throws AdaptorException;

        AdaptorShutdownPolicy are: HARD_STOP, GRACEFULLY, WAIT_TILL_FINISHED

        Minor change for the socket controller :
        s.setReuseAddress(true); for the socket controller

        Show
        jboulon Jerome Boulon added a comment - Backfilling tool: Class: BackfillingLoader Input: String cluster = args [0] ; String machine = args [1] ; String adaptorName = args [2] ; String recordType = args [3] ; String logFile = args [4] ; Major change API change: for backfilling I need to know when an adaptor has finished posting data to the queue. So instead of create yet another shutdown method I have deprecated the hardStop and the shutdown method to add a new one: public long shutdown(AdaptorShutdownPolicy shutdownPolicy) throws AdaptorException; AdaptorShutdownPolicy are: HARD_STOP, GRACEFULLY, WAIT_TILL_FINISHED Minor change for the socket controller : s.setReuseAddress(true); for the socket controller
        Hide
        jboulon Jerome Boulon added a comment -

        Only for 0.2.0

        Show
        jboulon Jerome Boulon added a comment - Only for 0.2.0
        Hide
        eyang Eric Yang added a comment -

        Could you provide a shell script warpper with usage print statement on how to use this? Thanks

        Show
        eyang Eric Yang added a comment - Could you provide a shell script warpper with usage print statement on how to use this? Thanks
        Hide
        eyang Eric Yang added a comment -

        Cancel patch for stuff that requires refinement.

        Show
        eyang Eric Yang added a comment - Cancel patch for stuff that requires refinement.
        Hide
        jboulon Jerome Boulon added a comment -

        Same code + wrapper script

        Show
        jboulon Jerome Boulon added a comment - Same code + wrapper script
        Hide
        eyang Eric Yang added a comment -

        +1 looks good.

        Show
        eyang Eric Yang added a comment - +1 looks good.
        Hide
        eyang Eric Yang added a comment -

        Changes to CHUKWA-268 prevent this from working. Please update the stale code.

        Show
        eyang Eric Yang added a comment - Changes to CHUKWA-268 prevent this from working. Please update the stale code.
        Hide
        jboulon Jerome Boulon added a comment -

        Ari, feel free to do the merge if you can get it done before Friday since I will not be available tomorrow.
        Thanks,
        Jerome.

        Show
        jboulon Jerome Boulon added a comment - Ari, feel free to do the merge if you can get it done before Friday since I will not be available tomorrow. Thanks, Jerome.
        Hide
        asrabkin Ari Rabkin added a comment -

        I got a bunch of null-pointer problems, I think unrelated to CHUKWA-268.

        Here's the merge I did. Passes unit tests on my end.

        Show
        asrabkin Ari Rabkin added a comment - I got a bunch of null-pointer problems, I think unrelated to CHUKWA-268 . Here's the merge I did. Passes unit tests on my end.
        Hide
        asrabkin Ari Rabkin added a comment -

        As per discussion with Eric, I just committed this. Both trunk and 0.2. Thanks Jerome, and sorry for the slightly rocky merge.

        Show
        asrabkin Ari Rabkin added a comment - As per discussion with Eric, I just committed this. Both trunk and 0.2. Thanks Jerome, and sorry for the slightly rocky merge.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #50 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/50/)
        . Backfilling tools. Contributed by Jerome Boulon.

        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #50 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/50/ ) . Backfilling tools. Contributed by Jerome Boulon.
        Hide
        asrabkin Ari Rabkin added a comment -

        Backfill script didn't get committed. (My fault.) Fixing that.

        Show
        asrabkin Ari Rabkin added a comment - Backfill script didn't get committed. (My fault.) Fixing that.
        Hide
        asrabkin Ari Rabkin added a comment -

        Just committed script. (Branch and trunk.) Apologies for the mistake.

        Show
        asrabkin Ari Rabkin added a comment - Just committed script. (Branch and trunk.) Apologies for the mistake.
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #61 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/61/ )
        Hide
        tanjiaqi Jiaqi Tan added a comment -

        Question about the Backfiller - does it require a Collector to be running somewhere, and then the Backfiller sends to that Collector?

        Show
        tanjiaqi Jiaqi Tan added a comment - Question about the Backfiller - does it require a Collector to be running somewhere, and then the Backfiller sends to that Collector?
        Hide
        jboulon Jerome Boulon added a comment -

        Backfiller is a standalone process per file and you don't need a collector. the backfiller output is a datasink file

        Show
        jboulon Jerome Boulon added a comment - Backfiller is a standalone process per file and you don't need a collector. the backfiller output is a datasink file

          People

          • Assignee:
            jboulon Jerome Boulon
            Reporter:
            jboulon Jerome Boulon
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development