Flume
  1. Flume
  2. FLUME-2049

Compile ElasticSearchSink with elasticsearch 0.90

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.4.0, v1.3.1
    • Fix Version/s: v1.4.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Release Note:
      Flume now compiles with elasticsearch 0.90.1. Running Flume with any other major version of elasticsearch will likely require that Flume is compiled with that version.

      Description

      The ElasticSearchSink currently compiles against es 0.19. Using 0.20 requires a recompile. I haven't tried 0.90 yet.

      0.90 has been out for 2-ish weeks so we should change to compiling with it prior to 1.4 release.

      I'm hoping to get priority for doing this in the next two weeks or so - however I have no issue with somebody else doing it.

      1. FLUME-2049.patch
        0.9 kB
        Edward Sargisson

        Issue Links

          Activity

          Hide
          Mike Percy added a comment -

          It took a bit of finagling... between FLUME-1741, a Jenkins build hack to delete the ES data dir on any existing Jenkins slaves, and adding the -Xrs flag to the MAVEN_OPTS everything lit up green!

          Show
          Mike Percy added a comment - It took a bit of finagling... between FLUME-1741 , a Jenkins build hack to delete the ES data dir on any existing Jenkins slaves, and adding the -Xrs flag to the MAVEN_OPTS everything lit up green!
          Hide
          Edward Sargisson added a comment -

          To follow up, you said,
          "Looks like the flume-ng-elasticsearch-sink/data directory is stuck in the
          Jenkins workspace and not being deleted between builds (even though we no
          longer use that path). Adding "git clean -f -d" as a pre-build step to get
          rid of it and re-running the build.

          Mike"

          As of build 457 I can see that the ElasticSearchSink build is successful (along with everything else!)
          https://builds.apache.org/job/flume-trunk/457/

          Show
          Edward Sargisson added a comment - To follow up, you said, "Looks like the flume-ng-elasticsearch-sink/data directory is stuck in the Jenkins workspace and not being deleted between builds (even though we no longer use that path). Adding "git clean -f -d" as a pre-build step to get rid of it and re-running the build. Mike" As of build 457 I can see that the ElasticSearchSink build is successful (along with everything else!) https://builds.apache.org/job/flume-trunk/457/
          Hide
          Mike Percy added a comment -

          Not your fault, I missed it too apparently! I will take a closer look this weekend to see if I can figure it out. If not, Monday should be alright.

          Show
          Mike Percy added a comment - Not your fault, I missed it too apparently! I will take a closer look this weekend to see if I can figure it out. If not, Monday should be alright.
          Hide
          Edward Sargisson added a comment -

          Revert it and I'll see if I can figure it out on Monday.

          My apologies for a patch that doesn't stick.

          Show
          Edward Sargisson added a comment - Revert it and I'll see if I can figure it out on Monday. My apologies for a patch that doesn't stick.
          Hide
          Mike Percy added a comment -

          Jeff put a patch up a while ago @ FLUME-1741 to change the data directory to target

          Show
          Mike Percy added a comment - Jeff put a patch up a while ago @ FLUME-1741 to change the data directory to target
          Hide
          Edward Sargisson added a comment -

          Rat plugin triggering on /data directory? I'm guessing but the tests passed.

          I won't have access to source until Monday am but we could exclude /target/data (or whereever it is) from Rat.

          I don't believe that the normal technique to stop data being written (setting client only) will work in the tests because they work against a server.

          Show
          Edward Sargisson added a comment - Rat plugin triggering on /data directory? I'm guessing but the tests passed. I won't have access to source until Monday am but we could exclude /target/data (or whereever it is) from Rat. I don't believe that the normal technique to stop data being written (setting client only) will work in the tests because they work against a server.
          Hide
          Hari Shreedharan added a comment -

          Any idea why this is failing on Jenkins?
          [INFO] Flume NG HBase Sink ............................... SUCCESS [1:44.084s]
          [INFO] Flume NG ElasticSearch Sink ....................... FAILURE [13.424s]
          [INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [41.772s]
          [INFO] Flume Sources ..................................... SUCCESS [0.183s]
          [INFO] Flume Scribe Source ............................... SUCCESS [1.328s]

          It builds fine locally for me

          Show
          Hari Shreedharan added a comment - Any idea why this is failing on Jenkins? [INFO] Flume NG HBase Sink ............................... SUCCESS [1:44.084s] [INFO] Flume NG ElasticSearch Sink ....................... FAILURE [13.424s] [INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [41.772s] [INFO] Flume Sources ..................................... SUCCESS [0.183s] [INFO] Flume Scribe Source ............................... SUCCESS [1.328s] It builds fine locally for me
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #437 (See https://builds.apache.org/job/flume-trunk/437/)
          FLUME-2049. Compile ElasticSearchSink with elasticsearch 0.90. (Revision a06a56085bd810929f27764925eb1b3f6006fb01)

          Result = FAILURE
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=a06a56085bd810929f27764925eb1b3f6006fb01
          Files :

          • pom.xml
          Show
          Hudson added a comment - Integrated in flume-trunk #437 (See https://builds.apache.org/job/flume-trunk/437/ ) FLUME-2049 . Compile ElasticSearchSink with elasticsearch 0.90. (Revision a06a56085bd810929f27764925eb1b3f6006fb01) Result = FAILURE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=a06a56085bd810929f27764925eb1b3f6006fb01 Files : pom.xml
          Mike Percy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Mike Percy added a comment -

          Pushed to trunk and flume-1.4 branches. Thanks for the patch Edward!

          Show
          Mike Percy added a comment - Pushed to trunk and flume-1.4 branches. Thanks for the patch Edward!
          Hide
          Mike Percy added a comment -

          +1. Edward I'm happy to defer to you on this one, everything looks OK to me. It's such a small change that people can recompile Flume 1.4 if they want something running against the old version.

          Show
          Mike Percy added a comment - +1. Edward I'm happy to defer to you on this one, everything looks OK to me. It's such a small change that people can recompile Flume 1.4 if they want something running against the old version.
          Mike Percy made changes -
          Assignee Edward Sargisson [ ejsarge ]
          Hide
          Edward Sargisson added a comment -

          No - sadly.

          When elasticsearch increments major versions the serialization format has probably changed. That's one of the reasons they increment major versions.
          Our documentation was recently updated to say this - the Flume tarball doesn't contain an elasticsearch jar in order to force users to get the jar they need.

          My expectation is that elasticsearch 0.20.x could be dropped in in place of elasticsearch 0.90.x.
          However, I know that the way we compile with 0.19.x means that using 0.20.x requires a recompile. 0.19.x is old enough that we should remove that need for the recompile by compiling with 0.90.x.

          Show
          Edward Sargisson added a comment - No - sadly. When elasticsearch increments major versions the serialization format has probably changed. That's one of the reasons they increment major versions. Our documentation was recently updated to say this - the Flume tarball doesn't contain an elasticsearch jar in order to force users to get the jar they need. My expectation is that elasticsearch 0.20.x could be dropped in in place of elasticsearch 0.90.x. However, I know that the way we compile with 0.19.x means that using 0.20.x requires a recompile. 0.19.x is old enough that we should remove that need for the recompile by compiling with 0.90.x.
          Hide
          Hari Shreedharan added a comment -

          Hi Edward,

          Will this continue to work against old elastic search clusters?

          Show
          Hari Shreedharan added a comment - Hi Edward, Will this continue to work against old elastic search clusters?
          Edward Sargisson made changes -
          Assignee Edward Sargisson [ ejsarge ]
          Edward Sargisson made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Flume now compiles with elasticsearch 0.90.1. Running Flume with any other major version of elasticsearch will likely require that Flume is compiled with that version.
          Fix Version/s v1.4.0 [ 12323372 ]
          Edward Sargisson made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 12317 ]
          Edward Sargisson made changes -
          Field Original Value New Value
          Attachment FLUME-2049.patch [ 12585637 ]
          Hide
          Edward Sargisson added a comment -

          Patch for 1.4 attached.

          Show
          Edward Sargisson added a comment - Patch for 1.4 attached.
          Edward Sargisson created issue -

            People

            • Assignee:
              Edward Sargisson
              Reporter:
              Edward Sargisson
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development