Uploaded image for project: 'ManifoldCF'
  1. ManifoldCF
  2. CONNECTORS-1636

ElasticSearch Connector not working with ingest pipeline processor attachment

    XMLWordPrintableJSON

Details

    Description

      While using Apache manifoldcf elasticsearch connector for elasticsearch version 6.6.x, I found that connector is not working as expected for pipeline processor "attachment". 

      The processor requires Base64 String to process input stream to content. 

      It is working for "mapper-attachment" plugin but that plugin is deprecated in newer versions of elasticsearch.

      In case elasticsearch pipeline is used and mapper-attachment is set to false. then the content is processed as byte Array to index document, which is not correct type for indexing to elasticsearch.

       

      if (!useMapperAttachments && inputStream != null) {
          if (contentAttributeName != null) {
              Reader r = new InputStreamReader(inputStream, Consts.UTF_8);
              if (needComma) {
                  pw.print(",");
              }
              pw.append(jsonStringEscape(contentAttributeName)).append(" : \"");
              char[] buffer = new char[65536];
              while (true) {
                  int amt = r.read(buffer, 0, buffer.length);
                  if (amt == -1)
                      break;
                  for (int j = 0; j < amt; j++) {
                      final char x = buffer[j];
                      if (x == '\n')
                          pw.append('\\').append('n');
                      else if (x == '\r')
                          pw.append('\\').append('r');
                      else if (x == '\t')
                          pw.append('\\').append('t');
                      else if (x == '\b')
                          pw.append('\\').append('b');
                      else if (x == '\f')
                          pw.append('\\').append('f');
                      else if (x < 32) {
                          pw.append("\\u").append(String.format(Locale.ROOT, "%04x", (int) x));
                      } else {
                          if (x == '\"' || x == '\\' || x == '/')
                              pw.append('\\');
                          pw.append(x);
                      }
                  }
              }
              pw.append("\"");
              needComma = true;
          }
      }
      

       

       

      Attachments

        Activity

          People

            kwright@metacarta.com Karl Wright
            RohitBatta Rohit Batta
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: