[CONNECTORS-1636] ElasticSearch Connector not working with ingest pipeline processor attachment

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: ManifoldCF 2.15
• Fix Version/s: ManifoldCF next
• Component/s: Elastic Search connector
• Labels: ManifoldCF connector elasticsearch manifoldcf
• Assignee: Karl Wright
• Reporter: Rohit Batta
• Created: 14/Feb/20 09:25
• Updated: 14/Feb/20 20:19

Description

While using Apache manifoldcf elasticsearch connector for elasticsearch version 6.6.x, I found that connector is not working as expected for pipeline processor "attachment". 

The processor requires Base64 String to process input stream to content. 

It is working for "mapper-attachment" plugin but that plugin is deprecated in newer versions of elasticsearch.

In case elasticsearch pipeline is used and mapper-attachment is set to false. then the content is processed as byte Array to index document, which is not correct type for indexing to elasticsearch.

 

if (!useMapperAttachments && inputStream != null) {
    if (contentAttributeName != null) {
        Reader r = new InputStreamReader(inputStream, Consts.UTF_8);
        if (needComma) {
            pw.print(",");
        }
        pw.append(jsonStringEscape(contentAttributeName)).append(" : \"");
        char[] buffer = new char[65536];
        while (true) {
            int amt = r.read(buffer, 0, buffer.length);
            if (amt == -1)
                break;
            for (int j = 0; j < amt; j++) {
                final char x = buffer[j];
                if (x == '\n')
                    pw.append('\\').append('n');
                else if (x == '\r')
                    pw.append('\\').append('r');
                else if (x == '\t')
                    pw.append('\\').append('t');
                else if (x == '\b')
                    pw.append('\\').append('b');
                else if (x == '\f')
                    pw.append('\\').append('f');
                else if (x < 32) {
                    pw.append("\\u").append(String.format(Locale.ROOT, "%04x", (int) x));
                } else {
                    if (x == '\"' || x == '\\' || x == '/')
                        pw.append('\\');
                    pw.append(x);
                }
            }
        }
        pw.append("\"");
        needComma = true;
    }
}