Solr
  1. Solr
  2. SOLR-1032

CSV loader to support literal field values

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 4.0-ALPHA
    • Component/s: update
    • Labels:
      None

      Description

      It would be very handy if the CSV loader could handle a literal field mapping, like the extracting request handler does. For example, in a scenario where you have multiple datasources (some data from a DB, some from file crawls, and some from CSV) it is nice to add a field to every document that specifies the data source. This is easily done with DIH with a template transformer, and Solr Cell with ext.literal.datasource=, but impossible currently with CSV.

      1. SOLR-1032.patch
        4 kB
        Erik Hatcher
      2. SOLR-1032.patch
        3 kB
        Simon Rosenthal

        Activity

        Hide
        Yonik Seeley added a comment -

        Seems reasonable...
        csv.literal.datasource=DB?

        Show
        Yonik Seeley added a comment - Seems reasonable... csv.literal.datasource=DB?
        Hide
        Erik Hatcher added a comment -

        Yeah, that parameter name convention is fine with me. Unfortunately the other parameters aren't prefixed by csv. though.

        Show
        Erik Hatcher added a comment - Yeah, that parameter name convention is fine with me. Unfortunately the other parameters aren't prefixed by csv. though.
        Hide
        Yonik Seeley added a comment -

        Perhaps just literal.datasource=DB then? the context will always be clear since this isn't a component being mixed in with others.

        Show
        Yonik Seeley added a comment - Perhaps just literal.datasource=DB then? the context will always be clear since this isn't a component being mixed in with others.
        Hide
        Erik Hatcher added a comment -

        +1

        Show
        Erik Hatcher added a comment - +1
        Hide
        Simon Rosenthal added a comment -

        here's a first cut at a patch. The syntax for the literal field is ....&f.<fieldname>.literal=<literalvalue>

        The supplied literal value is not processed in any way (e.g. split into multi values, quotes removed)

        No tests yet.

        Show
        Simon Rosenthal added a comment - here's a first cut at a patch. The syntax for the literal field is ....&f.<fieldname>.literal=<literalvalue> The supplied literal value is not processed in any way (e.g. split into multi values, quotes removed) No tests yet.
        Hide
        Simon Rosenthal added a comment -

        Patch is for 4.0

        Show
        Simon Rosenthal added a comment - Patch is for 4.0
        Hide
        Erik Hatcher added a comment -

        Attached is a patch adding a test case and switching to use the Solr Cell established convention of literal.field_name=value parameter naming.

        Show
        Erik Hatcher added a comment - Attached is a patch adding a test case and switching to use the Solr Cell established convention of literal.field_name=value parameter naming.
        Hide
        Erik Hatcher added a comment -

        Simon - thanks for the effort on this! I have taken a look and updated the patch with a test case and a change to use literal.field_name=value convention. I think for the sake of this feature, it's best to stick with the established Solr Cell convention. Perhaps in another issue we can take up refactoring parameter naming for this capability.

        Thoughts? Objections? I'll commit this to trunk once I hear Simon's signoff.

        Show
        Erik Hatcher added a comment - Simon - thanks for the effort on this! I have taken a look and updated the patch with a test case and a change to use literal.field_name=value convention. I think for the sake of this feature, it's best to stick with the established Solr Cell convention. Perhaps in another issue we can take up refactoring parameter naming for this capability. Thoughts? Objections? I'll commit this to trunk once I hear Simon's signoff.
        Hide
        Jan Høydahl added a comment -

        Nice.

        Show
        Jan Høydahl added a comment - Nice.
        Hide
        Simon Rosenthal added a comment -

        revised patch looks good - do commit.

        Show
        Simon Rosenthal added a comment - revised patch looks good - do commit.

          People

          • Assignee:
            Erik Hatcher
            Reporter:
            Erik Hatcher
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development