Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • 1.3, nutchgora
    • None
    • indexer
    • None
    • Patch Available

    Description

      Provide the facility to pass static data defined in solrindex-mapping.xml to solr during the mapping process.

      Attachments

        1. nutch_1.3_static_field.patch
          4 kB
          David Stuart
        2. nutch_1.3_static_field.patch
          4 kB
          David Stuart
        3. nutch_1.3_static_field2.patch
          5 kB
          David Stuart

        Issue Links

          Activity

            dstuart David Stuart added a comment -

            Extends SolrWriter and SolrMappingReader to take static field data defined in solrindex-mapping.xml

            e.g.

            <staticField dest="entity">node</staticField>

            dstuart David Stuart added a comment - Extends SolrWriter and SolrMappingReader to take static field data defined in solrindex-mapping.xml e.g. <staticField dest="entity">node</staticField>

            The functionality is useful, +1. But the patch has formatting errors. Please fix them before committing.

            The same functionality should be added to trunk, too.

            ab Andrzej Bialecki added a comment - The functionality is useful, +1. But the patch has formatting errors. Please fix them before committing. The same functionality should be added to trunk, too.
            markus17 Markus Jelsma added a comment -

            Yes, i'll look into it next week orso. The pro for this patch is that in some cases it can replace the subcollection plugin to set a static value.

            markus17 Markus Jelsma added a comment - Yes, i'll look into it next week orso. The pro for this patch is that in some cases it can replace the subcollection plugin to set a static value.
            dstuart David Stuart added a comment -

            updated with formattng, think I need eclipse code styles

            dstuart David Stuart added a comment - updated with formattng, think I need eclipse code styles
            markus17 Markus Jelsma added a comment -

            Great! The patch almost works as i expected. It:
            1. adds a single static value to a specified field
            2. adds another value to a field already mapped by Nutch (requires multi valued field)
            3. adds the last defined staticField value (doesn't require a multi valued field).

            I think 1 and 2 are desired, this way we can append values to fields. But i think 3 isn't a desirable feature. If we can use this to append a value for a multi valued field, then shouldn't it also work if we define multiple staticField elements with the same dest attribute?

            markus17 Markus Jelsma added a comment - Great! The patch almost works as i expected. It: 1. adds a single static value to a specified field 2. adds another value to a field already mapped by Nutch (requires multi valued field) 3. adds the last defined staticField value (doesn't require a multi valued field). I think 1 and 2 are desired, this way we can append values to fields. But i think 3 isn't a desirable feature. If we can use this to append a value for a multi valued field, then shouldn't it also work if we define multiple staticField elements with the same dest attribute?
            dstuart David Stuart added a comment -

            I have reworked the patch slightly to accomplish the following

            1. adds a single static value to a specified field
            2. adds another value to a field already mapped by Nutch (requires multi valued field)
            3. The ability to define multiple staticField's with the same dest attribute will create multiple entires (requires multi valued field)

            dstuart David Stuart added a comment - I have reworked the patch slightly to accomplish the following 1. adds a single static value to a specified field 2. adds another value to a field already mapped by Nutch (requires multi valued field) 3. The ability to define multiple staticField's with the same dest attribute will create multiple entires (requires multi valued field)
            dstuart David Stuart added a comment -

            Hey Markus,

            Is there anything else I need to do on this patch? Do you want me to roll a patch against trunk as per Andrzej?

            Regards,

            Dave

            dstuart David Stuart added a comment - Hey Markus, Is there anything else I need to do on this patch? Do you want me to roll a patch against trunk as per Andrzej? Regards, Dave
            markus17 Markus Jelsma added a comment -

            Yes, it needs to be added to trunk too. Please submit a patch if available.

            Thanks

            markus17 Markus Jelsma added a comment - Yes, it needs to be added to trunk too. Please submit a patch if available. Thanks
            jnioche Julien Nioche added a comment -

            Won't be fixed in 1.3. Marked as affecting 2.0 as well

            jnioche Julien Nioche added a comment - Won't be fixed in 1.3. Marked as affecting 2.0 as well
            dstuart David Stuart added a comment -

            Hi Julien,

            There is a patch outstanding on this for trunk but as far as I'm aware the 1.3 patch works as per Markus review. Was there anything outstanding as I think its good to go for 1.3??

            Regards,

            Dave

            dstuart David Stuart added a comment - Hi Julien, There is a patch outstanding on this for trunk but as far as I'm aware the 1.3 patch works as per Markus review. Was there anything outstanding as I think its good to go for 1.3?? Regards, Dave
            jnioche Julien Nioche added a comment -

            Hi David,

            I'd rather commit on both branches at the same time instead of having one functionality available in one version and not in the other. the plan is to release 1.3 quite soon hence my suggestion to commit your patch soon after that.

            Does it make sense?

            Julien

            jnioche Julien Nioche added a comment - Hi David, I'd rather commit on both branches at the same time instead of having one functionality available in one version and not in the other. the plan is to release 1.3 quite soon hence my suggestion to commit your patch soon after that. Does it make sense? Julien
            dstuart David Stuart added a comment -

            Yep totally.

            I start working on the 2.0 patch

            Cheers

            dstuart David Stuart added a comment - Yep totally. I start working on the 2.0 patch Cheers
            jnioche Julien Nioche added a comment -

            Guys,

            Wouldn't it be better to implement this functionality in a non-SOLR specific way as done in NUTCH-940?
            We only support SOLR for now but could possibly have more indexing backends in the future, like for instance ElasticSearch.

            Any thoughts on this?

            jnioche Julien Nioche added a comment - Guys, Wouldn't it be better to implement this functionality in a non-SOLR specific way as done in NUTCH-940 ? We only support SOLR for now but could possibly have more indexing backends in the future, like for instance ElasticSearch. Any thoughts on this?
            dstuart David Stuart added a comment -

            Hi Julien,

            Agreed, would it also be better to provide the facility that the solr field mapping functionality provides to an indexer as well? I'll test out the 940 patch but would really like it to support

            1. adds a single static value to a specified field
            2. adds another value to a field already mapped by Nutch (requires multi valued field)
            3. The ability to define multiple staticField's with the same dest attribute will create multiple entires (requires multi valued field)

            dstuart David Stuart added a comment - Hi Julien, Agreed, would it also be better to provide the facility that the solr field mapping functionality provides to an indexer as well? I'll test out the 940 patch but would really like it to support 1. adds a single static value to a specified field 2. adds another value to a field already mapped by Nutch (requires multi valued field) 3. The ability to define multiple staticField's with the same dest attribute will create multiple entires (requires multi valued field)
            markus17 Markus Jelsma added a comment -

            I am also a proponent of NUTCH-940 for the reasons described by Julien. With that you can still add static values to already existing field to create a multi valued field. I don't know about case three David, you can simlpy copyField it around in Nutch mapping or Solr.

            markus17 Markus Jelsma added a comment - I am also a proponent of NUTCH-940 for the reasons described by Julien. With that you can still add static values to already existing field to create a multi valued field. I don't know about case three David, you can simlpy copyField it around in Nutch mapping or Solr.
            dstuart David Stuart added a comment -

            Yep fine with me. Although it would be quite messy (but achievable), if you hav 10 static fields (categories say) you would have to give them individual names and copy them to a master in solr.

            dstuart David Stuart added a comment - Yep fine with me. Although it would be quite messy (but achievable), if you hav 10 static fields (categories say) you would have to give them individual names and copy them to a master in solr.
            markus17 Markus Jelsma added a comment -

            NUTCH-940 has been committed for 1.4.

            markus17 Markus Jelsma added a comment - NUTCH-940 has been committed for 1.4.

            People

              markus17 Markus Jelsma
              dstuart David Stuart
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: