OODT
  1. OODT
  2. OODT-404

Filemgr query not returning list of values for multi-value key

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.4
    • Fix Version/s: 0.4
    • Component/s: file manager
    • Labels:
      None

      Description

      Filemgr queries for a multi-value key returns only the first value and not a concatenated list of values (Tested on 0.4-SNAPSHOT).The lucene luke tool was used to confirm that the particular key has more than one value.

      The queries returned expected list of values on 0.3v.

        Activity

        Hide
        Paul Ramirez added a comment -

        Sheryl what interface were you using to look at the metadata. I just wrote a unit test to confirm that the value gets stored and is able to be retrieved so the bug seems to be at a higher level. Was this a command line client? or were you calling some API.

        Show
        Paul Ramirez added a comment - Sheryl what interface were you using to look at the metadata. I just wrote a unit test to confirm that the value gets stored and is able to be retrieved so the bug seems to be at a higher level. Was this a command line client? or were you calling some API.
        Hide
        Paul Ramirez added a comment -

        I read through your email again and my thought is that the command line clients have an issue. Those were recently refactored. I'll see if can get this fixed. Seems like something simple now that I know its not at the core.

        Show
        Paul Ramirez added a comment - I read through your email again and my thought is that the command line clients have an issue. Those were recently refactored. I'll see if can get this fixed. Seems like something simple now that I know its not at the core.
        Hide
        Sheryl John added a comment -

        Yes, I could see the values stored after pointing the Luke lucene index toolbox to the filemgr catalog. I guessed the issue would be in the command line side and I'm looking through the code to see where's the issue.

        Show
        Sheryl John added a comment - Yes, I could see the values stored after pointing the Luke lucene index toolbox to the filemgr catalog. I guessed the issue would be in the command line side and I'm looking through the code to see where's the issue.
        Hide
        Brian Foster added a comment -

        Hey Sheryl... Can you post the command line options you are using?... If query_tool 0.4 is returning only the first values but 0.3 query_tool is working correctly then the problem probably isn't on the command line end because that tool was left as is with this whole cas-cli plugin changes

        Show
        Brian Foster added a comment - Hey Sheryl... Can you post the command line options you are using?... If query_tool 0.4 is returning only the first values but 0.3 query_tool is working correctly then the problem probably isn't on the command line end because that tool was left as is with this whole cas-cli plugin changes
        Hide
        Brian Foster added a comment -

        i've attached a mod sqlQuery action unit-test to confirm that cas-cli filemgr sqlQuery action is getting back the multi-valued results... it comes back as a comma separated list of all the values... in the attached test Filename and Owners metadata is returned as: data.dat,Bob,Billy... this probably should be something more like: data.dat,[Bob,Billy]... but that is controlled by QueryResult class... but that is a separate issue, the results are being returned by the command line... i'll look into this more tomorrow (actually later today)... anyone have a 0.4 datasource filemgr up and running who can test if this problem exists for the datasource catalog as well?

        Show
        Brian Foster added a comment - i've attached a mod sqlQuery action unit-test to confirm that cas-cli filemgr sqlQuery action is getting back the multi-valued results... it comes back as a comma separated list of all the values... in the attached test Filename and Owners metadata is returned as: data.dat,Bob,Billy... this probably should be something more like: data.dat, [Bob,Billy] ... but that is controlled by QueryResult class... but that is a separate issue, the results are being returned by the command line... i'll look into this more tomorrow (actually later today)... anyone have a 0.4 datasource filemgr up and running who can test if this problem exists for the datasource catalog as well?
        Hide
        Paul Ramirez added a comment -

        Brian thats actually a bummer. The test I put in was only for the fmclient down that went to a lucene back end. From the email on list I believe Sheryl was using a Lucene backend but I could be wrong.

        Show
        Paul Ramirez added a comment - Brian thats actually a bummer. The test I put in was only for the fmclient down that went to a lucene back end. From the email on list I believe Sheryl was using a Lucene backend but I could be wrong.
        Hide
        Chris A. Mattmann added a comment -

        Hey Guys: if you want to test the DataSourceCatalog, you can use the HsqlDbFriendlyDataSourceCatalog inside of http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java. It should probably be refactored into its own class, but the methods in the ingest package and the system package tests will allow you to quickly populate a data source catalog and then test out the returns and so forth.

        Show
        Chris A. Mattmann added a comment - Hey Guys: if you want to test the DataSourceCatalog, you can use the HsqlDbFriendlyDataSourceCatalog inside of http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java . It should probably be refactored into its own class, but the methods in the ingest package and the system package tests will allow you to quickly populate a data source catalog and then test out the returns and so forth.
        Hide
        Sheryl John added a comment -

        Hi Brian,

        The command line options I used for query_tool:

        ./query_tool --url http://localhost:9000 --sql -query "SELECT CernerEpisodeStartTime FROM CernerLocations"
        Mar 12, 2012 7:15:20 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
        INFO: Query returned 1 results
        20xy-xx-yyT00:00:00.000-08:00

        Used the same option for 0.3 :
        ./query_tool --url http://localhost:9000 --sql -query "SELECT CernerEpisodeStartTime FROM CernerLocations"
        Mar 12, 2012 7:18:28 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
        INFO: Query returned 30 results

        The command line options I used for filemgr:
        ./filemgr-client --url http://localhost:9000 -op -sql -q 'SELECT CernerEpisodeStartTime FROM CernerLocations'
        Mar 12, 2012 7:16:34 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
        INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
        Mar 12, 2012 7:16:37 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
        INFO: Query returned 1 results
        20xy-xx-yyT00:00:00.000-08:00

        I was going through the complexQuery,filemgr and the cli yesterday and still couldn't find out where's the issue.
        Thanks!

        Show
        Sheryl John added a comment - Hi Brian, The command line options I used for query_tool: ./query_tool --url http://localhost:9000 --sql -query "SELECT CernerEpisodeStartTime FROM CernerLocations" Mar 12, 2012 7:15:20 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery INFO: Query returned 1 results 20xy-xx-yyT00:00:00.000-08:00 Used the same option for 0.3 : ./query_tool --url http://localhost:9000 --sql -query "SELECT CernerEpisodeStartTime FROM CernerLocations" Mar 12, 2012 7:18:28 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery INFO: Query returned 30 results – The command line options I used for filemgr: ./filemgr-client --url http://localhost:9000 -op -sql -q 'SELECT CernerEpisodeStartTime FROM CernerLocations' Mar 12, 2012 7:16:34 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init> INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties] Mar 12, 2012 7:16:37 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery INFO: Query returned 1 results 20xy-xx-yyT00:00:00.000-08:00 I was going through the complexQuery,filemgr and the cli yesterday and still couldn't find out where's the issue. Thanks!
        Hide
        Brian Foster added a comment -
        • attached mod to lucene catalog unit-test to prove that bug is in lucene's getReducedMetadata... getMetadata works fine, but something is wrong with getReducedMetadata
        Show
        Brian Foster added a comment - attached mod to lucene catalog unit-test to prove that bug is in lucene's getReducedMetadata... getMetadata works fine, but something is wrong with getReducedMetadata
        Hide
        Paul Ramirez added a comment -

        Thanks found the bug and apparently its been there since the getReducedMetadata function was added. I'll update it soon and commit those unit tests too.

        Show
        Paul Ramirez added a comment - Thanks found the bug and apparently its been there since the getReducedMetadata function was added. I'll update it soon and commit those unit tests too.
        Hide
        Chris A. Mattmann added a comment -

        +1, sounds good Paul and great work to find them Brian. Paul, can you assign the issue if you are going to resolve it? Also yes, please commit Brian's updated #TDD unit tests.

        Show
        Chris A. Mattmann added a comment - +1, sounds good Paul and great work to find them Brian. Paul, can you assign the issue if you are going to resolve it? Also yes, please commit Brian's updated #TDD unit tests.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4297/
        -----------------------------------------------------------

        Review request for Chris Mattmann and brian Foster.

        Summary
        -------

        The source of all evil was the getMetadata instead of the getAllMetadata in the LuceneCatalog.getReducedMetadata method

        This addresses bug OODT-404.
        https://issues.apache.org/jira/browse/OODT-404

        Diffs


        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1299812
        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1299812
        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java 1299812

        Diff: https://reviews.apache.org/r/4297/diff

        Testing
        -------

        Thanks,

        Paul

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4297/ ----------------------------------------------------------- Review request for Chris Mattmann and brian Foster. Summary ------- The source of all evil was the getMetadata instead of the getAllMetadata in the LuceneCatalog.getReducedMetadata method This addresses bug OODT-404 . https://issues.apache.org/jira/browse/OODT-404 Diffs http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1299812 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1299812 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java 1299812 Diff: https://reviews.apache.org/r/4297/diff Testing ------- Thanks, Paul
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4297/#review5852
        -----------------------------------------------------------

        Ship it!

        LGTM

        • brian

        On 2012-03-12 20:28:43, Paul Ramirez wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4297/

        -----------------------------------------------------------

        (Updated 2012-03-12 20:28:43)

        Review request for Chris Mattmann and brian Foster.

        Summary

        -------

        The source of all evil was the getMetadata instead of the getAllMetadata in the LuceneCatalog.getReducedMetadata method

        This addresses bug OODT-404.

        https://issues.apache.org/jira/browse/OODT-404

        Diffs

        -----

        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1299812

        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1299812

        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java 1299812

        Diff: https://reviews.apache.org/r/4297/diff

        Testing

        -------

        Thanks,

        Paul

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4297/#review5852 ----------------------------------------------------------- Ship it! LGTM brian On 2012-03-12 20:28:43, Paul Ramirez wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4297/ ----------------------------------------------------------- (Updated 2012-03-12 20:28:43) Review request for Chris Mattmann and brian Foster. Summary ------- The source of all evil was the getMetadata instead of the getAllMetadata in the LuceneCatalog.getReducedMetadata method This addresses bug OODT-404 . https://issues.apache.org/jira/browse/OODT-404 Diffs ----- http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1299812 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1299812 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java 1299812 Diff: https://reviews.apache.org/r/4297/diff Testing ------- Thanks, Paul
        Hide
        Sheryl John added a comment -

        Cool, Thanks!

        Show
        Sheryl John added a comment - Cool, Thanks!
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4297/#review5865
        -----------------------------------------------------------

        Ship it!

        LGTM

        • Chris

        On 2012-03-12 20:28:43, Paul Ramirez wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4297/

        -----------------------------------------------------------

        (Updated 2012-03-12 20:28:43)

        Review request for Chris Mattmann and brian Foster.

        Summary

        -------

        The source of all evil was the getMetadata instead of the getAllMetadata in the LuceneCatalog.getReducedMetadata method

        This addresses bug OODT-404.

        https://issues.apache.org/jira/browse/OODT-404

        Diffs

        -----

        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1299812

        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1299812

        http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java 1299812

        Diff: https://reviews.apache.org/r/4297/diff

        Testing

        -------

        Thanks,

        Paul

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4297/#review5865 ----------------------------------------------------------- Ship it! LGTM Chris On 2012-03-12 20:28:43, Paul Ramirez wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4297/ ----------------------------------------------------------- (Updated 2012-03-12 20:28:43) Review request for Chris Mattmann and brian Foster. Summary ------- The source of all evil was the getMetadata instead of the getAllMetadata in the LuceneCatalog.getReducedMetadata method This addresses bug OODT-404 . https://issues.apache.org/jira/browse/OODT-404 Diffs ----- http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1299812 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1299812 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java 1299812 Diff: https://reviews.apache.org/r/4297/diff Testing ------- Thanks, Paul
        Hide
        Paul Ramirez added a comment -

        Committed revision 1299971.

        Sheryl can you test this on the command line to confirm this resolves your issue.

        Show
        Paul Ramirez added a comment - Committed revision 1299971. Sheryl can you test this on the command line to confirm this resolves your issue.
        Hide
        Sheryl John added a comment -

        Hi Paul,

        Yes, I'm getting the list of expected values for the key when using the query-tool as shown below:
        ./query_tool --url http://localhost:9000 --sql -query "SELECT CernerEpisodeStartTime FROM CernerLocations"
        Mar 13, 2012 1:01:23 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
        INFO: Query returned 1 results
        2010-01-16,2010-11-29,2011-07-16

        However, for the filemgr-client command options, I'm getting an error similar to OODT-384.

        ./filemgr-client --url http://localhost:9000 -op -sql -q "SELECT CernerEpisodeStartTime FROM CernerLocations"
        ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
        ', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : Must specify url

        Show
        Sheryl John added a comment - Hi Paul, Yes, I'm getting the list of expected values for the key when using the query-tool as shown below: ./query_tool --url http://localhost:9000 --sql -query "SELECT CernerEpisodeStartTime FROM CernerLocations" Mar 13, 2012 1:01:23 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery INFO: Query returned 1 results 2010-01-16,2010-11-29,2011-07-16 However, for the filemgr-client command options, I'm getting an error similar to OODT-384 . ./filemgr-client --url http://localhost:9000 -op -sql -q "SELECT CernerEpisodeStartTime FROM CernerLocations" ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter ' ', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : Must specify url
        Hide
        Ricky Nguyen added a comment -

        Found the issue. We updated lib/cas-filemgr.jar, but did not update policy/cmd-line-options.xml, which had the incorrect "org.apache.oodt.cas.workflow.url" system property. Changed this property to "org.apache.oodt.cas.filemgr.url" and all is well.

        As a side note... Why does "org.apache.oodt.cas.filemgr.repositorymgr.dirs" or "org.apache.oodt.cas.filemgr.validation.dirs" dictate the location of the CLI configuration? Does that make sense?

        Show
        Ricky Nguyen added a comment - Found the issue. We updated lib/cas-filemgr.jar, but did not update policy/cmd-line-options.xml, which had the incorrect "org.apache.oodt.cas.workflow.url" system property. Changed this property to "org.apache.oodt.cas.filemgr.url" and all is well. As a side note... Why does "org.apache.oodt.cas.filemgr.repositorymgr.dirs" or "org.apache.oodt.cas.filemgr.validation.dirs" dictate the location of the CLI configuration? Does that make sense?
        Hide
        Sheryl John added a comment -

        Sorry the query works! Forgot to update the cmd-line-options.xml in filemgr's policy. It was reading the url as workflow's url and hence the above error.
        Thanks again.

        Show
        Sheryl John added a comment - Sorry the query works! Forgot to update the cmd-line-options.xml in filemgr's policy. It was reading the url as workflow's url and hence the above error. Thanks again.
        Hide
        Paul Ramirez added a comment -

        Committed revision 1299971.

        Show
        Paul Ramirez added a comment - Committed revision 1299971.
        Hide
        Chris A. Mattmann added a comment -

        As a side note... Why does "org.apache.oodt.cas.filemgr.repositorymgr.dirs" or "org.apache.oodt.cas.filemgr.validation.dirs" dictate the location of the CLI configuration? Does that make sense?

        Good question. I think it might have been Brian's take on reusing an existing policy/ directory deployment path. I think it would be an improvement and cleaner to have this separated out in its own namespace. Can you open a JIRA issue for 0.4 for this?

        Show
        Chris A. Mattmann added a comment - As a side note... Why does "org.apache.oodt.cas.filemgr.repositorymgr.dirs" or "org.apache.oodt.cas.filemgr.validation.dirs" dictate the location of the CLI configuration? Does that make sense? Good question. I think it might have been Brian's take on reusing an existing policy/ directory deployment path. I think it would be an improvement and cleaner to have this separated out in its own namespace. Can you open a JIRA issue for 0.4 for this?
        Hide
        Brian Foster added a comment -

        cas-cli doesn't use: "org.apache.oodt.cas.filemgr.repositorymgr.dirs" or "org.apache.oodt.cas.filemgr.validation.dirs"... the same properties are used across the board for all components:

        -Dorg.apache.oodt.cas.cli.action.spring.config=../policy/cmd-line-actions.xml \
        -Dorg.apache.oodt.cas.cli.option.spring.config=../policy/cmd-line-options.xml \

        Show
        Brian Foster added a comment - cas-cli doesn't use: "org.apache.oodt.cas.filemgr.repositorymgr.dirs" or "org.apache.oodt.cas.filemgr.validation.dirs"... the same properties are used across the board for all components: -Dorg.apache.oodt.cas.cli.action.spring.config=../policy/cmd-line-actions.xml \ -Dorg.apache.oodt.cas.cli.option.spring.config=../policy/cmd-line-options.xml \
        Hide
        Brian Foster added a comment -

        Also the org.apache.oodt.cas.workflow.url property in the filemgr has been changed to org.apache.oodt.cas.filemgr.url and checked it

        Show
        Brian Foster added a comment - Also the org.apache.oodt.cas.workflow.url property in the filemgr has been changed to org.apache.oodt.cas.filemgr.url and checked it
        Hide
        Ricky Nguyen added a comment -

        my mistake, sorry about that. I see it in the bin/filemgr-client script now. great work brian!

        Show
        Ricky Nguyen added a comment - my mistake, sorry about that. I see it in the bin/filemgr-client script now. great work brian!
        Hide
        Chris A. Mattmann added a comment -

        Yep my bad too - I didn't look and was just guessing at what the problem was. Typical though - bfost 10 steps ahead!

        Show
        Chris A. Mattmann added a comment - Yep my bad too - I didn't look and was just guessing at what the problem was. Typical though - bfost 10 steps ahead!

          People

          • Assignee:
            Paul Ramirez
            Reporter:
            Sheryl John
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development