Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.6
    • Fix Version/s: 0.8
    • Component/s: file manager
    • Labels:

      Description

      File Manager API Additions: (XmlRpcFileManager , XmlRpcFileManagerClient)

      • addParentForProductType - to support parent additions to data types in Wings Data Catalog
      • removeParentForProductType
        removeProductType
      • addElementsForProductType - to support property additions to data types in Wings Data Catalog
      • removeElementsForProductType

      Validation Layer Changes: (To support the above API additions)

      • addParentForProductType - invocation from API function above
      • removeParentForProductType - invocation from API function above

      Note: The attached patch only provides ValidationLayer implementations of the above functions for XmlValidationLayer, whereas only stub functions exist for DataSourceValidationLayer and ScienceDataValidationLayer.

      1. OODT-562.varunratnakar.021213.patch.txt
        17 kB
        Varun Ratnakar
      2. OODT-562.varunratnakar.101213.patch.txt
        12 kB
        Varun Ratnakar
      3. OODT-562.varunratnakar.101313.patch.txt
        14 kB
        Varun Ratnakar

        Activity

        Hide
        Chris A. Mattmann added a comment -

        Varun: I read through this patch. I actually feel that these functionality would be better placed in the CAS curator component.
        Then I think that we could build and extend CAS curator REST services to deal with this.

        This would have the advantage of:

        • Not changing the XML-RPC interface (most of the work we do now is in JAX-RS services – more people are interested in working on that, than XML-RPC, which we'd like to probably lock down for the moment since so many things are built on top of them)
        • Changing the XML-RPC interface would probably require a bunch of unit tests – which I'm tentative to write and don't exist in the provided patch.
        • The Curator is the logical place for policy editing/management to go and a REST interface that exposes some of this functionality would be better served there.

        I'll try and take a crack soon at modifying this patch to be housed in Curator. And/or you can do the same.

        Thanks!

        Show
        Chris A. Mattmann added a comment - Varun: I read through this patch. I actually feel that these functionality would be better placed in the CAS curator component. Then I think that we could build and extend CAS curator REST services to deal with this. This would have the advantage of: Not changing the XML-RPC interface (most of the work we do now is in JAX-RS services – more people are interested in working on that, than XML-RPC, which we'd like to probably lock down for the moment since so many things are built on top of them) Changing the XML-RPC interface would probably require a bunch of unit tests – which I'm tentative to write and don't exist in the provided patch. The Curator is the logical place for policy editing/management to go and a REST interface that exposes some of this functionality would be better served there. I'll try and take a crack soon at modifying this patch to be housed in Curator. And/or you can do the same. Thanks!
        Hide
        Varun Ratnakar added a comment -

        Hi Chris,

        please find my comments inline.

        Hmm.. Could you clarify this a bit. Doesn't the file manager deal with element validation ? And does the CAS curator also modify the same xml file to add element validation information ?

        Totally understandable !

        Sounds good. I was bypassing the curator really, but if you think this is where that functionality should go, then I will go with what you say

        I shall have to understand the curator code, and how element validation is done in it. I'll take a look at it as well.

        Regards,
        Varun

        Show
        Varun Ratnakar added a comment - Hi Chris, please find my comments inline. Hmm.. Could you clarify this a bit. Doesn't the file manager deal with element validation ? And does the CAS curator also modify the same xml file to add element validation information ? Totally understandable ! Sounds good. I was bypassing the curator really, but if you think this is where that functionality should go, then I will go with what you say I shall have to understand the curator code, and how element validation is done in it. I'll take a look at it as well. Regards, Varun
        Hide
        Chris A. Mattmann added a comment -

        Hey Varun – well the purpose of the curator is really to handle the policy management, metadata extraction, metadata/update, and automated ingest for a variety of Apache OODT services. Since the FM is really a read-only client of some of this information (well that's the intention), I'd rather reduce the dependencies on it to perform those types of policy updates. We've been moving periodically over the past few years to pushing these types of services in to the curator (and its APIs).

        I'll try and lead you in the right direction on this with a patch update, but won't get to it until later in the week.

        Show
        Chris A. Mattmann added a comment - Hey Varun – well the purpose of the curator is really to handle the policy management, metadata extraction, metadata/update, and automated ingest for a variety of Apache OODT services. Since the FM is really a read-only client of some of this information (well that's the intention), I'd rather reduce the dependencies on it to perform those types of policy updates. We've been moving periodically over the past few years to pushing these types of services in to the curator (and its APIs). I'll try and lead you in the right direction on this with a patch update, but won't get to it until later in the week.
        Hide
        Varun Ratnakar added a comment -

        Makes complete sense Chris.
        I shall try to look into this on my own too.

        Regards,
        Varun

        Show
        Varun Ratnakar added a comment - Makes complete sense Chris. I shall try to look into this on my own too. Regards, Varun
        Hide
        Varun Ratnakar added a comment -

        Chris,
        a small operational question about this approach. Would the file-manager have to be restarted if I use the curator to make changes ?

        I ask this, because initially I tried changing the xml directly, but the existing instance of file-manager service wouldn't update itself with the changed xml file. So, I went via the route of adding functionality in the file manager itself via RPC calls to the server, so that that internal structures of the existing instance of file-manager are updated (as well as the xml file). This way I don't need to restart the file-manager for every update (And there could be lots of updates).

        Regards,
        Varun

        Show
        Varun Ratnakar added a comment - Chris, a small operational question about this approach. Would the file-manager have to be restarted if I use the curator to make changes ? I ask this, because initially I tried changing the xml directly, but the existing instance of file-manager service wouldn't update itself with the changed xml file. So, I went via the route of adding functionality in the file manager itself via RPC calls to the server, so that that internal structures of the existing instance of file-manager are updated (as well as the xml file). This way I don't need to restart the file-manager for every update (And there could be lots of updates). Regards, Varun
        Hide
        Chris A. Mattmann added a comment -

        Hey Varun,

        Scope out:

        http://oodt.apache.org/components/maven/xref/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.html#145

        refreshConfigAndPolicy

        This method exists so Curator and other apps can leverage it to update the policy and config.

        Cheers,
        Chris

        Show
        Chris A. Mattmann added a comment - Hey Varun, Scope out: http://oodt.apache.org/components/maven/xref/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.html#145 refreshConfigAndPolicy This method exists so Curator and other apps can leverage it to update the policy and config. Cheers, Chris
        Hide
        Varun Ratnakar added a comment -

        Very cool

        Show
        Varun Ratnakar added a comment - Very cool
        Hide
        Varun Ratnakar added a comment -

        Hi again Chris,

        finally got some time to start working on this, and I would appreciate some advice from you on this.

        1) For the FileManager: I'm thinking of updating the xml files directly from php rather than go through the curator, and then calling the "refreshConfigAndPolicy" function. Do you see any issues with that ?

        2) Is there a similar function for WorkflowManager ? I couldn't find any. And, I would still need to add the "executeWorkflow" function in WorkflowManager as it is required to execute dynamically created workflow objects composed of wings tasks.

        Regards,
        Varun

        Show
        Varun Ratnakar added a comment - Hi again Chris, finally got some time to start working on this, and I would appreciate some advice from you on this. 1) For the FileManager: I'm thinking of updating the xml files directly from php rather than go through the curator, and then calling the "refreshConfigAndPolicy" function. Do you see any issues with that ? 2) Is there a similar function for WorkflowManager ? I couldn't find any. And, I would still need to add the "executeWorkflow" function in WorkflowManager as it is required to execute dynamically created workflow objects composed of wings tasks. Regards, Varun
        Hide
        Chris A. Mattmann added a comment -

        Hey Varun sorry it took me so long to get back on this, comments below:

        1) For the FileManager: I'm thinking of updating the xml files directly from php rather than go through the curator, and then calling the "refreshConfigAndPolicy" function. Do you see any issues with that ?

        I would put it in the curator. If we put it into PHP then we have introduced a PHP dep that didn't exist anymore. If we put it in the Curator, PHP could then call it (which is great) but other clients would also be able to call it (written in other languages).

        2) Is there a similar function for WorkflowManager ? I couldn't find any. And, I would still need to add the "executeWorkflow" function in WorkflowManager as it is required to execute dynamically created workflow objects composed of wings tasks.

        See OODT-213. It exists internally at JPL in the old OPSUI, I'll work to at least throw up a patch on OODT-213 so we can work from there. Then we need a JAX-RS API that lives in PCS, maybe pcs/pcs-services.

        Show
        Chris A. Mattmann added a comment - Hey Varun sorry it took me so long to get back on this, comments below: 1) For the FileManager: I'm thinking of updating the xml files directly from php rather than go through the curator, and then calling the "refreshConfigAndPolicy" function. Do you see any issues with that ? I would put it in the curator. If we put it into PHP then we have introduced a PHP dep that didn't exist anymore. If we put it in the Curator, PHP could then call it (which is great) but other clients would also be able to call it (written in other languages). 2) Is there a similar function for WorkflowManager ? I couldn't find any. And, I would still need to add the "executeWorkflow" function in WorkflowManager as it is required to execute dynamically created workflow objects composed of wings tasks. See OODT-213 . It exists internally at JPL in the old OPSUI, I'll work to at least throw up a patch on OODT-213 so we can work from there. Then we need a JAX-RS API that lives in PCS, maybe pcs/pcs-services.
        Hide
        Varun Ratnakar added a comment -

        Thanks Chris ! I might have more questions for you when I restart working on this.

        Regards,
        Varun

        Show
        Varun Ratnakar added a comment - Thanks Chris ! I might have more questions for you when I restart working on this. Regards, Varun
        Hide
        Chris A. Mattmann added a comment -
        • push to 0.7
        Show
        Chris A. Mattmann added a comment - push to 0.7
        Hide
        Varun Ratnakar added a comment - - edited

        @Chris and others: I've been working on putting most of this functionality into the curator (CurationPolicyManager). So far I've been testing it by calling the functions directly, but this assumes that the file-manager policy files are in the same server as the Wings portal, and hence are directly editable. Since, we might want to have Wings and OODT on different servers, I had a couple of questions:

        • Is there a remotely-accessible Curation API that I could add this functionality to ?
        • If not, would it be improper to create a Curation API Servlet ?

        Regards,
        Varun

        Show
        Varun Ratnakar added a comment - - edited @Chris and others: I've been working on putting most of this functionality into the curator (CurationPolicyManager). So far I've been testing it by calling the functions directly, but this assumes that the file-manager policy files are in the same server as the Wings portal, and hence are directly editable. Since, we might want to have Wings and OODT on different servers, I had a couple of questions: Is there a remotely-accessible Curation API that I could add this functionality to ? If not, would it be improper to create a Curation API Servlet ? Regards, Varun
        Hide
        Varun Ratnakar added a comment - - edited

        The latest patch moves the implementation from File Manager into the Curation Service. It however relies on some changes in the file manager

        Show
        Varun Ratnakar added a comment - - edited The latest patch moves the implementation from File Manager into the Curation Service. It however relies on some changes in the file manager
        Hide
        Chris A. Mattmann added a comment -

        Hey Varun, thanks, a few comments:

        1. In the patch anywhere in the curation service that adds something should IMO be using PUT or POST; and anything that removes something should be using DELETE. The GET methods look OK.

        2. Public new methods at the bottom of the patch should have javadoc, or should be considered helper package protected functions or private ones.

        Feel free to use http://review.apache.org/ (Review Board) for the next patch review for inline commenting. Thanks!

        Show
        Chris A. Mattmann added a comment - Hey Varun, thanks, a few comments: 1. In the patch anywhere in the curation service that adds something should IMO be using PUT or POST; and anything that removes something should be using DELETE. The GET methods look OK. 2. Public new methods at the bottom of the patch should have javadoc, or should be considered helper package protected functions or private ones. Feel free to use http://review.apache.org/ (Review Board) for the next patch review for inline commenting. Thanks!
        Hide
        Varun Ratnakar added a comment -

        New patch added. Please review at
        https://reviews.apache.org/r/14626/

        Show
        Varun Ratnakar added a comment - New patch added. Please review at https://reviews.apache.org/r/14626/

          People

          • Assignee:
            Unassigned
            Reporter:
            Varun Ratnakar
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development