Uploaded image for project: 'OFBiz'
  1. OFBiz
  2. OFBIZ-9498

Improve DevOps using environment variable configuration

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • Trunk, Upcoming Branch
    • None
    • framework
    • None

    Description

      Discussed in thread : https://s.apache.org/Mh3q

      This Jira will present the improvment proposal giving a way to configure OFBiz using environment variable.

      Attachments

        1. OFBIZ-9498.patch
          11 kB
          Gil Portenseigne

        Issue Links

          Activity

            With the patch applied, could be set :
            export OFB_POSTGRES_DB="jdbc:postgresql://127.0.0.1/mydb"
            export OFB_POSTGRES_USER="ofbiz"
            export OFB_POSTGRES_PASS="secret"

            export OFB_MEM_XMS="-Xms256M"
            export OFB_MEM_XMX="-Xmx2048M"

            And any other ENV from a customized OFBiz property file like ${env:ENV:defaultValue}
            export OFB_INSTANCE_ID="ofbiz-instance-prod-01"

            pgil Gil Portenseigne added a comment - With the patch applied, could be set : export OFB_POSTGRES_DB="jdbc:postgresql://127.0.0.1/mydb" export OFB_POSTGRES_USER="ofbiz" export OFB_POSTGRES_PASS="secret" export OFB_MEM_XMS="-Xms256M" export OFB_MEM_XMX="-Xmx2048M" And any other ENV from a customized OFBiz property file like ${env:ENV:defaultValue} export OFB_INSTANCE_ID="ofbiz-instance-prod-01"

            Hi Gil,

            I think environment variables are also a good solutions but to sensitive data as I explained in http://markmail.org/message/uqzjxsx5vs2o37cw

            Also I suggested at that we could have several ways suggested for DevOps in OFBiz http://markmail.org/message/gcp3tstywehm67dd

            jleroux Jacques Le Roux added a comment - Hi Gil, I think environment variables are also a good solutions but to sensitive data as I explained in http://markmail.org/message/uqzjxsx5vs2o37cw Also I suggested at that we could have several ways suggested for DevOps in OFBiz http://markmail.org/message/gcp3tstywehm67dd
            jleroux Jacques Le Roux added a comment - Hi Gil, mbrohl , What's up with that? Could we not fill something like we had at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65865831#AddressingCustomRequirementsInOFBiz-PatchManagementUsingAntTargets ?
            pgil Gil Portenseigne added a comment - - edited

            Hi Jacques, we are still using it in our project but we haven't progress on the matter... I planned to analyse template system with gradle as discussed with Taher and Michael, but that's low my prio list .

            pgil Gil Portenseigne added a comment - - edited Hi Jacques, we are still using it in our project but we haven't progress on the matter... I planned to analyse template system with gradle as discussed with Taher and Michael, but that's low my prio list .
            jleroux Jacques Le Roux added a comment - - edited

            Hi Gil,

            Thanks for the update, we all have priorities

            jleroux Jacques Le Roux added a comment - - edited Hi Gil, Thanks for the update, we all have priorities
            marco_adeti Marco RODRIGUES added a comment - - edited

             
            Hi,

            I am starting a training course with Néréide to improve my skills in community contribution and get to know OfBiz better

            I went through a lot of the related threads to get an idea of where this is at.
            Globally, there seem to have thre main ways with various implementation :
            - in code
            - in the application database
            - in system (file or environment variable)
            If solution that seems the most used by mainstrem projects, is to store overridable sensitive data in a system file, there seems to have no perfect solution.
            It depends on the person responsible for security and the policy they apply. Both to secure and to identify and respond to intrusions

            As Jacques Le Roux says : "I think we can suggest more than one solution and people can then pick the one fitting more for them"

            For the overload in the application code, this is a case which I think is handled by the usual development processes.
            For the other cases, which use a resource that can be qualified as external, and which make it possible to respond to problems of the "ISO/CEI 27002, 6.1.2 Segregation of duties" type, this requires that the application code provide for an activation of the available cases and a mechanism. overloading of variable values.
            Then each production manager will be able to manage independently of the application.

            Nereide agreed to pull us the code they use to manage environment variables on https://github.com/apache/ofbiz-framework/pull/355
            It seems to me that this is a good base on which to work. With your help, I should be able to come up with a solution that is useful and acceptable to all. {{}}

            marco_adeti Marco RODRIGUES added a comment - - edited   Hi, I am starting a training course with NĂ©rĂ©ide to improve my skills in community contribution and get to know OfBiz better I went through a lot of the related threads to get an idea of where this is at. Globally, there seem to have thre main ways with various implementation : - in code - in the application database - in system (file or environment variable) If solution that seems the most used by mainstrem projects, is to store overridable sensitive data in a system file, there seems to have no perfect solution. It depends on the person responsible for security and the policy they apply. Both to secure and to identify and respond to intrusions As Jacques Le Roux says : "I think we can suggest more than one solution and people can then pick the one fitting more for them" For the overload in the application code, this is a case which I think is handled by the usual development processes. For the other cases, which use a resource that can be qualified as external, and which make it possible to respond to problems of the "ISO/CEI 27002, 6.1.2 Segregation of duties" type, this requires that the application code provide for an activation of the available cases and a mechanism. overloading of variable values. Then each production manager will be able to manage independently of the application. Nereide agreed to pull us the code they use to manage environment variables on https://github.com/apache/ofbiz-framework/pull/355 It seems to me that this is a good base on which to work. With your help, I should be able to come up with a solution that is useful and acceptable to all. {{}}

            Hi Marco, welcome

            I twice reviewed the PR and it indeed looks good to me... as a starting example...

            jleroux Jacques Le Roux added a comment - Hi Marco, welcome I twice reviewed the PR and it indeed looks good to me... as a starting example...
            mbrohl Michael Brohl added a comment -

            This reminds me to write up the configuration mechanism we use in our projects, which allows for a git versioned maintenance of configuration sets for target systems. I think this would be another interesting approach.

            mbrohl Michael Brohl added a comment - This reminds me to write up the configuration mechanism we use in our projects, which allows for a git versioned maintenance of configuration sets for target systems. I think this would be another interesting approach.
            pierresmits Pierre Smits added a comment -

            Looking forward to seeing that appearing.

            pierresmits Pierre Smits added a comment - Looking forward to seeing that appearing.

            Hi,
            I will have some days off, two next week.

            If you have material to integrate into the common reflection, it would be interesting to send it. Even if it is not finalized, it will provide ideas that could be integrated, trying to fit the needs.

            marco_adeti Marco RODRIGUES added a comment - Hi, I will have some days off, two next week. If you have material to integrate into the common reflection, it would be interesting to send it. Even if it is not finalized, it will provide ideas that could be integrated, trying to fit the needs.

            I think it's a good evolution of the current situation. 

            Related to this part of the code.

            I know I looked at UtilProperties  some time and I noticed it pulls in dependency on UtilCache and that pulls in some other stuff.

            Ideally these would be composeable and replaceable. 

            It would be nice to be able to swap in a more modern configuration management.

            Since UtilPropertiees is based on static methods, one way (the only way ?!) is to replace the class on the class path. 

            This could be done if UtilProperties where in a `configuration-properties` module that we could  replace during distribution assembly with `configuration-awesome-new` that contains a class with same binary signature. 

            But having static initialized fields makes it hard to draw a clear boundary. 

            ieugen Ioan Eugen Stan added a comment - I think it's a good evolution of the current situation.  Related to this part of the code. I know I looked at UtilProperties  some time and I noticed it pulls in dependency on UtilCache and that pulls in some other stuff. Ideally these would be composeable and replaceable.  It would be nice to be able to swap in a more modern configuration management. Since UtilPropertiees is based on static methods, one way (the only way ?!) is to replace the class on the class path.  This could be done if UtilProperties where in a `configuration-properties` module that we could  replace during distribution assembly with `configuration-awesome-new` that contains a class with same binary signature.  But having static initialized fields makes it hard to draw a clear boundary. 

            pgil : Have you made any progress/changes on this? 
            I would love to test it out in the near future, hopefully with a client use case (Docker deploy). 

            That will bring valuable input and feedback that I would love to get back in OFBIz. 

            ieugen Ioan Eugen Stan added a comment - pgil : Have you made any progress/changes on this?  I would love to test it out in the near future, hopefully with a client use case (Docker deploy).  That will bring valuable input and feedback that I would love to get back in OFBIz. 
            ieugen Ioan Eugen Stan added a comment - - edited

            pgil
            I would keep JVM options inside the default Java env variables: JAVA_TOOLS_OPTIONS or JVM_OPTS (check this to be sure). 

            It's usually straight forward to pass those env vars to an app or a container. 

            I would NOT introduce new things like: 

            export OFB_MEM_XMS="-Xms256M"
            export OFB_MEM_XMX="-Xmx2048M"

            For sensitive data (cc jleroux ) , the practices for containers are to support another an extension to env variable properties. 

            We add support for `OFB_POSTGRES_PASS` that will contain the actual secret and support also `OFB_POSTGRES_PASS_FILE` that will contain a path to a file that holds the sensitive data. 

            Files can be made accessible only to the running application on most modern OS's

            Details of how this works for Docker Swarm is in the bellow document and other places online:  https://docs.docker.com/engine/swarm/secrets/ .

            Most container engines support handling secrets that are mounted as files in special memory file systems  that don't reach disk and are even protected / encrypted (modern CPU's have this capability). 

            See docker swarm secrets, kubernetes secrets, etc. 

            ieugen Ioan Eugen Stan added a comment - - edited pgil :  I would keep JVM options inside the default Java env variables: JAVA_TOOLS_OPTIONS or JVM_OPTS (check this to be sure).  It's usually straight forward to pass those env vars to an app or a container.  I would NOT introduce new things like:  export OFB_MEM_XMS="-Xms256M" export OFB_MEM_XMX="-Xmx2048M" For sensitive data (cc jleroux ) , the practices for containers are to support another an extension to env variable properties.  We add support for `OFB_POSTGRES_PASS` that will contain the actual secret and support also `OFB_POSTGRES_PASS_FILE` that will contain a path to a file that holds the sensitive data.  Files can be made accessible only to the running application on most modern OS's Details of how this works for Docker Swarm is in the bellow document and other places online:   https://docs.docker.com/engine/swarm/secrets/ . Most container engines support handling secrets that are mounted as files in special memory file systems  that don't reach disk and are even protected / encrypted (modern CPU's have this capability).  See docker swarm secrets, kubernetes secrets, etc. 

            Hello ieugen ,

            Thanks for your review, I work with marco_adeti whom i gave the relay to animate the subject. He should make a first proposal about a new idea, in some times.

            About the current patch, no progress yet, it is usable, and used in several customer project in our side, feel free to experience.

            pgil Gil Portenseigne added a comment - Hello ieugen , Thanks for your review, I work with marco_adeti whom i gave the relay to animate the subject. He should make a first proposal about a new idea, in some times. About the current patch, no progress yet, it is usable, and used in several customer project in our side, feel free to experience.

            Thanks.

             

            I saw there is the PR here that can improve life in these situations.

            I did add some feedback there https://github.com/apache/ofbiz-framework/pull/355 .

            Any plans to merge that now or later after more work? 

             

             

            ieugen Ioan Eugen Stan added a comment - Thanks.   I saw there is the PR here that can improve life in these situations. I did add some feedback there  https://github.com/apache/ofbiz-framework/pull/355 . Any plans to merge that now or later after more work?     

            I plan to complete the PR, and if no one oppose after last review, merge it.

            pgil Gil Portenseigne added a comment - I plan to complete the PR, and if no one oppose after last review, merge it.
            pierresmits Pierre Smits added a comment -

            Hi Gil,

            What is missing in this PR? Can this 'enhancement' not be addressed in a subsequent improvement commit/merge?

            pierresmits Pierre Smits added a comment - Hi Gil, What is missing in this PR? Can this 'enhancement' not be addressed in a subsequent improvement commit/merge?
            marco_adeti Marco RODRIGUES added a comment - - edited

            We are working on a more complete version. We have to manage the problem in a more general and homogeneous way with the values reported by the "Properties". And also allowing to merge and to overload the values.
            The current code will add heterogeneity when we want to move forward on a global solution quickly.
            We are currently evaluating the impact and the process that we could propose to do so.
            I will post the elements here as they are written so that you can participate as best as possible in the reflection and progress of this point. 
            Thanks

            marco_adeti Marco RODRIGUES added a comment - - edited We are working on a more complete version. We have to manage the problem in a more general and homogeneous way with the values reported by the "Properties". And also allowing to merge and to overload the values. The current code will add heterogeneity when we want to move forward on a global solution quickly. We are currently evaluating the impact and the process that we could propose to do so. I will post the elements here as they are written so that you can participate as best as possible in the reflection and progress of this point.  Thanks
            pierresmits Pierre Smits added a comment -

            Thanks marco_adeti ,

            Working on a 'more' complete version implies that current version (PR355) works. (And this has not been contested). Any improvements to that (the current PR) can be handled in due time as subsequent PRs.

            There is no argument to delay/postponing merging this PR, other than this 'there is more in the pipeline' comment. 

            pierresmits Pierre Smits added a comment - Thanks marco_adeti  , Working on a 'more' complete version implies that current version (PR355) works. (And this has not been contested). Any improvements to that (the current PR) can be handled in due time as subsequent PRs. There is no argument to delay/postponing merging this PR, other than this 'there is more in the pipeline' comment. 
            mbrohl Michael Brohl added a comment -

            The contributors of the PR themselves (Gil/Marco) want to improve the PR. I see no reason why you are pushing them to merge a PR they want to improve first.

            It's a better approach to quality than to hurry with a "it works" attitude and they need no argument to do so. I appreciate this approach.

            mbrohl Michael Brohl added a comment - The contributors of the PR themselves (Gil/Marco) want to improve the PR. I see no reason why you are pushing them to merge a PR they want to improve first. It's a better approach to quality than to hurry with a "it works" attitude and they need no argument to do so. I appreciate this approach.

            hi pierresmits,

            Your reasoning is correct if it was just a question of improving the existing PR by adding code. But the aim is ti replace the way the UtilProperties.getEnvironmentProperty function integrates with the application.

            Here are the two issues for which I advise against accepting this PR as is:

            1. The function must be replaced/renamed, or managed in another class because it does not respect the definition of the Utilproperties class: "Utilities for working with properties files" (https://ci.apache.org/projects/ofbiz/site/stable/javadocs/org/apache/ofbiz/base/util/UtilProperties.html)
            At least because environment variables are not "properties files" in the way they are handled here.

            2. The direct use of the UtilProperties.getEnvironmentProperty function in the java code for calling variables is not the better way. It is specific to environment variables, and therefore blocks or complicates any evolution.
            Better to start on another function / class which would manage which would frame the call to variables and manage mergers and overloads.
            But we still need to make a quick inventory of what already exists in order to be able to offer a quick implementation that does not block us.

            Hoping to be clear and remaining available for any questions.

            P.S. Sorry for not giving these details before, but since I haven't had detailed feedback and any other alternative implementation examples, I didn't think it would be useful to pollute this thread with Nereid code-specific explanations. but you are right to have all explanations.
            This thread has been relaunched, first of all, to share the contributions in order to move forward on a solution. Not to push this PR as it is. 

            So if you can retrieve others implementation of this issue to enrich the reflexion , they are welcomed !
            Thanks for your feedback.

            marco_adeti Marco RODRIGUES added a comment - hi pierresmits , Your reasoning is correct if it was just a question of improving the existing PR by adding code. But the aim is ti replace the way the UtilProperties.getEnvironmentProperty function integrates with the application. Here are the two issues for which I advise against accepting this PR as is: 1. The function must be replaced/renamed, or managed in another class because it does not respect the definition of the Utilproperties class: "Utilities for working with properties files" ( https://ci.apache.org/projects/ofbiz/site/stable/javadocs/org/apache/ofbiz/base/util/UtilProperties.html ) At least because environment variables are not "properties files" in the way they are handled here. 2. The direct use of the UtilProperties.getEnvironmentProperty function in the java code for calling variables is not the better way. It is specific to environment variables, and therefore blocks or complicates any evolution. Better to start on another function / class which would manage which would frame the call to variables and manage mergers and overloads. But we still need to make a quick inventory of what already exists in order to be able to offer a quick implementation that does not block us. Hoping to be clear and remaining available for any questions. P.S. Sorry for not giving these details before, but since I haven't had detailed feedback and any other alternative implementation examples, I didn't think it would be useful to pollute this thread with Nereid code-specific explanations. but you are right to have all explanations. This thread has been relaunched, first of all, to share the contributions in order to move forward on a solution. Not to push this PR as it is.  So if you can retrieve others implementation of this issue to enrich the reflexion , they are welcomed ! Thanks for your feedback.
            pierresmits Pierre Smits added a comment -

            HI marco_adeti ,

            First let me wish you a healthy, happy, interesting and prosperous New Year too.

            Thanks for your apologies for not providing additional details before. We all appreciate this, and understand that (more) pressing/interesting things get in the way. 

            For sure, an overhaul of how OFBiz is currently configured (through EntityUtilProperties for runtime configuration, UtilProperties for configuration through property files, and this PR for configuration via environment variables as in e.g. Docker containers) appears to be warranted. Only the 'nobody' contributor could be against that. 

            And equally obvious is that such will take a significant amount of time. Not only the investigation what it will impact, and how it will be implemented. The remarks in your posted comment suggests that. As such it is symptomatic for the entire state of OFBiz: a continuous state of Work In Progress/Improvement.

            Nonetheless, we need not have that (continuous state of WIP) delay that what will work now (the PR) going into the codebase. As always, more pressing/interesting things can be expected to pop up. 
            As is commonly understood: don't delay what can be achieved now!

            pierresmits Pierre Smits added a comment - HI marco_adeti  , First let me wish you a healthy, happy, interesting and prosperous New Year too. Thanks for your apologies for not providing additional details before. We all appreciate this, and understand that (more) pressing/interesting things get in the way.  For sure, an overhaul of how OFBiz is currently configured (through EntityUtilProperties for runtime configuration, UtilProperties for configuration through property files, and this PR for configuration via environment variables as in e.g. Docker containers) appears to be warranted. Only the 'nobody' contributor could be against that.  And equally obvious is that such will take a significant amount of time. Not only the investigation what it will impact, and how it will be implemented. The remarks in your posted comment suggests that. As such it is symptomatic for the entire state of OFBiz: a continuous state of Work In Progress/Improvement. Nonetheless, we need not have that (continuous state of WIP) delay that what will work now (the PR) going into the codebase. As always, more pressing/interesting things can be expected to pop up.  As is commonly understood: don't delay what can be achieved now!

            People

              pgil Gil Portenseigne
              pgil Gil Portenseigne
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: