Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2608

Comma in oozie.service.JPAService.jdbc.password value results in authentication error

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 4.2.0
    • 5.0.0b1
    • core
    • None
    • HDP 2.4 latest, Oozie 4.2.0.2.4.2.0-258

    Description

      I have attempted to install Oozie with Postgres db backend via HDP 2.4. I was able to test DB connection via Ambari, everything was correct. However, the server failed to start with this error:

      2016-07-15 11:46:39,580  INFO URIHandlerService:520 - SERVER[hadoop-master01.production.sociablelabs.net] Loaded default urihandler org.apache.oozie.dependency.FSURIHandler
      2016-07-15 11:46:40,323  INFO HadoopAccessorService:520 - SERVER[hadoop-master01.production.sociablelabs.net] password key is oozie.service.JPAService.jdbc.password
      2016-07-15 11:46:40,331  INFO HadoopAccessorService:520 - SERVER[hadoop-master01.production.sociablelabs.net] credential provider path is null
      2016-07-15 11:46:40,337  INFO HadoopAccessorService:520 - SERVER[hadoop-master01.production.sociablelabs.net] password is null
      2016-07-15 11:46:41,176 FATAL Services:514 - SERVER[hadoop-master01.production.sociablelabs.net] Runtime Exception during Services Load. Check your list of 'oozie.services' or 'o
      ozie.services.ext'
      2016-07-15 11:46:41,194 FATAL Services:514 - SERVER[hadoop-master01.production.sociablelabs.net] E0103: Could not load service classes, Cannot create PoolableConnectionFactory (F
      ATAL: password authentication failed for user "oozie")
      org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Cannot create PoolableConnectionFactory (FATAL: password authentication failed for user "oozie")
              at org.apache.oozie.service.Services.loadServices(Services.java:309)
              at org.apache.oozie.service.Services.init(Services.java:213)
      ....
      Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "oozie"
              at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
              at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
              at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
      ...
      

      Schema verification at startup worked fine and I could not find any bugs related to it. So, I started getting suspicious about my password value, which was like "pO,e43DpjK" (not exactly but it was a bunch of letters, number and one comma). I suspected that the comma was probably interpreted as a value list separator so I have changed the password removing the comma. Server started flawlessly.

      I am not 100% sure if it is Ambari that should be escaping (or validating?) the value or Oozie that fails to read the value as single string.

      Attachments

        1. OOZIE-2608-003.patch
          1 kB
          GĂ©zapeti
        2. OOZIE-2608-002.patch
          2 kB
          GĂ©zapeti
        3. OOZIE-2608-001.patch
          1 kB
          GĂ©zapeti

        Issue Links

          Activity

            julian.eberius Julian Eberius added a comment -

            We encountered the same issue when installing Oozie as part of Cloudera's CDH. Our password contained the special characters "!" and "*". The scripts creating the Oozie DB ran fine (using Postgres), but Oozie itself wouldn't start, printing a similar authentication error. Removing the special characters solved the issue.

            Therefore, this does not seem to be an Ambari issue, but something in core Oozie is mangling the passwords...

            julian.eberius Julian Eberius added a comment - We encountered the same issue when installing Oozie as part of Cloudera's CDH. Our password contained the special characters "!" and "*". The scripts creating the Oozie DB ran fine (using Postgres), but Oozie itself wouldn't start, printing a similar authentication error. Removing the special characters solved the issue. Therefore, this does not seem to be an Ambari issue, but something in core Oozie is mangling the passwords...
            gezapeti GĂ©zapeti added a comment -

            I've found this Atlassian issue which is similar.
            I've changed the jpa properties we're putting together that it handles the username and the password in separate jpa properties. I'm working on a way to test this out in a junit test.

            gezapeti GĂ©zapeti added a comment - I've found this Atlassian issue which is similar. I've changed the jpa properties we're putting together that it handles the username and the password in separate jpa properties. I'm working on a way to test this out in a junit test.
            hadoopqa Hadoop QA added a comment -

            Testing JIRA OOZIE-2608

            Cleaning local git workspace

            ----------------------------

            +1 PATCH_APPLIES
            +1 CLEAN
            -1 RAW_PATCH_ANALYSIS
            . +1 the patch does not introduce any @author tags
            . +1 the patch does not introduce any tabs
            . +1 the patch does not introduce any trailing spaces
            . +1 the patch does not introduce any line longer than 132
            . -1 the patch does not add/modify any testcase
            +1 RAT
            . +1 the patch does not seem to introduce new RAT warnings
            +1 JAVADOC
            . +1 the patch does not seem to introduce new Javadoc warnings
            +1 COMPILE
            . +1 HEAD compiles
            . +1 patch compiles
            . +1 the patch does not seem to introduce new javac warnings
            +1 There are no new bugs found in total.
            . +1 There are no new bugs found in [server].
            . +1 There are no new bugs found in [client].
            . +1 There are no new bugs found in [core].
            . +1 There are no new bugs found in [docs].
            . +1 There are no new bugs found in [hadooplibs/hadoop-utils-2].
            . +1 There are no new bugs found in [tools].
            . +1 There are no new bugs found in [examples].
            . +1 There are no new bugs found in [sharelib/streaming].
            . +1 There are no new bugs found in [sharelib/sqoop].
            . +1 There are no new bugs found in [sharelib/distcp].
            . +1 There are no new bugs found in [sharelib/oozie].
            . +1 There are no new bugs found in [sharelib/hcatalog].
            . +1 There are no new bugs found in [sharelib/hive].
            . +1 There are no new bugs found in [sharelib/hive2].
            . +1 There are no new bugs found in [sharelib/pig].
            . +1 There are no new bugs found in [sharelib/spark].
            +1 BACKWARDS_COMPATIBILITY
            . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
            . +1 the patch does not modify JPA files
            +1 TESTS
            . Tests run: 1831
            +1 DISTRO
            . +1 distro tarball builds with the patch

            ----------------------------
            -1 Overall result, please check the reported -1(s)

            The full output of the test-patch run is available at

            . https://builds.apache.org/job/oozie-trunk-precommit-build/3507/

            hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2608 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 There are no new bugs found in total. . +1 There are no new bugs found in [server] . . +1 There are no new bugs found in [client] . . +1 There are no new bugs found in [core] . . +1 There are no new bugs found in [docs] . . +1 There are no new bugs found in [hadooplibs/hadoop-utils-2] . . +1 There are no new bugs found in [tools] . . +1 There are no new bugs found in [examples] . . +1 There are no new bugs found in [sharelib/streaming] . . +1 There are no new bugs found in [sharelib/sqoop] . . +1 There are no new bugs found in [sharelib/distcp] . . +1 There are no new bugs found in [sharelib/oozie] . . +1 There are no new bugs found in [sharelib/hcatalog] . . +1 There are no new bugs found in [sharelib/hive] . . +1 There are no new bugs found in [sharelib/hive2] . . +1 There are no new bugs found in [sharelib/pig] . . +1 There are no new bugs found in [sharelib/spark] . +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 1831 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/3507/
            gezapeti GĂ©zapeti added a comment -

            I can confirm that Oozie master won't start with the password oo*!zi,e before the patch and it works fine with the same password after the patch is applied on MySQL.

            gezapeti GĂ©zapeti added a comment - I can confirm that Oozie master won't start with the password oo*!zi,e before the patch and it works fine with the same password after the patch is applied on MySQL.
            gezapeti GĂ©zapeti added a comment -

            asasvari, please review

            gezapeti GĂ©zapeti added a comment - asasvari , please review

            Can you add a unit test that covers this?

            asasvari Attila Sasvári added a comment - Can you add a unit test that covers this?

            On second thought, that might not be necessary as it would basically re-test MessageFormat.
            So +1

            asasvari Attila Sasvári added a comment - On second thought, that might not be necessary as it would basically re-test MessageFormat . So +1

            gezapeti could you please rebase the patch? Thanks

            asasvari Attila Sasvári added a comment - gezapeti could you please rebase the patch? Thanks
            gezapeti GĂ©zapeti added a comment -

            Attaching rebased patch.
            Also, I've removed the "url" from the comma-separated list which will probably make the change introduced in OOZIE-2574 not needed any more. abhishekbafna, can you check on this issue? I don't have easy access to a replication-enabled MySQL.

            gezapeti GĂ©zapeti added a comment - Attaching rebased patch. Also, I've removed the "url" from the comma-separated list which will probably make the change introduced in OOZIE-2574 not needed any more. abhishekbafna , can you check on this issue? I don't have easy access to a replication-enabled MySQL.
            asasvari Attila Sasvári added a comment - - edited

            gezapeti it looks good to me, +1 (pending jenkins)

            asasvari Attila Sasvári added a comment - - edited gezapeti it looks good to me, +1 (pending jenkins)
            gezapeti GĂ©zapeti added a comment -

            It looks like there are some errors with this patch:

            -1 TESTS
                Tests run: 2009
                Tests failed: 0
                Tests errors: 1769
            
                The patch failed the following testcases:
            

            It can't report to Jira bacuase the list is too long:

            bin/test-patch: line 126: /home/jenkins/jenkins-slave/workspace/oozie-trunk-precommit-build/test-patch/tools/jira-cli/jira.sh: Argument list too long
            
            gezapeti GĂ©zapeti added a comment - It looks like there are some errors with this patch: -1 TESTS Tests run: 2009 Tests failed: 0 Tests errors: 1769 The patch failed the following testcases: It can't report to Jira bacuase the list is too long: bin/test-patch: line 126: /home/jenkins/jenkins-slave/workspace/oozie-trunk-precommit-build/test-patch/tools/jira-cli/jira.sh: Argument list too long
            gezapeti GĂ©zapeti added a comment -

            Putting back url into the basic connection props as it breaks things.

            gezapeti GĂ©zapeti added a comment - Putting back url into the basic connection props as it breaks things.
            hadoopqa Hadoop QA added a comment -

            Testing JIRA OOZIE-2608

            Cleaning local git workspace

            ----------------------------

            +1 PATCH_APPLIES
            +1 CLEAN
            -1 RAW_PATCH_ANALYSIS
            . +1 the patch does not introduce any @author tags
            . +1 the patch does not introduce any tabs
            . +1 the patch does not introduce any trailing spaces
            . +1 the patch does not introduce any line longer than 132
            . -1 the patch does not add/modify any testcase
            +1 RAT
            . +1 the patch does not seem to introduce new RAT warnings
            +1 JAVADOC
            . +1 the patch does not seem to introduce new Javadoc warnings
            . WARNING: the current HEAD has 77 Javadoc warning(s)
            +1 COMPILE
            . +1 HEAD compiles
            . +1 patch compiles
            . +1 the patch does not seem to introduce new javac warnings
            +1 There are no new bugs found in total.
            . +1 There are no new bugs found in [client].
            . +1 There are no new bugs found in [sharelib/hive].
            . +1 There are no new bugs found in [sharelib/sqoop].
            . +1 There are no new bugs found in [sharelib/spark].
            . +1 There are no new bugs found in [sharelib/streaming].
            . +1 There are no new bugs found in [sharelib/hive2].
            . +1 There are no new bugs found in [sharelib/oozie].
            . +1 There are no new bugs found in [sharelib/hcatalog].
            . +1 There are no new bugs found in [sharelib/pig].
            . +1 There are no new bugs found in [sharelib/distcp].
            . +1 There are no new bugs found in [docs].
            . +1 There are no new bugs found in [server].
            . +1 There are no new bugs found in [core].
            . +1 There are no new bugs found in [examples].
            . +1 There are no new bugs found in [tools].
            +1 BACKWARDS_COMPATIBILITY
            . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
            . +1 the patch does not modify JPA files
            +1 TESTS
            . Tests run: 2009
            . Tests rerun: 17
            . Tests failed at first run: org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand,org.apache.oozie.jms.TestJMSJobEventListener,
            +1 DISTRO
            . +1 distro tarball builds with the patch

            ----------------------------
            -1 Overall result, please check the reported -1(s)

            . There is at least one warning, please check

            The full output of the test-patch run is available at

            . https://builds.apache.org/job/oozie-trunk-precommit-build/4027/

            hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2608 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings . WARNING : the current HEAD has 77 Javadoc warning(s) +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 There are no new bugs found in total. . +1 There are no new bugs found in [client] . . +1 There are no new bugs found in [sharelib/hive] . . +1 There are no new bugs found in [sharelib/sqoop] . . +1 There are no new bugs found in [sharelib/spark] . . +1 There are no new bugs found in [sharelib/streaming] . . +1 There are no new bugs found in [sharelib/hive2] . . +1 There are no new bugs found in [sharelib/oozie] . . +1 There are no new bugs found in [sharelib/hcatalog] . . +1 There are no new bugs found in [sharelib/pig] . . +1 There are no new bugs found in [sharelib/distcp] . . +1 There are no new bugs found in [docs] . . +1 There are no new bugs found in [server] . . +1 There are no new bugs found in [core] . . +1 There are no new bugs found in [examples] . . +1 There are no new bugs found in [tools] . +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 2009 . Tests rerun: 17 . Tests failed at first run: org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand,org.apache.oozie.jms.TestJMSJobEventListener, +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) . There is at least one warning, please check The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/4027/

            +1

            Thanks gezapeti, committed to master!

            asasvari Attila Sasvári added a comment - +1 Thanks gezapeti , committed to master!

            Closing issue; Oozie 5.0.0-beta1 is released

            asasvari Attila Sasvári added a comment - Closing issue; Oozie 5.0.0-beta1 is released

            People

              gezapeti GĂ©zapeti
              ngrigoriev@gmail.com Nikolai Grigoriev
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: