OAM 11G R2 PS2 Upgrade Issue:

Error:
oracle.security.am.admin.config.util.DataConnectionUtility getConnection
WARNING: Failed to get connection
java.sql.SQLException: ORA-01017: invalid username/password; logon denied

        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)
        at oracle.jdbc.driver.T4CTTIfun.processError(T4CTTIfun.java:600)
        at oracle.jdbc.driver.T4CTTIoauthenticate.processError(T4CTTIoauthenticate.java:445)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
        at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:380)
        at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:760)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:401)
        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
        at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:207)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:157)
        at oracle.security.am.admin.config.util.DataConnectionUtility.getConnection(DataConnectionUtility.java:139)
        at oracle.security.am.upgrade.framework.psfe.util.PSFEUtils.validateDBConnectionDetails(PSFEUtils.java:232)
        at oracle.security.am.upgrade.framework.psfe.PSFEFramework.processOffline(PSFEFramework.java:193)
        at oracle.security.am.wlst.util.WLSTServerUtil.executeCommand(WLSTServerUtil.java:230)
        at oracle.security.am.wlst.WLSTWrapper.executeCommand(WLSTWrapper.java:152)
        at oracle.security.am.wlst.WLSTWrapper.execute(WLSTWrapper.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.python.core.PyReflectedFunction.__call__(Unknown Source)
        at org.python.core.PyReflectedFunction.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
        at OamInternal_handler$py.patchUpgradeImpl$78(/Middleware/Oracle_IDM1/common/script_handlers/OamInternal_handler.py:1313)
        at OamInternal_handler$py.call_function(/Middleware/Oracle_IDM1/common/script_handlers/OamInternal_handler.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
        at Oam_common$py.upgradeConfig$73(/Middleware/Oracle_IDM1/common/script_handlers/Oam_common.py:883)
        at Oam_common$py.call_function(/Middleware/Oracle_IDM1/common/script_handlers/Oam_common.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
        at org.python.pycode._pyx49.upgradeConfig$74(/Middleware/Oracle_IDM1/common/wlst/Oam.py:553)
        at org.python.pycode._pyx49.call_function(/Middleware/Oracle_IDM1/common/wlst/Oam.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.pycode._pyx65.f$0(<console>:1)
        at org.python.pycode._pyx65.call_function(<console>)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyCode.call(Unknown Source)
        at org.python.core.Py.runCode(Unknown Source)
        at org.python.core.Py.exec(Unknown Source)
        at org.python.util.PythonInterpreter.exec(Unknown Source)
        at org.python.util.InteractiveInterpreter.runcode(Unknown Source)
        at org.python.util.InteractiveInterpreter.runsource(Unknown Source)
        at org.python.util.InteractiveInterpreter.runsource(Unknown Source)
        at weblogic.management.scripting.WLST.main(WLST.java:188)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at weblogic.WLST.main(WLST.java:29)
Oct 30, 2014 11:59:25 PM oracle.security.am.upgrade.framework.psfe.PSFEFramework processOffline
SEVERE: Database connection details are incorrect
Oct 30, 2014 11:59:25 PM oracle.security.am.wlst.WLSTWrapper execute
SEVERE: Unable to execute the WLST Command
java.lang.Exception: upgradeConfig failed
        at oracle.security.am.wlst.util.WLSTServerUtil.executeCommand(WLSTServerUtil.java:232)
        at oracle.security.am.wlst.WLSTWrapper.executeCommand(WLSTWrapper.java:152)
        at oracle.security.am.wlst.WLSTWrapper.execute(WLSTWrapper.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.python.core.PyReflectedFunction.__call__(Unknown Source)
        at org.python.core.PyReflectedFunction.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
        at OamInternal_handler$py.patchUpgradeImpl$78(/Middleware/Oracle_IDM1/common/script_handlers/OamInternal_handler.py:1313)
        at OamInternal_handler$py.call_function(/Middleware/Oracle_IDM1/common/script_handlers/OamInternal_handler.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
        at Oam_common$py.upgradeConfig$73(/Middleware/Oracle_IDM1/common/script_handlers/Oam_common.py:883)
        at Oam_common$py.call_function(/Middleware/Oracle_IDM1/common/script_handlers/Oam_common.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
        at org.python.pycode._pyx49.upgradeConfig$74(/Middleware/Oracle_IDM1/common/wlst/Oam.py:553)
        at org.python.pycode._pyx49.call_function(/Middleware/Oracle_IDM1/common/wlst/Oam.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.pycode._pyx65.f$0(<console>:1)
        at org.python.pycode._pyx65.call_function(<console>)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyCode.call(Unknown Source)
        at org.python.core.Py.runCode(Unknown Source)
        at org.python.core.Py.exec(Unknown Source)
        at org.python.util.PythonInterpreter.exec(Unknown Source)
        at org.python.util.InteractiveInterpreter.runcode(Unknown Source)
        at org.python.util.InteractiveInterpreter.runsource(Unknown Source)
        at org.python.util.InteractiveInterpreter.runsource(Unknown Source)
        at weblogic.management.scripting.WLST.main(WLST.java:188)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at weblogic.WLST.main(WLST.java:29)
Unable to execute the commandupgradeConfig failed

Solution: 

Make sure the schema details are in UPPER case(SYS,PREFIX_OAM) and password is correct.
   

Error Code: 28001 org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLException: ORA-28001: the password has expired

Error:
######
Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Info: Data source is: opss-DBDS
INFO: Found persistence provider "org.eclipse.persistence.jpa.PersistenceProvider". OpenJPA will not be used.
INFO: Found persistence provider "org.eclipse.persistence.jpa.PersistenceProvider". OpenJPA will not be used.
[EL Severe]: 2014-10-14 09:42:20.49--ServerSession(957605646)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.1.v20111018-r10243): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: ORA-28001: the password has expired

Error Code: 28001
Oct 14, 2014 9:42:20 AM oracle.security.jps.internal.credstore.ldap.LdapCredentialStore init
WARNING: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
Error: Diagnostics data was not saved to the credential store.
Error: Validate operation has failed.
Need to do the security configuration first!


Solution:
#########
Run the below queries

SELECT owner, comp_name, version, status, upgraded FROM schema_version_registry WHERE comp_name like '%Oracle%';esukl

Result:
#######
PROFILE                RESOURCE_NAME                                RESOURCE LIMIT
------------------------------ -------------------------------- ------------------------------------
DEFAULT                FAILED_LOGIN_ATTEMPTS              PASSWORD10
DEFAULT                PASSWORD_LIFE_TIME                     PASSWORDUNLIMITED
DEFAULT                PASSWORD_REUSE_TIME                 PASSWORDUNLIMITED
DEFAULT                PASSWORD_REUSE_MAX                  PASSWORDUNLIMITED
DEFAULT                PASSWORD_VERIFY_FUNCTION     PASSWORDNULL
DEFAULT                PASSWORD_LOCK_TIME                   PASSWORD1
DEFAULT                PASSWORD_GRACE_TIME                PASSWORDUNLIMITED
         
select USERNAME,EXPIRY_DATE,LOCK_DATE,ACCOUNT_STATUS from dba_users where username like '%DEV%';

Result:
#######
USERNAME               EXPIRY_DA LOCK_DATE    ACCOUNT_STATUS
------------------------------ --------- -----------------------------------------
DEV_SOAINFRA         OPEN
DEV_OIF                      OPEN
DEVBI_MDS                OPEN
DEV_IAU_APPEND    OPEN
DEV_ORASDPM         OPEN
DEV_IAU                     OPEN
DEVBI_BIPLATFORM    OPEN
DEV_IAU_VIEWER    OPEN
DEV_OAAM                OPEN
DEV_MDS                   OPEN
DEV_OAM                  OPEN
DEV_OPSS               09-OCT-14    EXPIRED

12 rows selected.

Per above result, it is evident that DEV_OPSS password has been expired. so we need to reset that

> alter user DEV_OPSS identified by Password123;

Restart everything :)

OIM Important Queries.

User Details:

• select * from USR where UPPER(usr_login)='username'

AUDIT: 

• select * from dev_oim.UPA_FIELDS UPA_FIELDS LEFT outer JOIN dev_oim.upa_usr  upa_USR ON upa_fields.upa_usr_key=upa_USR.UPA_USR_KEY where UPPER(upa_usr.usr_login) like 'userlogin'

Resource Audit: 

• select * from dev_oim.UPA_RESOURCE

Plugins Info: 

• select * from dev_oim.plugins

Audit Table: 

•   select * from dev_oim.UPA where usr_key='userkey' order by upa_key desc

Jar file: 

• select * from dev_oim.oimhome_jars

Plugins:

•   select * from dev_oim.plugins

OIM SOAP Error: Bad response: 404 Not Found from url http://:14000/reqsvc/reqsvc

Error
Oct 15, 2014 5:50:49 PM EDT> <Warning> <oracle.integration.platform.blocks.soap> <BEA-000000> <Unable to invoke endpoint URI "http://:14000/reqsvc/reqsvc" successfully due to: Unable to invoke endpoint URI "http://:14000/reqsvc/reqsvc" successfully due to: javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 404 Not Found from url http://:14000/reqsvc/reqsvc>
<Oct 15, 2014 5:50:49 PM EDT> <Error> <oracle.soa.bpel.engine.ws> <BEA-000000> <got FabricInvocationException
javax.xml.soap.SOAPException: Bad response: 404 Not Found from url http://:14000/reqsvc/reqsvc
        at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection.processHttpResponseCode(HttpSOAPConnection.java:636)
        at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection.post2(HttpSOAPConnection.java:501)
        at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection$PrivilegedPost.run(HttpSOAPConnection.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection.call2(HttpSOAPConnection.java:229)
        at oracle.j2ee.ws.common.transport.HttpTransport.transmit(HttpTransport.java:75)
        at oracle.j2ee.ws.common.async.MessageSender.call(MessageSender.java:67)
        at oracle.j2ee.ws.common.async.Transmitter.transmitSync(Transmitter.java:134)
        at oracle.j2ee.ws.common.async.Transmitter.transmit(Transmitter.java:90)
        at oracle.j2ee.ws.common.async.RequestorImpl.transmit(RequestorImpl.java:273)


Solution: 

Deploy following jar on OIM server.

OIM_HOME/server/webapp/optional/reqsvc.ear.