OAM Policy Store unavailabe due to database archiver error

ISSUE: 

Policy Store unavailable in OAM Console due to database archiver error.

ERROR OBSERVED:

Logs @DOMAIN_HOME/server/oam_server1/oam_serrver1-diagnostic.log

[2015-11-09T08:08:04.076+03:00] [oam_server1] [WARNING] [OAMSSA-04006] [oracle.oam.foundation.access] [tid: OAM - SME Expired Session Reaper (JDBC)] [userId: <anonymous>] [ecid: 0000L3S8nVCCwkG6yzfd6G1MF7Cz000003,1:27746] [APP: oam_server#11.1.2.0.0] Cannot get connection from data source.[[
weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: ORA-00257: archiver error. Connect internal only, until freed.
at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:250)
at weblogic.jdbc.pool.Driver.connect(Driver.java:142)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
at weblogic.jdbc.jts.Driver.connect(Driver.java:127)
at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
at oracle.security.am.foundation.mapimpl.coherence.store.DataConnectionUtility.configureConnection(DataConnectionUtility.java:223)
at oracle.security.am.foundation.mapimpl.coherence.store.DataConnectionUtility.getConnection(DataConnectionUtility.java:191)
at oracle.security.am.engines.sme.mapimpl.db.SmeStoreLifecycle.run(SmeStoreLifecycle.java:262)
at java.lang.Thread.run(Thread.java:682)

While checking connection of oamDS datasource with database from Weblogic Console , same  error will be  observed.
 

SOLUTION:

This is database issue so ask your database team to do the needful.
In this you want to resolve this issue by your own then follow below steps.

Step 1) Login to target database and find location where archive logs are stored.

$sqlplus / as sysdba
SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 493
Next log sequence to archive 495
Current log sequence 495
SQL>

In above case Archive log  Destination is  USE_DB_RECOVERY_FILE_DEST which means archive logs are stored in flash recovery area(By default its
USE_DB_RECOVERY_FILE_DEST)
But in most cases archive log destination will be some user defined location.
In either of cases , we need to delete old archive logs from location mentioned in Archive destination.

Step 2) Go to that location where archive logs are stored and delete old archive logs manually with OS command.
 In our case USE_DB_RECOVERY_FILE_DEST =
/u02/oracle/devoracle/flash_recovery_area/OAMDB/archivelog
$ cd /u02/oracle/devoracle/flash_recovery_area/OAMDB/archivelog
$ ls -lrth
total 57
drwxr-x--- 2 oracle dba 19 Oct 21 23:34 2015_10_21
drwxr-x--- 2 oracle dba 17 Oct 22 22:03 2015_10_22
drwxr-x--- 2 oracle dba 20 Oct 23 23:11 2015_10_23
drwxr-x--- 2 oracle dba 20 Oct 24 22:12 2015_10_24
drwxr-x--- 2 oracle dba 20 Oct 25 22:31 2015_10_25
drwxr-x--- 2 oracle dba 30 Oct 26 22:08 2015_10_26
drwxr-x--- 2 oracle dba 13 Oct 27 23:00 2015_10_27
drwxr-x--- 2 oracle dba 11 Oct 28 22:02 2015_10_28
drwxr-x--- 2 oracle dba 22 Oct 29 22:06 2015_10_29
drwxr-x--- 2 oracle dba 14 Oct 30 22:08 2015_10_30
drwxr-x--- 2 oracle dba 11 Oct 31 15:50 2015_10_31
drwxr-x--- 2 oracle dba 12 Nov 2 22:06 2015_11_02
drwxr-x--- 2 oracle dba 11 Nov 3 22:24 2015_11_03
drwxr-x--- 2 oracle dba 30 Nov 4 23:06 2015_11_04
drwxr-x--- 2 oracle dba 29 Nov 5 22:55 2015_11_05
drwxr-x--- 2 oracle dba 19 Nov 6 22:03 2015_11_06
drwxr-x--- 2 oracle dba 2 Nov 7 00:00 2015_11_07
drwxr-x--- 2 oracle dba 2 Nov 8 00:00 2015_11_08
drwxr-x--- 2 oracle dba 5 Nov 9 08:08 2015_11_09

$ rm -rf  *;

Step 3) As archive logs are useful to restore database in case of any failure. So after deleting archive logs with OS command make sure you execute below command after connecting to RMAN  (Recovery Manager)
rman target /
RMAN> crosscheck archivelog all;
RMAN> delete expired archivelog all;

Check  Policy Store in OAM Console and we should be able to access the same.
Also connection of oamDS datasource with database is successful as shown below.