Persistent store "_WLS_AdminServer" could not be deployed in Weblogic Server

ISSUE:

The persistent store "_WLS_AdminServer" could not be deployed resulting in failure of Admin Server during startup.

ERROR OBSERVED:

Logs : $DOMAIN_NAME/servers/AdminServer/logs/AdminServer.out

<<

<Error> <Store> <BEA-280061> <The persistent store "_WLS_AdminServer" could not be deployed: weblogic.store.PersistentStoreException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.

weblogic.store.PersistentStoreException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.

at weblogic.store.io.file.Heap.open(Heap.java:312)

at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:413)

at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:404)

  at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)

Truncated. see log file for complete stacktrace

> <Critical> <WebLogicServer> <BEA-000362> <Server failed.>

>>

ROOT CAUSE:

1) Above error comes mostly when weblogic is not able to read the file store .DAT file. 

2) Also this file could be corrupted due to forceful shutdown (kill -9 pid)

SOLUTION:

Step 1) Go to below mentioned directory

cd $DomainHome/servers/AdminServer/data/store

Step 2) find . –name  *.DAT

cd $DomainHome/servers/AdminServer/data/store

username@hostname # ls

default diagnostics

username@hostname # find . .name *.DAT

.

./diagnostics

./diagnostics/WLS_DIAGNOSTICS000000.DAT

./default

./default/_WLS_ADMINSERVER000000.DAT

Step 3) Verify the file name in your result and error message should be same.

Step 4) Rename this file and move from this directory to some other directory.

cd $DomainHome/servers/AdminServer/data/store/default

username@hostname# ls

_WLS_ADMINSERVER000000.DAT

mv _WLS_ADMINSERVER000000.DAT _WLS_ADMINSERVER000000.DAT_ORG

Step 5) Find "EmbeddedLDAP.lok" and "AdminServer.lok" as well and remove the same.

cd $DomainHome/servers/AdminServer/data//ldap/ldapfiles

rm -rf EmbeddedLDAP.lok

cd $DomainHome/servers/AdminServer/tmp

rm -rf AdminServer.lok

Step 6) Check the port using netstat –an | grep <Weblogic server port>, there should not be any open connection to this port.

username@hostname # netstat -an | grep 7001

username@hostname #

Step 7) Start your Weblogic Admin Server.

Upon startup above files(EmbeddedLDAP.lok, AdminServer.lok and _WLS_ADMINSERVER000000.DAT) shall be auto created.

Note: .DAT file is very important file, and contains business data as well in Production system. Please take a backup of this file before doing any operation on it, so that later this file can be analyzed to complete those transaction.