EmbeddedLDAP java.lang.ArrayIndexOutOfBoundsException in Admin Server

ISSUE:

Admin server fails during start up.

ERROR OBSERVED:

--><Error> <EmbeddedLDAP> <ADMIN01> <AdminServer> <VDE Replication Thread> <> <> <>
<1294659336925> <BEA-000000> <Error reading changelog entry#: 0>
<Critical> <EmbeddedLDAP> <ADMIN01> <AdminServer> <VDE Replication Thread> <> <> <>
<1294659336925> <BEA-000000> <java.lang.NullPointerException
at com.octetstring.vde.EntryChanges.readBytes(EntryChanges.java:279)
at com.octetstring.vde.EntryChanges.<init>(EntryChanges.java:72)
at com.octetstring.vde.replication.BackendChangeLog.getChange(BackendChangeLog.java:548)
at com.octetstring.vde.replication.Replicator.run(Replicator.java:180)
at com.octetstring.vde.replication.Replication.run(Replication.java:339)
>

---><Critical> <EmbeddedLDAP> <BEA-000000> <java.lang.ArrayIndexOutOfBoundsException:
Array index out of range: 260
at com.octetstring.vde.EntryChanges.readBytes(EntryChanges.java:279)
at com.octetstring.vde.EntryChanges.<init>(EntryChanges.java:72)
at com.octetstring.vde.replication.BackendChangeLog.getChange(BackendChangeLog.java:548)
at com.octetstring.vde.replication.Replicator.run(Replicator.java:180)
at com.octetstring.vde.replication.Replication.run(Replication.java:339)
>

---><Critical> <EmbeddedLDAP> <domain> <adminserver> <VDE Replication Thread> <<anonymous>> <> <> <1381951928629> <BEA-000000> <java.lang.NegativeArraySizeException
at com.octetstring.vde.EntryChanges.readBytes(EntryChanges.java:274)
at com.octetstring.vde.EntryChanges.<init>(EntryChanges.java:72)
at com.octetstring.vde.replication.BackendChangeLog.getChange(BackendChangeLog.java:548)
at com.octetstring.vde.replication.Replicator.run(Replicator.java:180)
at com.octetstring.vde.replication.Replication.run(Replication.java:339)
>

---> <Critical> <EmbeddedLDAP> <000000> <java.lang.ArrayIndexOutOfBoundsException: arraycopy
    at java.lang.System.arraycopy(Ljava/lang/Object;ILjava/lang/Object;II)V(Unknown Source)
    at com.octetstring.vde.Entry.readBytes([B)V(Entry.java:456)
    at com.octetstring.vde.Entry.<init>([B)V(Entry.java:98)

CAUSE:

All above exceptions indicate corruption in either or both the changelog.data and changelog.index files with the domain configured to use embedded LDAP with replication. The corruption can happen because of following reasons.

1.  While the admin server was writing an LDAP entry to the changelog, it was interrupted by a forced shutdown, which made the changelog partially update. 
2. When the admin server rebooted, it attempted to process the changelog (i.e., send the entries to the managed servers), but encountered the partially updated changelog.
3. The partial update was an entry that had been assigned a change number, but there was no data for the entry.
4. When the change log writer is interrupted between the index update and the data update and this update is in a synchronized method.

SOLUTION:

Step 1) Go to below mentioned directory

cd $DomainHome/servers/AdminServer/data/ldap/ldapfiles

Step 2)username@Hostname # ls

EmbeddedLDAP.data EmbeddedLDAP.index EmbeddedLDAP.tran EmbeddedLDAP.twpos changelog.index

EmbeddedLDAP.delete EmbeddedLDAP.lok EmbeddedLDAP.trpos changelog.data

Step 3) Take backup of changelog.data & changelog.index

mv changelog.data changelog.data_ORG

mv changelog.index changelog.inde_ORG

Step 4) Start your Weblogic Admin Server.

GENERIC SOLUTION:

Following is list of general action which we can perform in Test environment to fix the server startup problem -

Step 1) Try to delete all the .lok file from below location

$cd $DomainHome/servers/AdminServer/tmp

$ rm -rf AdminServer.lok

Step 2) Try to delete .DAT file from below location after taking proper backup

$ cd $DomainHome/servers/AdminServer/data/store/default

$ mv  _WLS_ADMINSERVER000000.DAT _WLS_ADMINSERVER000000.DAT_BKP

Step 3) Take a backup of "tmp" folder inside servers folder or  delete “tmp” folder.

$ cd $DomainHome/servers/AdminServer

$ mv tmp tmp_BKP

Step 4) Take a backup of "data" foler inside servers folder or delete “data” folder .

$ cd $DomainHome/servers/AdminServer

$ mv data data_BKP

Step 5) Take a backup of "cache" folder inside servers folder or delete "cache” folder .

$ cd $DomainHome/servers/AdminServer

$ mv cache cache_BKP

Note :

1) Upon startup above backed up or deleted files/folders shall be automatically created.

2) Please take backup of files/folder before deleting to be on safe side.

3) Above action plan also applies to Weblogic Manage Server

4) Its important to take backup of data folder as it holds critical business data.