EmbeddedLDAP java.lang.ArrayIndexOutOfBoundsException in Admin Server

ISSUE:

Admin server fails during start up.

ERROR OBSERVED:

--><Error> <EmbeddedLDAP> <ADMIN01> <AdminServer> <VDE Replication Thread> <> <> <>
<1294659336925> <BEA-000000> <Error reading changelog entry#: 0>
<Critical> <EmbeddedLDAP> <ADMIN01> <AdminServer> <VDE Replication Thread> <> <> <>
<1294659336925> <BEA-000000> <java.lang.NullPointerException
at com.octetstring.vde.EntryChanges.readBytes(EntryChanges.java:279)
at com.octetstring.vde.EntryChanges.<init>(EntryChanges.java:72)
at com.octetstring.vde.replication.BackendChangeLog.getChange(BackendChangeLog.java:548)
at com.octetstring.vde.replication.Replicator.run(Replicator.java:180)
at com.octetstring.vde.replication.Replication.run(Replication.java:339)
>

---><Critical> <EmbeddedLDAP> <BEA-000000> <java.lang.ArrayIndexOutOfBoundsException:
Array index out of range: 260
at com.octetstring.vde.EntryChanges.readBytes(EntryChanges.java:279)
at com.octetstring.vde.EntryChanges.<init>(EntryChanges.java:72)
at com.octetstring.vde.replication.BackendChangeLog.getChange(BackendChangeLog.java:548)
at com.octetstring.vde.replication.Replicator.run(Replicator.java:180)
at com.octetstring.vde.replication.Replication.run(Replication.java:339)
>

---><Critical> <EmbeddedLDAP> <domain> <adminserver> <VDE Replication Thread> <<anonymous>> <> <> <1381951928629> <BEA-000000> <java.lang.NegativeArraySizeException
at com.octetstring.vde.EntryChanges.readBytes(EntryChanges.java:274)
at com.octetstring.vde.EntryChanges.<init>(EntryChanges.java:72)
at com.octetstring.vde.replication.BackendChangeLog.getChange(BackendChangeLog.java:548)
at com.octetstring.vde.replication.Replicator.run(Replicator.java:180)
at com.octetstring.vde.replication.Replication.run(Replication.java:339)
>

---> <Critical> <EmbeddedLDAP> <000000> <java.lang.ArrayIndexOutOfBoundsException: arraycopy
    at java.lang.System.arraycopy(Ljava/lang/Object;ILjava/lang/Object;II)V(Unknown Source)
    at com.octetstring.vde.Entry.readBytes([B)V(Entry.java:456)
    at com.octetstring.vde.Entry.<init>([B)V(Entry.java:98)

CAUSE:

All above exceptions indicate corruption in either or both the changelog.data and changelog.index files with the domain configured to use embedded LDAP with replication. The corruption can happen because of following reasons.

1.  While the admin server was writing an LDAP entry to the changelog, it was interrupted by a forced shutdown, which made the changelog partially update. 
2. When the admin server rebooted, it attempted to process the changelog (i.e., send the entries to the managed servers), but encountered the partially updated changelog.
3. The partial update was an entry that had been assigned a change number, but there was no data for the entry.
4. When the change log writer is interrupted between the index update and the data update and this update is in a synchronized method.

SOLUTION:

Step 1) Go to below mentioned directory

cd $DomainHome/servers/AdminServer/data/ldap/ldapfiles

Step 2)username@Hostname # ls

EmbeddedLDAP.data EmbeddedLDAP.index EmbeddedLDAP.tran EmbeddedLDAP.twpos changelog.index

EmbeddedLDAP.delete EmbeddedLDAP.lok EmbeddedLDAP.trpos changelog.data

Step 3) Take backup of changelog.data & changelog.index

mv changelog.data changelog.data_ORG

mv changelog.index changelog.inde_ORG

Step 4) Start your Weblogic Admin Server.

GENERIC SOLUTION:

Following is list of general action which we can perform in Test environment to fix the server startup problem -

Step 1) Try to delete all the .lok file from below location

$cd $DomainHome/servers/AdminServer/tmp

$ rm -rf AdminServer.lok

Step 2) Try to delete .DAT file from below location after taking proper backup

$ cd $DomainHome/servers/AdminServer/data/store/default

$ mv  _WLS_ADMINSERVER000000.DAT _WLS_ADMINSERVER000000.DAT_BKP

Step 3) Take a backup of "tmp" folder inside servers folder or  delete “tmp” folder.

$ cd $DomainHome/servers/AdminServer

$ mv tmp tmp_BKP

Step 4) Take a backup of "data" foler inside servers folder or delete “data” folder .

$ cd $DomainHome/servers/AdminServer

$ mv data data_BKP

Step 5) Take a backup of "cache" folder inside servers folder or delete "cache” folder .

$ cd $DomainHome/servers/AdminServer

$ mv cache cache_BKP

Note :

1) Upon startup above backed up or deleted files/folders shall be automatically created.

2) Please take backup of files/folder before deleting to be on safe side.

3) Above action plan also applies to Weblogic Manage Server

4) Its important to take backup of data folder as it holds critical business data.

 

SOA Server start up issue due to low on PermGen space

ISSUE:

SOA server health was impacted because of below error.

ERROR OBSERVED:

Logs : $MW_HOME/user_projects/domain/$DOMAIN_NAME/servers/soa_server/logs/soa_server.out

<<

java.lang.OutOfMemoryError: PermGen space

java.lang.OutOfMemoryError: PermGen space

javax.ejb.EJBException: EJB encountered System Exception: : java.lang.OutOfMemoryError: PermGen space

>>

SOLUTION:

1) Permanent generation of heap is used to store String pool and various Meta data required by JVM related to Class, method and other java primitives. 

2) You can easily run out of memory if you have too many classes or huge number of Strings in your project.

3) It doesn't depends on –Xmx value so no matter how big your total heap size you can ran OutOfMemory in perm space.

4) However we specify size of permanent generation using JVM options "-XX:PermSize" and "-XX:MaxPermSize" based on project need.

Step 1) Please check if it's feasible for you to increase -XX:PermSize/-XX:MaxPermSize parameters.

(We need to check whether enough free memory is available in server through top cmd)

Eg: $ top

load averages: 1.96, 1.70, 1.45; up 121+18:33:20 08:59:31
235 processes: 180 sleeping, 1 zombie, 50 stopped, 4 on cpu
CPU states: 97.9% idle, 1.1% user, 1.0% kernel, 0.0% iowait, 0.0% swap
Memory: 64G phys mem, 19G free mem, 48G total swap, 48G free swap

Step 2) In above example it shows that 19GB free memory is available and hence it would be possible to increase the value of -XX:PermSize=768 and -XX:MaxPermSize=1280 parameter as below:

Step 3) Login to Weblogic Administration Console.

Step 4) Click Lock & Edit

Step 5) Click Servers-->soa_server-->Configuration-->Server Start-->Arguments

Current: -Xms4096m -Xmx4096m -Xmn1024m -Xss1024k -XX:PermSize=512m -XX:MaxPermSize=1024m

New: -Xms4096m -Xmx4096m -Xmn1024m -Xss1024k -XX:PermSize=768m -XX:MaxPermSize=1280m

Note: It is always advisable to increase all memory parameter in multiples of 256m.

Note : In case of cluster setup repeat above step(1,2 and 5) for another instance of SOA server to maintain consistency.

Step 6) Click Save

Step 7) Click Activate changes

Step 8) Shut down SOA Managed Server from console

Step 9) Take backup of tmp and cache directory @ $MW_HOME/user_projects/domain/$DOMAIN_NAME/servers/soa_server

mv tmp tmp_BKP

mv cache_cache_BKP

Step 10) Start SOA Managed Server (Upon startup above folders will be automatically created)

Step 11) Retest the issue.

Ref:

JVM Memory Monitoring, Tuning, Garbage Collection, Out of Memory, and Heap Dump Analysis For SOA Suite Integration 11g (Doc ID 1358719.1)

Oracle HTTP Server (OHS) startup error

ISSUE:

Unable to start Oracle HTTP server.

$cd $OHS_HOME/instances/$INSTANCE_NAME/bin

$./opmnctl startall

opmnctl startall: starting opmn and all managed processes...
Response: 0 of 1 processes started.
Error
--> Process (index=1,uid=978846797,pid=14844)
failed to start a managed process after the maximum retry limit
Log:
$OHS_HOME/instances/$INSTANCE_NAME/diagnostics/logs/OHS/ohs1/console~OHS~1.log

ERROR OBSERVED:

Logs : @ $OHS_HOME/instances/$INSTANCE_NAME/diagnostics/logs/OHS/ohs1/ohs1.log

<<

2015-04-05T15:03:31.1435+12:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [core.c] [host_id: ] [host_addr: 192.168.8.9] [pid: 14844] [tid: 140174359107376] [user: fusion] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address 192.168.8.9:10621

[2015-04-05T15:03:31.1435+12:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [core.c] [host_id: ] [host_addr: 192.168.8.9] [pid: 14844] [tid: 140174359107376] [user: fusion] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address 192.168.8.9:10621

[2015-04-05T15:03:31.1435+12:00] [OHS] [INCIDENT_ERROR:20] [OHS-9999] [core.c] [host_id: [host_addr: 192.168.8.9] [pid: 14844] [tid: 140174359107376] [user: fusion] [VirtualHost: main] no listening sockets available, shutting down

>>

It seems the port 10621 is already in use by some other process and hence the error was observed.

SOLUTION:

Step 1) Find the process ID  which is using port 10621 using below script.

__________________________

#!/bin/bash

# is the port we are looking for

if [ $# -lt 1 ]

then

echo "Please provide a port number parameter for this script"

echo "e.g. %content 1521"

exit

fi

echo "We are checking , please be patient (CTRL+C breaks) . "

for i in `ls /proc`

do

pfiles $i | grep AF_INET | grep $1

if [ $? -eq 0 ]

then

echo Is owned by pid $i

echo ..

fi

done

______________________________

Step 2) Save above mentioned script as shell script(.sh)

Eg: find_pid_with_port.sh

Step 3) Execute script as below

$./find_pid_with_port.sh 10621 ( since we need to find process holding on port 10621)

O/P:

pfiles: permission denied: 6

pfiles: permission denied: 607

pfiles: permission denied: 862

sockname: AF_INET 192.168.8.9 port:10621

Is owned by pid 8856

Step 4) In Step 3 we found PID which is holding port 10621

So we can kill that process id(PID)

$kill -9 8856

Step 5) Try starting OHS again and it comes up with no issues.

$cd $OHS_HOME/instances/$INSTANCE_NAME/bin

$./opmnctl startall

Note: Same steps are applicable for any ports and 10621 is example port here.

Ref:

http://www.unixarena.com/2014/01/solaris-local-zone-how-to-find-out-what.html

http://onlineappsdba.com/index.php/2008/06/10/how-to-find-pid-listening-particular-port-on-linuxsolaris-10/

 

Persistent store "_WLS_AdminServer" could not be deployed in Weblogic Server

ISSUE:

The persistent store "_WLS_AdminServer" could not be deployed resulting in failure of Admin Server during startup.

ERROR OBSERVED:

Logs : $DOMAIN_NAME/servers/AdminServer/logs/AdminServer.out

<<

<Error> <Store> <BEA-280061> <The persistent store "_WLS_AdminServer" could not be deployed: weblogic.store.PersistentStoreException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.

weblogic.store.PersistentStoreException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.

at weblogic.store.io.file.Heap.open(Heap.java:312)

at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:413)

at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:404)

  at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)

Truncated. see log file for complete stacktrace

> <Critical> <WebLogicServer> <BEA-000362> <Server failed.>

>>

ROOT CAUSE:

1) Above error comes mostly when weblogic is not able to read the file store .DAT file. 

2) Also this file could be corrupted due to forceful shutdown (kill -9 pid)

SOLUTION:

Step 1) Go to below mentioned directory

cd $DomainHome/servers/AdminServer/data/store

Step 2) find . –name  *.DAT

cd $DomainHome/servers/AdminServer/data/store

username@hostname # ls

default diagnostics

username@hostname # find . .name *.DAT

.

./diagnostics

./diagnostics/WLS_DIAGNOSTICS000000.DAT

./default

./default/_WLS_ADMINSERVER000000.DAT

Step 3) Verify the file name in your result and error message should be same.

Step 4) Rename this file and move from this directory to some other directory.

cd $DomainHome/servers/AdminServer/data/store/default

username@hostname# ls

_WLS_ADMINSERVER000000.DAT

mv _WLS_ADMINSERVER000000.DAT _WLS_ADMINSERVER000000.DAT_ORG

Step 5) Find "EmbeddedLDAP.lok" and "AdminServer.lok" as well and remove the same.

cd $DomainHome/servers/AdminServer/data//ldap/ldapfiles

rm -rf EmbeddedLDAP.lok

cd $DomainHome/servers/AdminServer/tmp

rm -rf AdminServer.lok

Step 6) Check the port using netstat –an | grep <Weblogic server port>, there should not be any open connection to this port.

username@hostname # netstat -an | grep 7001

username@hostname #

Step 7) Start your Weblogic Admin Server.

Upon startup above files(EmbeddedLDAP.lok, AdminServer.lok and _WLS_ADMINSERVER000000.DAT) shall be auto created.

Note: .DAT file is very important file, and contains business data as well in Production system. Please take a backup of this file before doing any operation on it, so that later this file can be analyzed to complete those transaction.

Unable to obtain an exclusive lock to embedded LDAP data files directory in Weblogic

ISSUE:

Could not obtain an exclusive lock to the embedded LDAP data files directory and due to this Admin Server was failing during startup.

ERROR OBSERVED:

Logs : $DOMAIN_NAME/servers/AdminServer/logs/AdminServer.out

<<

<Error> <EmbeddedLDAP> <BEA-171519> <Could not obtain an exclusive lock to the embedded LDAP data files directory: /u04/app/product/fusion/user_projects/domains/product_domain/servers/AdminServer/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely shutdown and restart the server.>

<Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

<Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

<Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

>>

ROOT CAUSE:

Some time even after proper shutdown or forceful shutdown .lok file does not get removed automatically and when users try to restart the server again , since file is already present ,it fails to start the servers.

SOLUTION:

Step 1) Delete EmbeddedLDAP.lok file from below location

$cd DOMAIN_HOME/servers/AdminServer/data/ldap/ldapfiles

$ rm -rf EmbeddedLDAP.lok

Step 2) Restart Weblogic Admin Server and retest issue.

Oracle Webcenter Portal Upgrade from 11.1.1.5 to 11.1.1.8

As we know that Oracle Webcenter Portal 11.1.1.5 is now out of support by Oracle. So in this post we shall see up gradation of Oracle Webcenter Portal Suite from 11.1.1.5 to 11.1.1.8.

Pre-Upgrade Steps:

Step 1) Download binaries from my oracle support.

ofm_wc_generic_11.1.1.8.0_disk1_1of1.zip

Step 2) Upgrade Oracle Weblogic to 10.3.6

Step 3) Backup Oracle WC home folder , Inventory directory and database.

Step 4) Shutdown below services:

Admin Server , All Manage server and Node Manager.

Step 5) Unzip the patch 

Upgrade Oracle WC Home:

Step 1) Execute below command from Disk1.

Step 2) On Welcome Screen click Next.

Step 3) Select radio button Skip Software Updates and click Next

Step 4) Make sure it passes all per-requisites checks.Once done click Next.

 

Step 5) In the Specify Installation Location screen make sure the Oracle Middleware Home and Oracle Home Directory are correct for the Web center portal

Oracle Middleware Home=$MW_HOME
Oracle Home Directory=Oracle_WC1(Default name)

 

Step 6) Click Yes to confirm with Upgrade.

 

Step 7) In the Application Server screen the Weblogic Server radio button should be checked, click the Next button.

 

Step 8)  In the Installation Summary screen click the Install button.

Step 9)  When the Progress is 100% click the Next button.

 

Step 10) Click the Finish button

Upgrade Webcenter Portal Schema:

Step 1) Execute ./psa from $ORACLE_WC1/bin in GUI mode.

Step 2) Click Next.

 

Step 3) In Select Component field select the schema that you wish to upgrade . 

 

Step 4) Make sure per-requisites are completed and click Next

 

Step 5) Enter schema details for 'XXX_WEBCENTER' user

Connect String: DBHostaName:DBPort/DBName
DBA User Name: sys as sysdba
DBA Password : Password for sysdba
Schema User Name : Dropdown list and select 'XXX_WEBCENTER' user
Schema Password : Password of 'XXX_WEBCENTER' user

 

Step 6) Repeat Step 5 for 'XXX_MDS' , 'XXX_DISCUSSIONS' , 'XXX_DISCUSSION_CRAWLER' and 'XXX_ACTIVITIES' schema. ( Only if you have them in your environment)

Step 7)  Click Next once succeeded.

 

Step 8) Confirm everything and click Upgrade.

Step 9) Check status and once 100% completed,click Done.

 

Step 10) Login to database and confirm that 'XXX_WEBCENTER' , 'XXX__MDS' , 'XXX_DISCUSSIONS' , 'XXX_DISCUSSION_CRAWLER' and 'XXX_ACTIVITIES' schema has been upgraded to 11.1.1.8 with below command:

"Select owner,version,status,upgraded from schema_version_registry"

XXX_ACTIVITIES 11.1.1.8.0 VALID Y

XXX_DISCUSSIONS 11.1.1.8.0 VALID Y

XXX_DISCUSSIONS 11.1.1.8.0 VALID Y

XXX_DISCUSSIONS_CRAWLER 11.1.1.8.0 VALID Y

XXX_DISCUSSIONS_CRAWLER 11.1.1.8.0 VALID Y