Oracle SOA 11.1.1.7 startup issue after upgrade

In this post we shall discuss about issues observed after SOA upgrade to 11.1.1.7. We were unable to start SOA server from Weblogic Console.

ISSUE :

Error Observed:

<<

<Feb 23, 2015 6:09:28 AM GMT> <Info> <Management> <BEA-141107> <Version: WebLogic Server Temporary Patch for 19066738 Thu Jul 17 12:55:31 IST 2014
WebLogic Server Temporary Patch for BUG13717183 Mon Mar 19 14:12:18 IST 2012
WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050 >
<Feb 23, 2015 6:09:42 AM GMT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Feb 23, 2015 6:09:42 AM GMT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
Feb 23, 2015 6:09:50 AM oracle.security.jps.internal.common.util.XmlSchemaValidationUtil$StrictErrorHandler warning
WARNING: Invalid xml content was found. SchemaLocation: schemaLocation value = 'http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd' must have even number of URI's. Location: line 2 column 221.
<Feb 23, 2015 6:10:16 AM GMT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.AssertionError: java.lang.ClassNotFoundException: weblogic.jndi.internal.ServerNamingNode_1035_WLStub
java.lang.AssertionError: java.lang.ClassNotFoundException: weblogic.jndi.internal.ServerNamingNode_1035_WLStub
at weblogic.jndi.WLInitialContextFactoryDelegate.newRootNamingNodeStub(WLInitialContextFactoryDelegate.java:610)
at weblogic.jndi.WLInitialContextFactoryDelegate.newRemoteContext(WLInitialContextFactoryDelegate.java:577)
at weblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:482)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:376)
at weblogic.jndi.Environment.getContext(Environment.java:315)
Truncated. see log file for complete stacktrace
Caused By: java.lang.ClassNotFoundException: weblogic.jndi.internal.ServerNamingNode_1035_WLStub
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
Truncated. see log file for complete stacktrace
>
***************************************************************************
The WebLogic Server encountered a critical failure
Reason: Failed to load stub for class class weblogic.server.RemoteLifeCycleOperationsImpl
***************************************************************************
>>

Cause :

While upgrading Oracle Weblogic from 10.3.5 to 10.3.6, accidentally  Node manager was kept up and running and hence while starting SOA server from console(using Node manager) it was taking reference of old node manager(10.3.5) jar file and hence above errors were observed.

Solution :

Step 1) Stop or kill the node manager  PID.
 Eg:
 1) username@hostname # ps -ef | grep -i Node
username  5777     1   0   Feb 25 ?           0:00 /bin/sh /u02/app/oracle/product/fmw/wlserver_10.3/server/bin/startNodeManager.sh  
 2) Kill -9 5777

Step 2) Clean tmp,data,cache,and logs folder for  SOA server located at :
$MW_HOME/user_projects/domain/$DOMAIN_NAME/servers/WLS_SOA1

Eg: mv tmp tmp_BKP

      mv cache cache_BKP

      mv data data_BKP

      mv logs logs_BKP

Step 3) Start node manager process - On starting node manager it will take reference of new jar file of Weblogic Server 10.3.6.

Eg: $WL_HOME/server/bin

     nohup ./startNodeManager.sh &

Step 4) Start SOA server from console.

Note : Before starting with any upgrade (weblogic or SOA) make sure that node manager and all other services (Admin + Manage Server) are down.