OHS Webgate Error ( Due to unavailability of Access Server)

ISSUE:

A strange issue was observed in our OHS server. While starting OHS server below error was observed in putty session.

ERROR OBSERVED:

<<

Oblix: [ID 702911 user.emerg] 2013/04/08@16:33:35.28514 15656 53 ACCESS_GATE FATAL 0x00001520

/ade/aime_ngamac_934556/ngamac/src/palantir/webgate2/src/apache2entry_web_gate.cpp:591 "Exception thrown during WebGate initialization"

Message from syslogd@hostname at Mon Apr 8 19:33:35 2013 ...

hostname Oblix: [ID 702911 user.emerg] 2013/04/08@16:33:35.44396 15656 53 ACCESS_GATE FATAL 0x0000182C

/ade/aime_ngamac_934556/ngamac/src/palantir/webgate2/src/apache2entry_web_gate.cpp:592 "The AccessGate is unable to contact any Access Servers." raw_code^301

Message from syslogd@hostname at Mon Apr 8 19:33:58 2013 ...

hostname Oblix: [ID 702911 user.emerg] 2013/04/08@16:33:58.08668 15656 53 ACCESS_GATE FATAL 0x00001520

/ade/aime_ngamac_934556/ngamac/src/palantir/webgate2/src/apache2entry_web_gate.cpp:591 "Exception thrown during WebGate initialization"

Message from syslogd@hostname at Mon Apr 8 19:33:58 2013 ...

hostname Oblix: [ID 702911 user.emerg] 2013/04/08@16:33:58.31786 15656 53 ACCESS_GATE FATAL 0x0000182C

/ade/aime_ngamac_934556/ngamac/src/palantir/webgate2/src/apache2entry_web_gate.cpp:592 "The AccessGate is unable to contact any Access Servers."

raw_code^301

>> 

Note: Error was appearing on putty screen itself on start-up of OHS.

DESCRIPTION:

1) OHS Server : 11.1.1.7.0

2) OAM Server 11gR2 Integrated with OHS

3) Webgate 11.2.0.0 Integrated with OHS

When we were trying to start OHS server through putty session above error message was continuously observed in putty session(even Ctrl C didnt work)

SOLUTION:

1) As immediate measure we tried to take bounce of Solaris server.

Login as root user :

$init 6 ( This will restart your solaris server)

Note: Please take help of OS team before executing above command.

2) Once Solaris server was up and running,we again tried to start OHS through putty session but same above error was observed.

3) Then we check OAM server and to our surprise OAM server was down due to some error and hence when we were trying to start OHS server , it was unable to contact any configured ( integrated) OAM server (Access server ) as showed in logs.

4) We then restarted OAM server through Weblogic Console.

5) Once OAM server was up and running we started OHS server

$cd $OHS_HOME/instances/$INSTANCE_NAME/bin

$./opmnctl startall