Friday, December 30, 2016

What is OCR (Oracle Cluster Registry)?

Oracle Cluster Registry

OCR is the central repository for the Oracle cluster.
OCR has information about Port information,node failures, node reconfiguration,Database Status,Listener Status,Instance status, Network Status in real time.


Default Location of ocr.loc file on linux --  /etc/oracle/ocr.loc

Oracle Clusterware reads ocr.loc to determine the registry information and application resources that need to started up on each cluster node.

The below processes update OCR with realtime information.

1. CRSd updates  OCR with  information about node failure 
2. CSSd updates the OCR when a node is added or deleted
3. NetCA, DBCA, SRVCTL updates  OCR with  services related information

OCR also has information about  Software active version

Software  version can also be found using below command
$ crsctl query crs activeversion



OCR Backup and Restore



OCR is a binary file and cannot be edited manually.By Default, OCR is backed up automatically every 4 hours (from Instance Startup time).
OCR backups are cached across all nodes of a cluster.However,Only Master RAC node can write to OCR.
OCR backups are overwritten automatically. Clusterware maintains last 3 backups of OCR. 

Backup location can be defined using the below command.

$ocrconfig -backuploc <backup location>

It is recommended to use shared location for storing OCR backups.

To view the current OCR backups available, use the below commands

$ocrconfig –showbackup auto 

$ocrconfig –showbackup manual


After any clusterware related changes, OCR backup can be taken manually.

$ocrconfig –manualbackup


If OCR file is Corrupted or lost, CRS Instance may crash.

To restore OCR from Backup,follow the procedure below


1.Using ocrconfig identify the latest OCR backup using command below

$ocrconfig -showbackup
2.Shutdown CRS Services on all RAC nodes
$crsctl stop crs
  
3.Start CRS on one node in exclusive mode
$crsctl start crs -excl 
Stop crsd , if it is running

$ crsctl stop resource ora.crsd -init

4.Restore the OCR 
$ocrconfig –restore <backuplocation/backupfilename>
   
5. Verify the integrity of OCR
$ocrcheck
6. Shutdown  CRS Services on the node where you had started in exclusive mode
$crsctl stop crs 


7.Start CRS on all RAC nodes 
$crsctl start crs
8. Verify the restored OCR 
$cluvfy comp ocr –n all -verbose

Weblogic Managed Servers Crashed due to weblogic.socket.MaxMessageSizeExceededException

ISSUE
=====

Weblogic Managed Servers are crashing frequently .

Admin Server Logfile has the below error message.

 <IOException occurred on socket: Socket[addr=test1.localhost.com/180.81.41.219,port=1345,localport=47093]
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'.
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'
at weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMuxableSocket.java:230)
at weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:351)
at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:989)
at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:920)
at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:897)
at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:215)
at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:177)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) 

Due to above error, Admin Server is going into WARNING Status and Managed Servers are disconnected from ADMIN Server.


SOLUTION
========

1. Increase Maxmessage size from WLS console


Login to weblogic console url as weblogic admin user


WebLogic Console -> Servers ->Admin Server ->  Protocols -> General -> Max Message 
Size

Increase Max Message Size to  20000000 (Any value higher than Incoming message size from error message)

2.Restart MT Services