Useful srvctl commands

Useful srvctl commands for ORACLE RAC MAINTENANCE

srvctl -help or srvctl -v

srvctl commands to startup database:

srvctl start database -d db_name [-o start_options] [-c connect_str|-q]

srvctl start database -d db_name [-o open]

srvctl start database -d db_name -o nomount

srvctl start database -d db_name -o mount

srvctl commands to shutdown database:

srvctl stop database -d db_name [-o stop_options] [-c connect_str|-q]

srvctl stop database -d db_name [-o normal]

srvctl stop database -d db_name -o transactional

srvctl stop database -d db_name -o immediate

srvctl stop database -d db_name -o abort

srvctl config database

srvctl config database -d db_name [-a] [-t]

srvctl start listener -n node_name [-l listener_names]

srvctl stop listener -n node_name [-l listener_names]

srvctl status listener [-n node_name] [-l listener_names] 

srvctl config listener -n node_name

What is Split Brain Syndrome in Oracle RAC?

Split brain syndrome occurs when the Oracle RAC nodes are unable to communicate with each other via private interconnect, but the communication between client and RAC node is maintained. This can cause data Integrity issues  when the same block is read or updated by two nodes and changes done from one node are overwritten by the other node because the block being changed is not locked.

When a node fails, the failed node is prevented from accessing all the shared disk devices and groups. This methodology is called I/O Fencing, Disk Fencing or Failure Fencing.

The node which first detects that one of the node is not accessible will evict that node from the RAC cluster group.This problem is solved by configuring the heartbeat connections through the same communication channels that are used to access the clients.

What causes Node eviction in Oracle RAC?

What is node eviction?

Node eviction in RAC is done when a heartbeat indicates that a node is not responding, 

and the evicted node is re-started to make it a part of cluster.

Causes for RAC node eviction:

Node eviction on Oracle RAC environment can be due to any of the below reasons. 

- A failure of any of the major hardware components (CPU, RAM, network interconnect).

- A server that is experiencing RAM swapping.

- When communications to the voting disk is interrupted, causing the disconnected node to be evicted and re-boot.

- Database or ASM hang condition.

Below is the list of important log files to review in case of a node eviction

- Clusterware alert log

- Database alert log

- CSSD agent logs

- CSSD monitor logs

- System Message logs (/var/log/messages)

Connecting to a Database from SQLDeveloper fails with IO Error: The Network Adapter Could Not Establish The Connection

Issue: New Users were trying to access a 3-node RAC Oracle Database system using SQLDeveloper

Below is the connection string provided to user for Oracle Database Connection. RAC Virtual IP is provided to connect to Database.


TESTDB01=
       (DESCRIPTION=
               (ADDRESS=(PROTOCOL=tcp)(HOST=test001-vip1.domain.com)(PORT=1521))
               (ADDRESS=(PROTOCOL=tcp)(HOST=test002-vip1.domain.com)(PORT=1521))
               (ADDRESS=(PROTOCOL=tcp)(HOST=test003-vip1.domain.com)(PORT=1521))
           (CONNECT_DATA=
               (SERVICE_NAME=TESTDB01)
               (INSTANCE_NAME=TESTDB01)
           )
       )


User tried to connect using one of the VIP and got the below error in SQLDeveloper.
Status : Failure -Test failed: IO Error: The Network Adapter Could Not Establish The Connection.

Analysis:

Verified the Oracle Database Services and Listener Services are running fine.
Other Users were able to connect to SQLDeveloper using the same Connection String.

Solution:

User does not have network communication open to port 1521 on Oracle RAC Virtual IP.

ping  test001-vip1.domain.com
ping  test002-vip1.domain.com
ping  test003-vip1.domain.com

telnet test001-vip1.domain.com 1521
telnet test002-vip1.domain.com 1521
telnet test003-vip1.domain.com 1521

Ping to VIP address works fine but Telnet to port 1521 on the VIP address is giving "Connect Failed" Error.

When Network communication is opened on port 1521 on all the Oracle RAC Virtual IP servers/Addresses, User can connect to Oracle Database using SQLDeveloper.

How to enable ArchiveLog mode in Oracle Database RAC Environment?

Steps to enable Archivelog mode in RAC environment:

The following steps need to be taken to enable archive logging in a RAC database environment:

1. Shutdown immediate all database instances
$ srvctl stop database -d <db_unique_name>

2. Startup database in mount mode
$ srvctl start database -d <db_unique_name> -o mount

3. Enable archive logging
$ sqlplus / as sysdba
sql> alter database archivelog;
sql> exit;

4. Stop database
$ srvctl stop database -d <db_unique_name>

5.Start all database instances
$ srvctl start database -d <db_unique_name>

6.Verify archiving is enabled/disabled 
sql> archive log list;