Friday, 30 December 2016

What is OCR (Oracle Cluster Registry)?

Oracle Cluster Registry

OCR is the central repository for the Oracle cluster.
OCR has information about Port information,node failures, node reconfiguration,Database Status,Listener Status,Instance status, Network Status in real time.


Default Location of ocr.loc file on linux --  /etc/oracle/ocr.loc

Oracle Clusterware reads ocr.loc to determine the registry information and application resources that need to started up on each cluster node.

The below processes update OCR with realtime information.

1. CRSd updates  OCR with  information about node failure 
2. CSSd updates the OCR when a node is added or deleted
3. NetCA, DBCA, SRVCTL updates  OCR with  services related information

OCR also has information about  Software active version

Software  version can also be found using below command
$ crsctl query crs activeversion



OCR Backup and Restore



OCR is a binary file and cannot be edited manually.By Default, OCR is backed up automatically every 4 hours (from Instance Startup time).
OCR backups are cached across all nodes of a cluster.However,Only Master RAC node can write to OCR.
OCR backups are overwritten automatically. Clusterware maintains last 3 backups of OCR. 

Backup location can be defined using the below command.

$ocrconfig -backuploc <backup location>

It is recommended to use shared location for storing OCR backups.

To view the current OCR backups available, use the below commands

$ocrconfig –showbackup auto 

$ocrconfig –showbackup manual


After any clusterware related changes, OCR backup can be taken manually.

$ocrconfig –manualbackup


If OCR file is Corrupted or lost, CRS Instance may crash.

To restore OCR from Backup,follow the procedure below


1.Using ocrconfig identify the latest OCR backup using command below

$ocrconfig -showbackup
2.Shutdown CRS Services on all RAC nodes
$crsctl stop crs
  
3.Start CRS on one node in exclusive mode
$crsctl start crs -excl 
Stop crsd , if it is running

$ crsctl stop resource ora.crsd -init

4.Restore the OCR 
$ocrconfig –restore <backuplocation/backupfilename>
   
5. Verify the integrity of OCR
$ocrcheck
6. Shutdown  CRS Services on the node where you had started in exclusive mode
$crsctl stop crs 


7.Start CRS on all RAC nodes 
$crsctl start crs
8. Verify the restored OCR 
$cluvfy comp ocr –n all -verbose

Weblogic Managed Servers Crashed due to weblogic.socket.MaxMessageSizeExceededException

ISSUE
=====

Weblogic Managed Servers are crashing frequently .

Admin Server Logfile has the below error message.

 <IOException occurred on socket: Socket[addr=test1.localhost.com/180.81.41.219,port=1345,localport=47093]
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'.
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'
at weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMuxableSocket.java:230)
at weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:351)
at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:989)
at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:920)
at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:897)
at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:215)
at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:177)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) 

Due to above error, Admin Server is going into WARNING Status and Managed Servers are disconnected from ADMIN Server.


SOLUTION
========

1. Increase Maxmessage size from WLS console


Login to weblogic console url as weblogic admin user


WebLogic Console -> Servers ->Admin Server ->  Protocols -> General -> Max Message 
Size

Increase Max Message Size to  20000000 (Any value higher than Incoming message size from error message)

2.Restart MT Services

Tuesday, 23 August 2016

How to delete Managed Servers in R12.2

The procedure to increase/decrease the number of oacore/oafm servers in R12.2.X is different from R12.1.x.
Because in R12.2.x OPMN is replaced by Weblogic Server.


Steps to delete a managed Server in R12.2.x

1. Ensure there is no active adop cycle.

2. Shut down the managed server which needs to be deleted from the cluster/domain.

 $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop <MANAGED SERVER NAME>

Example:

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server5


3. Execute the script adProvisionEBS.pl  on the host where the managed server was created.

Source the RUN filesystem before running the command.

 perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-delete-managedserver \
-contextfile=<CONTEXT_FILE> -managedsrvname=<MANAGED_SERVER_NAME> \
-servicetype=<SERVICE_TYPE> -logfile=<LOGFILE>

Sample Output:

bash-4.1$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
> ebs-delete-managedserver \
> -contextfile=$CONTEXT_FILE -managedsrvname=oacore_server6 \
> -servicetype=oacore -logfile=$APPLRGF/TXK/delMS_oacoreserver6.log

Enter the APPS Schema password:
Enter the WebLogic AdminServer password:

ManagedServer oacore_server5 deleted successfully.


The above command deletes the managed server and also updates related parameters in context file


To Verify the entries are deleted in Context File,check the output of below command

grep -i oacore_server6 $CONTEXT_FILE


4.Execute the following command to delete details of the managed server from the OHS configuration files mod_wl_ohs.conf and apps.conf


$ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=removeMS \
-oacore=<host>.<domain>:<port> \
-oafm=<host>.<domain>:<port> \
-forms=<host>.<domain>:<port> \
-formsc4ws=<host>.<domain>:<port> \
-ekanban=<host>.<domain>:<port> \
-accessgate=<host>.<domain>:<port> \
-yms=<host>.<domain>:<port>




Sample Output:

bash-4.1$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
 -contextfile=$CONTEXT_FILE \
-configoption=removeMS -oacore=localhost.localdomain.com:7205


*** LOG FILE:
/test12/applmgr/fs1/inst/apps/test12_localhost/logs/appl/rgf/TXK/txkSetAppsConf_08009354.log


5. Restart HTTP Server

cd $ADMIN_SCRIPTS_HOME

adapcctl.sh stop

adapcctl.sh start


6. Repeat the above steps on PATCH filesystem.

adpreclone.pl error while running copyBinary.sh on a 12.2 Environment

In a 12.2.5 environment adpreclone.pl errors out while running copyBinary.sh adpreclone.pl on appsTier



$perl adpreclone.pl appsTier

                     Copyright (c) 2011, 2014 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adpreclone Version 120.31.12020000.20

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

 Checking the status of the Oracle WebLogic Administration Server....

 Running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:
perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress



 Setting the wls environment

Beginning application tier Stage - Thu Aug 18 02:43:26 2016

/test12/applmgr/fs1/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /test12/applmgr/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/test12/applmgr/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/test12/applmgr/fs1/EBSapps/comn/java/classes:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/share.jar:/test12/applmgr/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test12/applmgr/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/test12/applmgr/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test12/applmgr/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -stage /test12/applmgr/fs1/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg

Log file located at /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/StageAppsTier_08180243.log

  \     20% completed
ERROR while running Stage...
Thu Aug 18 02:50:43 2016

ERROR while running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress ...
Thu Aug 18 02:50:43 2016
[aptest12@vmohswrly128 <RUN> ]


adpreclone log files shows the below error message

Error Message:

START: Creating FMW archive.
Running /test12/applmgr/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /test12/applmgr/fs1/EBSapps/comn/clone/FMW/t2pjdk -al /test12/applmgr/fs1/EBSapps/comn/clone/FMW/FMW_Home.jar -smw /test12/applmgr/fs1/FMW_Home -ldl /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
Script Executed in 298222 milliseconds, returning status 1
ERROR: Script failed, exit code 1

Solution:

1. set the environment variable T2P_JAVA_OPTIONS as given below

$export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/test12/applmgr/temp"

2. Make sure enough free space is available in the temp directory provided above

3.Rerun adpreclone.pl on appsTier



Tuesday, 5 July 2016

ADOP Prepare Phase Fails with ORA-31011: XML parsing failed

While applying Patch on a test instance (R12.2.5), Prepare phase failed with error ORA-31011: XML parsing failed

Complete error message from adop logfile
==================================

[aptest12@localhost <RUN> ]$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Initializing.
    Run Edition context  : /test12/applmgr/fs2/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch edition context: /test12/applmgr/fs1/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch file system free space: 1130.34 GB

Validating system setup.
    [ERROR]     Failed to execute SQL statement:
select node_name from FND_OAM_CONTEXT_FILES
where NAME not in ('TEMPLATE','METADATA','config.txt') and
CTX_TYPE='A' and (status is null or upper(status) in ('S','F'))
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'run'
and EXTRACTVALUE(XMLType(TEXT),'//oa_service_group_status[@oa_var=''s_web_admin_status'']')='enabled'
and EXTRACTVALUE(XMLType(TEXT),'//oa_service_list/oa_service[@type=''admin_server'']/oa_service_status')='enabled'

    [ERROR]     Error Message:
    [ERROR]     ORA-31011: XML parsing failed
    [ERROR]     ORA-19202: Error occurred in XML processing
    [ERROR]     LPX-00229: input source is empty
    [ERROR]     ORA-06512: at "SYS.XMLTYPE", line 272
    [ERROR]     ORA-06512: at line 1 (DBD ERROR: error possibly near <*> indicator at char 208 in '    select node_name from FND_OAM_CONTEXT_FILES
    [ERROR]         where NAME not in ('TEMPLATE','METADATA','config.txt') and
    [ERROR]             CTX_TYPE='A' and (status is null or upper(status) in ('S','F'))
    [ERROR]             and EXTRACTVALUE(<*>XMLType(TEXT),'//file_edition_type') = 'run'
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),'//oa_service_group_status[@oa_var=''s_web_admin_status'']')='enabled'
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),'//oa_service_list/oa_service[@type=''admin_server'']/oa_service_status')='enabled'
    [ERROR]     ')
    [UNEXPECTED]Error occurred identifying Admin server node


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


Cause:

fnd_oam_context_files table has incorrect information  or the table is corrupted


Solution:


1. Login to sqlplus as applsys and take backup of fnd_oam_context_files table.


SQL> conn applsys
Enter password:
Connected.
SQL> create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;

Table created.

2. Truncate Table fnd_oam_context_files

SQL>  truncate table fnd_oam_context_files;

Table truncated.


3. Bring down MT Services

4. Run Autoconfig to populate fnd_oam_context_files  with right information.


Sunday, 12 June 2016

Adpatch Worker Fails with "ORA-01031: insufficient privileges"

While Applying Patch on Customer environment, adpatch failed with error

Worker log has the below error.
=====================
GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

AD Worker error:
The following ORACLE error:

ORA-01031: insufficient privileges


occurred while executing the SQL statement:

GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

Error occurred in file

/test12/applmgr/1200/ad/12.0.0/patch/115/sql/ademusr.sql

with arguments '&systempwd &un_fnd &pw_fnd &un_apps &pw_apps'.

==============


Solution
=========

1.Login to oracle database server as oracle user

2. Connect to sqlplus as sysdba and run the failed sql statement
sqlplus / as sysdba

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role
  2  ;

Grant succeeded.

SQL>


3. Use adctrl to skip the failed worker (Option 8) and continue with Patching

Weblogic Admin Server Startup Fails with "weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid"

After Changing Weblogic Password in boot.properties, Admin Server Fails to Startup.
Below error is reported in Admin Server Logfile.

<May 10, 2017 10:10:18 AM CDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.> 


Solution
=======

1.Shutdown Admin Server and Nodemanager
2.Go to $DOMAIN_HOME/servers/Admin Server>/data
3.Delete the boot.properties
4.Delete the whole ldap folder
5.Start the node-manager
6.Start Admin Server

Monday, 23 May 2016

"lsnrctl: symbol lookup error: lsnrctl: undefined symbol: nlgh" while starting Oracle Database listener

After Applying PSU, database listener failed to start with below error

 $ lsnrctl start testdb
lsnrctl: symbol lookup error: lsnrctl: undefined symbol: nlgh


Solution
==========
1.Relink network executables using below command

$  make -kf ins_net_server.mk install
/bin/chmod 755 /testdb/oracle/product/11204/bin/
 - Linking tnslsnr
rm -f tnslsnr
gcc -o tnslsnr -m64 -z noexecstack -L/testdb/oracle/product/11204/network/lib/ -L/testdb/oracle/product/11204/lib/ -L/testdb/oracle/product/11204/lib/stubs/  /testdb/oracle/product/11204/network/lib/s0nsgl.o /testdb/oracle/product/11204/network/lib/snsglp.o -lclntsh  `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lztkg11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11   `cat /testdb/oracle/product/11204/lib/sysliblist` -Wl,-rpath,/testdb/oracle/product/11204/lib -lm    `cat /testdb/oracle/product/11204/lib/sysliblist` -ldl -lm   -L/testdb/oracle/product/11204/lib -lons -lnl11  -ln11 -lnlsnr11 -lnlsnrc11 -lnsgr11 -lnms -lncm11 -lnmsp -lpeer -lnro11 -ln11   -lnl11 -lnlsnr11 -lnlsnrc11 -lnms -lncm11 -lnmsp -ln11     -lm
mv -f /testdb/oracle/product/11204/bin/tnslsnr /testdb/oracle/product/11204/bin/tnslsnr0
mv tnslsnr /testdb/oracle/product/11204/bin/tnslsnr
/bin/chmod 751 /testdb/oracle/product/11204/bin/tnslsnr
 - Linking lsnrctl
rm -f lsnrctl
gcc -o lsnrctl -m64 -z noexecstack -L/testdb/oracle/product/11204/network/lib/ -L/testdb/oracle/product/11204/lib/ -L/testdb/oracle/product/11204/lib/stubs/  /testdb/oracle/product/11204/network/lib/s0nsglc.o -lclntsh  `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lztkg11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11   `cat /testdb/oracle/product/11204/lib/sysliblist` -Wl,-rpath,/testdb/oracle/product/11204/lib -lm    `cat /testdb/oracle/product/11204/lib/sysliblist` -ldl -lm   -L/testdb/oracle/product/11204/lib -lnl11  -ln11 -lnlsnr11 -lnlsnrc11 -lnsgr11 -lnms -lncm11 -lnmsp -lpeer -lnro11 -ln11   -lnl11 -lnlsnr11 -lnlsnrc11 -lnms -lncm11 -lnmsp -ln11     -lm
mv -f /testdb/oracle/product/11204/bin/lsnrctl /testdb/oracle/product/11204/bin/lsnrctl0
mv lsnrctl /testdb/oracle/product/11204/bin/lsnrctl
/bin/chmod 751 /testdb/oracle/product/11204/bin/lsnrctl


2. Restart Database Listener

$ lsnrctl start testdb

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 16-MAY-2016 02:00:20

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Starting /testdb/oracle/product/11204/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.4.0 - Production
System parameter file is /testdb/oracle/product/11204/network/admin/testdb/listener.ora
Log messages written to /testdb/log/network/testdb.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dblocalhost)(PORT=1523)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dblocalhost)(PORT=1523)))
STATUS of the LISTENER
------------------------
Alias                     testdb
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                16-MAY-2016 02:00:20
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /testdb/oracle/product/11204/network/admin/testdb/listener.ora
Listener Log File         /testdb/log/network/testdb.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dblocalhost)(PORT=1523)))
Services Summary...
Service "testdb" has 1 instance(s).
  Instance "testdb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

What Happens during adop cutover Phase?

Cutover Phase Requires a brief downtime.

Following activities occur during Cutover:

+Users are logged off the system
+Finalize is run as part of cutover -if not run earlier in adop cycle
+Cutover phase cancels ADZPPATCH Concurrent Program
+Stops all Services on RUN and PATCH editions , switches filesystem and starts services on new RUN FS
+Flipping snapshots in run and patch editions.
+Users are brought back online on the patched system 

Environment is changed after cutover. Need to Re-source the env, for performing any actions after cutover.

Syntax:

$ adop phase=cutover

Multi-node Considerations
========================
By default cutover is invoked automatically on remote nodes.
If you want cutover individually:      
1. First node the cutover should be:           
$ adop phase=cutover allnodes=no       
2. For all other nodes you should do:        
$ adop phase=cutover allnodes=no action=nodb           
 * nodb because db work is already done


Cutover Parameters:
===================

mtrestart :Specifies whether or not to start Mid-Tier services at end of cutover, it allows you to perform some additional maintenance operations in the system before bringing it up.        - Takes values "yes" or "no".        - Default: yes
allnodes : Specifies whether or not to run on all the nodes.        - Takes values "yes" or "no".        - Default: yes
action: Specifies whether or not to take database action        - Takes values "db" or "nodb"        - Default: db

Monday, 9 May 2016

What is the difference between dbtechstack,dbtier and dbconfig modes in Cloning?

While Cloning Oracle Applications, adcgclone can be executed in 3 Modes on Database node.


1)perl adcfgclone.pl dbTier

 It configures the ORACLE_HOME on the target database node and  recreate the controlfiles.
 This is used in case of standby database/hot backups.   


2)perl adcfgclone.pl dbTechStack 

It configures the ORACLE_HOME on the target database tier node only. Relink the oracle home.

The below steps has to be performed manually
1. Create the Target Database control files.
2. Start the Target System Database in open mode
3. Run the library update script against the Database
cd $RDBMS_ORACLE_HOME/appsutil/install/[CONTEXT NAME]
sqlplus "/ as sysdba" @adupdlib.sql [libext]
 Where [libext] should be set to 'sl' for HP-UX, 'so' for any other UNIX platform, 
or 'dll' for Windows.

3)perl adcfgclone.pl dbconfig

It configures the database with context file. Database must be open while running adcfgclone in dbconfig mode.
e.g 
cd $RDBMS_ORACLE_HOME/appsutil/clone/bin
perl adcfgclone.pl dbconfig <Context_file>

Where Target Context File is:
$RDBMS_ORACLE_HOME/appsutil/<SID_HOSTNAME>.xml

Sunday, 10 April 2016

What are the daily activities of Oracle Applications DBA?


  1. Maintaining Oracle Application File System and Database using AD Utilities.
  2. Starting, stopping and troubleshooting issues related to Application server components.
  3. Using adadmin to relink executables, generate forms/reports,generate jar files, compile/validate APPS schema.
  4. Analyzing patch pre-requisites and Applying the Patches using AD Patch Utility on Application file System and Technology Stack.
  5. Applying Mini Packs, Family Packs, Maintenance Pack and Rollup patches.
  6. Applying database patches using opatch utility and troubleshooting issues related to OPatch.
  7. System administration activities like creating users, assigning Responsibilities, defining custom managers.
  8. Changing configuration files as per the requirement.
  9. Changing APPS password and Application database users’ password for security.
  10. Performing Healthchecks.
  11. Splicing off-cycle products using adsplice utility.
  12. Cloning of Oracle Applications 11i from Production to Test Instance.
  13. Taking backups- physical, logical, RMAN and performing Restore, Recovery.
  14. Troubleshooting issues related to concurrent managers.
  15. Managing System Configuration using Autoconfig.
  16. Modifying initialization parameters as per the requirement-PFILE AND SPFILE.
  17. Reorganization of tables that are fragmented.
  18. Creating users and assigning appropriate roles, privileges.
  19. Managing tablespaces and data files.
  20. Monitoring alert log file.
  21. Troubleshooting ORA- errors.
  22. Analyzing AWR reports.


Thursday, 7 April 2016

How to Recover lost APPS Password in Oracle Applications R12?



Have you ever forgot or lost apps password in R12. In Oracle Applications 11i, It is easy to recover apps password as it is stored in application files appsweb.cfg and wdbsvr.app.
Apps password is not available in the application files in R12. We can still recover the lost password by creating a decrypt function.

Steps to retreive apps password in R12:

1.Create function Using below syntax

SQL> create or replace FUNCTION apps.decrypt_pin_func(in_chr_key IN VARCHAR2,in_chr_encrypted_pin IN VARCHAR2) RETURN VARCHAR2
AS
LANGUAGE JAVA NAME ‘oracle.apps.fnd.security.WebSessionManagerProc.decrypt(java.lang.String,java.lang.String) return java.lang.String’;
4
5 /

Function created.


2. Find the encrypted password using below query

SQL> select ENCRYPTED_FOUNDATION_PASSWORD from apps.fnd_user where USER_NAME=’GUEST’;

ENCRYPTED_FOUNDATION_PASSWORD
——————————————————————————–
JH89E045BEA551FD01A8D71029ACA879F209U8BD898F451EBDB987C818E7608CE0CC0167BDCAF9D1D04D9C9CEE418CFE615A


3.Run the below query to decrypt the password


SQL> SELECT apps.decrypt_pin_func(‘GUEST/ORACLE’,’JH89E045BEA551FD01A8D71029ACA879F209U8BD898F451EBDB987C818E7608CE0CC0167BDCAF9D1D04D9C9CEE418CFE615A') from dual;

APPS.DECRYPT_PIN_FUNC(‘GUEST/ORACLE’,’ZG761B2FB6A2C49BC802F57E55AC26637388B4EA99
——————————————————————————–
PASSWORD


4. Verify Database Connection using the password retreived


SQL> conn apps/PASSWORD
Connected.

How to enable Trace for Application User in Oracle Applications 11i/R12?


Enabling Tracing for a Application User makes diagnostics easier to analyze.Since only the sql operations performed by a specific User are Traced.


Steps to Enable Trace for Application User:

1.Login to Oracle Applications and select the System Administrator responsibility. 
Navigate to  Profile -->  System 

2.Search for Profile Option 'Initialization SQL Statement - Custom' .Also Specify the username whom you wish to enable tracing during Search.

3.
Under username column Update the below line

BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'USER_TRACE_IDENTIFER' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;

Note : Level cane be 4,8,12 depends on your requirement 


Save the Changes done in Profile Option

4. Be careful while updating the Profile Option, Any incorrect Syntax may Prevent user from Logging in

5. Inform User to Re-login and Perform the Activity which needs to be traced.

6. Trace Files can be found in the directory specified by "user_dump_directory" parameter on the database server.
Use the identifier given in Profile Option to search the trace file in udump directory.

7. After the issue is reproduced and required trace files are collected for analysis, disable Tracing from Front End.
Its recommended to disable the trace immediately after the diagnostics are collected, as it will generate huge logfiles.

8. Navigate to system=> Profile form under System Administrator Responsibility.

Query for the username  and Profile Option name "Initialization SQL Statement - Custom"

Remove the values and make it blank.

Save the changes.


Thursday, 24 March 2016

Discoverer Login Fails with ORA-1017

Logging to discoverer url as sysadmin gives the following error.

ORA-1017: Invalid username/password
A connection error.  
- Oracle BI Discoverer is unable to authenticate using the password provided. This can happen due to an invalid password or because the password was lost while using back, forward, or refresh in your browser. Enter the password again to continue.
- Failed to connect to database - Unable to connect to Oracle Applications database (afscpgcs) 

Cause:

APPL_SERVER_ID parameter in .dbc file of Applications MT node(applmgr) and discoverer user is not matched


$diff test1_disco.dbc test1_mt.dbc

4c4
< APPL_SERVER_ID=178***********
---
> APPL_SERVER_ID=1A***********



Solution:

Copy the <SID>.dbc from $FND_TOP/secure or $FND_SECURE of Applications Middle tier to Disoverer user

Bounce Discoverer Services

Oracle RMAN Inremental Backup Algorithm and Change Tracking File

During an incremental backup, RMAN reads the SCN of each data block in the input file and compares it to the checkpoint SCN
of the parent incremental backup. If the SCN in the input data block is greater than or equal to the checkpoint SCN of the
parent, then RMAN copies the block.


Performance of incremental Backup can be improved by enabling block change tracking.When Change tracking is enabled, all the changes to datablocks are recored in change tracking file. RMAN can read the changed SCN's from change tracking file, instead of reading each and every input file.
One change tracking file is created for the entire database.
By default, the change tracking file is created as an Oracle managed file in DB_CREATE_FILE_DEST. We can specify the name of the block change tracking file by providing the location you choose.


SQL command to enable block change tracking

SQL> ALTER DATABASE ENABLE BLOCK CHANGE TRACKING;
This creates change tracking file in "db_create_file_dest" directory


SQL> ALTER DATABASE ENABLE BLOCK CHANGE TRACKING  USING FILE '/test1/data/rman_change_track.f' REUSE;
This creates the change tracking file in the location specified.


SQL command to disable block change tracking

SQL> ALTER DATABASE DISABLE BLOCK CHANGE TRACKING;
By Default,  block change tracking is disabled.
If the change tracking feature is disabled, any existing change tracking files will be deleted


SQL query to find change tracking file Details

select filename, status from v$block_change_tracking


How to Relocate the change tracking file


1.Shut down the database

SHUTDOWN IMMEDIATE

2.Using OS commands, move the change tracking file to a new location.

3.Mount the database and move the change tracking file to a location that has more space.

SQL query to rename change tracking file

ALTER DATABASE RENAME FILE   '/test1/data/rman_change_track.f' TO '/newtest1/data1/change_trk.f';

4.Open the database:

ALTER DATABASE OPEN;


If we cannot afford downtime for database shutdown, change tracking file can be relocated as below


ALTER DATABASE DISABLE BLOCK CHANGE TRACKING;
ALTER DATABASE ENABLE BLOCK CHANGE TRACKING USING FILE 'new-location';

Contents of change tracking file will be lost if we use the above method.So its not a preferable method.
RMAN will have to scan the entire file until the next level 0 incremental backup is done

Advantages of RMAN Incremental Backups

What is RMAN Incremental Backup?

Incremental Backups take  back up  of only datafile blocks that have changed after a previous full backup. 
Incremental backups are applicable for databases, individual tablespaces or datafiles.


Benefits of RMAN Incremental Backup:

1.RMAN Incremental backups are used to periodically roll forward an image copy of the database

2.Reduced amount of time needed for Backups

3.Less Network bandwidth is required

4.To get adequate backup performance when the aggregate tape bandwidth available for tape write I/Os is much less than the aggregate disk bandwidth for disk read I/Os

5.To  recover changes to objects created with the NOLOGGING option. 

6.To reduce backup sizes for NOARCHIVELOG databases. Instead of making a whole database backup every time, you can make incremental backups.

Note:

If the database is in ARCHIVELOG mode,  incremental backups can be taken if the database is open;
If the database is in NOARCHIVELOG mode, then we can only make incremental backups after a consistent shutdown.


Click Here to know more about how incremental backup is implemented by RMAN and the imporatnace of block change tracking feature.

Tuesday, 22 March 2016

Thursday, 17 March 2016

How to Perform Clusterware Compatibility Testing in Oracle RAC

+ Compatiblity Testing can be done using Oracle Certification Environment (OCE) kit

Installing the Oracle Certification Environment Software for Oracle RAC

The OCE Certification Kit required to certify the system for Oracle RAC 11g Release 1 (11.1) is available for download only. The Single Instance certification tests should be completed prior to installing the OCE kit for Oracle RAC. Refer to the previous section if necessary. Once Single Instance testing has successfully completed, the single instance OCE installations must be archived to allow OCE installations for Oracle RAC to succeed. The OCE kit for Oracle RAC should be installed separately on each node of the cluster. If the ORACLE_HOME is located on a shared disk, multiple installations of OCE will not be possible. In that case, it will not be possible to run OCE tests simultaneously, and the time required to complete certification will be greatly increased. To install the OCE Kit:
1.      Download OCE for Oracle RAC 11g Release 1 (11.1) archive to a suitable location, such as /tmp/oce. The OCE archives are either CPIO archives, or compressed CPIO archives.
If compressed, extract as follows:
gunzip -c OCE ARCHIVE | cpio -idmv
If not compressed, extract as follows:
cpio -idmv < OCE_ARCHIVE
Where, OCE_ARCHIVE is the name of the archive.
2.      Enter the following command to run the OCE Installer:
3.  $ archive_location/oce_install.sh
The Environment variable screen is displayed.
4.      Enter values for each of the environment variables as described on screen.
Note: You must press the Enter key once to enter the new variable value, and then press the Enter key again to move onto the next variable.
5.      Type Done when finished and press the Enter key.
The installation progress screen is displayed. When all stages are complete, the installer will exit.
6.      Check $ORACLE_HOME/OCEinstallRAC.log file, and verify that there are no errors.
7.      Download the Binaries Package for OCE for Oracle RAC 11g Release 1 (11.1) which matches your platform, and extract the archive as explained in Step 1.
8.      Enter the following command to run the OCE binaries installation script.
9.  $ /tmp/oce/oce_exes_install.sh
10.  Check the OCE Kit installation log file, $ORACLE_HOME/OCE/install_log.txt, and verify that the installation was successful.
The kit is installed in the $ORACLE_HOME/oce directory.

Preparing the System for Multi-Node High Availability Services Testing
The Oracle RAC High Availability Services test suite must use shared storage for data files. Depending on the type of shared storage utilized, some preliminary setup may be required. Before running the Oracle RAC High Availability Services test suite, complete the following:
  • If you are using raw devices or logical volumes for shared storage perform the following steps:

Note:
In this example, the OCE user is oracle, which is a member of the dba group; there are 4 nodes in the cluster; and the OCE logical volumes are located in /dev/ocevg/ directory.

    1. Set up the devices or logical volumesrequired by the tests.
    2. Ensure raw devices are accessible and writable across all nodes by the OCE user
      1. # chown -R oracle:dba /dev/ocevg
      2. # chmod -R og+w /dev/ocevg
    3. Export $ORACLE_HOME/oce/work$ORACLE_HOME/dbs, and $ORACLE_HOME/network/admin from the node 1 to all other nodes in the cluster.
On node 1:
      1. # exportfs -i -o rw <node2>:$ORACLE_HOME/oce/work \
<node3>:$ORACLE_HOME/oce/work \
<node4>:$ORACLE_HOME/oce/work
      1. # exportfs -i -o rw <node2>:$ORACLE_HOME/dbs \
<node3>:$ORACLE_HOME/dbs \
<node4>:$ORACLE_HOME/dbs
    1. # exportfs -i -o rw <node2>:$ORACLE_HOME/network/admin \
<node3>:$ORACLE_HOME/network/admin \
<node4>:$ORACLE_HOME/network/admin
    1. $ORACLE_HOME/oce/work$ORACLE_HOME/network/admin, and $ORACLE_HOME/dbs must be mounted on all secondary nodes from the primary (exported) node.
On all nodes except node 1:
      1. # mkdir –p $ORACLE_HOME/oce/work
      2. # chown oracle:dba $ORACLE_HOME/oce/work
      3. # mount <node1>:$ORACLE_HOME/oce/work $ORACLE_HOME/oce/work
      4. # mount <node1>:$ORACLE_HOME/dbs $ORACLE_HOME/dbs
      5. # mount <node1>:$ORACLE_HOME/network/admin \
$ORACLE_HOME/network/admin
  • If you are using OCFS or NAS or a vendor clustered file system (CFS), and ORACLE_HOME directory is not located on shared partition, then perform the following steps:

Note:
If NAS, ensure that the appropriate mount options are employed when mounting the NAS partition. Oracle requires specific mount options. Consult your NAS Filer documentation for further details.

    1. Symbolically link $ORACLE_HOME/dbs to the OCFS/CFS/NAS partition on all nodes (in this example, the OCFS/CFS/NAS partition is at /sharedfs).
On node 1:
    1. mkdir /sharedfs/dbs
    2. chown oracle:dba /sharedfs
On all nodes:
      1. mv $ORACLE_HOME/dbs $ORACLE_HOME/dbs.BAK
      2. ln -s /sharedfs/dbs $ORACLE_HOME/dbs
    1. Export $ORACLE_HOME/oce/work and $ORACLE_HOME/network/admin from the primary node.
      1. # exportfs -i -o rw <node2>:$ORACLE_HOME/oce/work \
<node3>:$ORACLE_HOME/oce/work \
<node4>:$ORACLE_HOME/oce/work
    1. # exportfs -i -o rw <node2>:$ORACLE_HOME/network/admin \
<node3>:$ORACLE_HOME/network/admin \
<node4>:$ORACLE_HOME/network/admin
    1. $ORACLE_HOME/oce/work and $ORACLE_HOME/network/admin must be mounted on all secondary nodes from the primary (exported) node. Default mount options will suffice.
On all nodes except node 1:
      1. # mkdir –p $ORACLE_HOME/oce/work
      2. # chown oracle:dba $ORACLE_HOME/oce/work
      3. # mount <node1>:$ORACLE_HOME/oce/work $ORACLE_HOME/oce/work
      4. # mount <node1>:$ORACLE_HOME/dbs $ORACLE_HOME/dbs
      5. # mount <node1>:$ORACLE_HOME/network/admin \
$ORACLE_HOME/network/admin
  • If you are using OCFS or CFS accessing a shared Oracle home directory, no setup is required.
  • Ensure that no databases are running.

Starting Test Manager
To start Test Manager:
  1. Ensure that the DISPLAY environment variable is set appropriately for your system. To verify that it is, try starting up xclock. If you do not see the clock, or you receive errors, DISPLAY is not set appropriately. You must correct any errors before proceeding.
  2. Enter the following command to launch OCE Test Manager:
3.  $ORACLE_HOME/oce/bin/startTM.sh > /tmp/OCETM.log 2>&1
The OCE Main Menu and OCE Test Manager windows appear.
Running a Test for the First Time
If this is your first time running certification tests, you must perform the following steps:
  1. Start Test Manager as described in Starting Test Manager.
  2. From the OCE - Main Menu window, double click Utilities.
  3. Run the bmchk test by selecting it and clicking Execute.
  4. When the test completes, click Results in the Test Manager window to check the outcome. If the test fails, you must analyze the output ($INST_HOME/work/bmchk) and resolve any issues. Do not proceed with testing until bmchk executes successfully.
  5. Run sdbck (the Seed Database Verification utility) test by selecting it and clicking Execute.
  6. When the test completes, click Results in the Test Manager window to check the outcome. If the test fails, you must analyze the output ($INST_HOME/work/sdbck) and resolve any issues. Do not proceed with testing until sdbck executes successfully.
  7. Run cssck (the CSS Daemon Verification utility) test by selecting it and clicking Execute.
  8. When the test completes, click Results in the Test Manager window to check the outcome. If the test fails, you must analyze the output ($INST_HOME/work/cssck) and resolve any issues. Do not proceed with testing until cssck executes successfully.

Running the OCE Test Suites

The OCE release consists of a set of test suites that you run from Test Manager. Each test suite consists of one or more individual tests. To complete the certification, run each of the Test Suites in the kit for the product for which you are certifying your system.
1.      From the OCE - Main Menu screen, double-click Complete test suites.
2.      From the screen that appears, select the test suite you want to run and click Execute to run it.
The test suite runs.
Test Manager creates two entries for the test suite in the Test Manager window; one in the Suite Name field and another in the History field:
o    The entry in the Current Test field is displayed only for the duration of a test. It displays the time at which you requested the test, and if it starts, when it started. Test Manager might display some tests with a status of Waiting until resources become available on the system.
o    The entry in the History field displays the time you requested it.
When a test finishes, Test Manager deletes its entry in the Current Tests field and adds another entry to the History field showing when the test finished.






Below are the Test plans for Oracle Clusterware Compatibility (Destructive) Testing
(Category : ORACLE HIGH AVAILABILITY FEATURES)
Clusterware Test Category
[Test Code]
Action  Target
Detailed Test Execution
Expected Test Outcome
Actual Test Outcome
[D]
Oracle HA Features


[HW-CW-09]

Run multiple cluvfy operations during Oracle Clusterware and RAC install  All RAC hosts

Configuration:
GNS:
Gns with dhcp (1)
Gns without dhcp (2)
Without gns (3)
Preferred option 1, if not applicable option   2,  if still not applicable option 3

ASM:
Flex asm (1)
Standard asm (2)
Preferred option 1, if not applicable option 2

DB:
CDB
Preconditions:
·          Type `cluvfy` to see all available command syntax and options

Steps:
1- Run cluvfy precondition
2- Do the next install step
3- Run cluvfy post-condition
(cluvfy comp software –n node_list) to check the file permissions
No need to collect CRS/RDBMS log for this test.  You need to submit the output for cluvfy.


Vendor Clusterware:
- same as RAC

RAC:
-           Correct cluster verification checks given the state of the cluster hardware and software

Pls provide cvu related logs under
$CRS_HOME/cv/log


[HW-CW-10]

Run concurrent crsctl start/stop crs commands to stop or start Oracle Clusterware in planned mode  All RAC hosts

Configuration:
GNS:
Gns with dhcp (1)
Gns without dhcp (2)
Without gns (3)
Preferred option 1, if not applicable option 2,  if still not applicable option 3

ASM:
Flex asm (1)
Standard asm (2)
Preferred option 1, if not applicable option 2

DB:
CDB


Preconditions:
·          Initiate all Workloads
·          Identify both CSS and CRS master nodes
·          Type `crsctl` as root to see all available command syntax and options

Steps:
1- As root user, run `crsctl stop crs` command concurrently on more than one RAC host, to stop the resident Oracle Clusterware stack
2- Wait until the target Oracle Clusterware stack is fully stopped (via `ps` command)
3- As root user, run `crsctl start crs
    -wait` command concurrently on more than one RAC host, to start the resident Oracle Clusterware stack


Vendor Clusterware:
- N/A

RAC:
Stop:  All Oracle Clusterware daemons stop without leaving open ports or zombie processes
Start:  All Oracle Clusterware daemons start without error messages in stdout or any of the CRS, CSS or EVM traces
Start:  All registered HA resource states match the “target” states, as per 
“crsctl stat res –t”

For 12cR1, collect
“crsctl stat res –t” in a 60s loop from beginning till the end of run.  Attach the output for auditing.

[HW-CW-11]

Run other concurrent crsctl commands, such as crsctl check crs,   All RAC hosts

Configuration:
GNS:
Gns with dhcp (1)
Gns without dhcp (2)
Without gns (3)
Preferred option 1, if not applicable option   2,  if still not applicable option 3

ASM:
Flex asm (1)
Standard asm (2)
Preferred option 1, if not applicable option 2

DB:
CDB
Preconditions:
·          Initiate all Workloads
·          Identify both CSS and CRS master nodes
·          Type `crsctl` as root to see all available command syntax and options

Steps:
1-        As root user, run any `crsctl check crs` commands concurrently on all nodes
2-        As root user, run any `crsctl check cluster -all` commands concurrently on all nodes


Vendor Clusterware:
- same as RAC

RAC:
-           Both `crsctl check crs` and `crsctl check cluster -all` commands produce the appropriate, useful output, without any error messages
-           Collect output for step 1 and step 2

[HW-CW-12]

Votedisk and OCR operation

Configuration:
GNS:
Gns with dhcp (1)
Gns without dhcp (2)
Without gns (3)
Preferred option 1, if not applicable option 2,  if still not applicable option 3

ASM:
Flex asm (1)
Standard asm (2)
Preferred option 1, if not applicable option 2

DB:
CDB
Preconditions:
·          Make sure votedisk on ASM diskgroup
·          Make sure ASM OCR files are used
·          Make sure at least one normal redundancy ASM Diskgroup with three failgroups is created and its “compatible.asm” attribute is set to “11.2”;

Steps:
1-        Make sure crs stack are running in all nodes.

2-        Run “crsctl query css votedisk” to check configured VFs;
3-        Run “crsctl replace votedisk +{ASM_DG_NAME}”(As crs user or root user);
4-        Run “crsctl query css votedisk” to get the new VF list;
5-        Run “ocrconfig –add +{ASM_DGNAME}” as root user;
6-        Run “ocrcheck” to verify the OCR files;
7-        Restart CRS stack and then verify the VF/OCR after it comes back;

Variants:
1. Add up to 5 OCR files and restart CRS stack;


RAC:
-           In 12cR1, we can support up to 5 OCRs;

-            





[HW-CW-13]
crsctl command to manage Oracle clusterware stack

Configuration:
GNS:
Gns with dhcp (1)
Gns without dhcp (2)
Without gns (3)
Preferred option 1, if not applicable option 2,  if still not applicable option 3

ASM:
Flex asm (1)
Standard asm (2)
Preferred option 1, if not applicable option 2

DB:
CDB
Preconditions:
·          CRS stack is up and running on all nodes.

Steps:
1-        Run ‘crsctl check cluster –all’ to get the stack status on all cluster nodes. Make sure stack status of all cluster nodes are correct;
2-        Run ‘crsctl stop cluster –all’ to stop all CRS resource (CSSD/CRSD/EVMD) with application resources;
3-        Run ‘crsctl status cluster –all’ to make sure CRS resource are OFFLINE;
4-        Run ‘crsctl start cluster –all’ to bring back the whole cluster stack
RAC:
-           After running “crsctl stop cluster –all”, make sure all ocssd/evmd/crsd processes are stopped on all cluster nodes by “ps –ef”.
For 12cR1, collect
“crsctl stat res –t” in a 60s loop from beginning till the end of run.  Attach the output for auditing.


[HW-CW-14]

OCR stores in ASM’s diskgroup and kill asm fatal process

Configuration:
GNS:
Gns with dhcp (1)
Gns without dhcp (2)
Without gns (3)
Preferred option 1, if not applicable option   2,  if still not applicable option 3

ASM:
Flex asm (1)
Standard asm (2)
Preferred option 1, if not applicable option 2

DB:
CDB
Preconditions:
·          Initiate Workloads

Steps:
·          Make sure only ASM OCR files   are used by “ocrcheck –config”;

·          Kill the ASM pmon process on   the CRSD PE Master node;

Variants:
   Repeat the same test on non-OCR Master node.
Clusterware:
Because OCR is stored in ASM, if ASM fails or is brought down on crsd pe master, CRSD pe master will exit and  select a new crsd pe master

- ASM, CRSD will be automatically restarted.

-RDBMS instance should connect to other available asm instance in flex asm env 

- After CRSD restart, all resources’ state shouldn’t change

- New crsd pe master node should be the old crsd pe standby master A new crsd pe standby master should be   elected on other nodes.

(CRSD should recover resources’ previous state)

For 12cR1, collect
“crsctl stat res –t” in a 60s loop from beginning till the end of run.  Attach the output for auditing.



Collect Logfiles
Run each destructive test, taking note of the test start time, test stop time and fault injection time.  On the surviving node (if applicable), run the “date; crsctl stat res –t; sleep 60” in a loop

At the end of the test run, please collect the following logs and put them in directory <CRSHome>/log with the name format as [log_name]_[hostname] and then tar up and compress  with file name <VendorName>_<TestCode>.tar.gz.(e.g. WidgetCorp_HW-STOR-07.tar.gz):
·         Under <CRSHome>/log/<hostname>, the following logs are required
o   alert[hostname].log
o   crsd/crsd.log
o   cssd/ocssd.log
o   evmd/evmd.log
o   ohasd/ohasd.log
o   gpnpd/gpnpd.log
o   diskmon/diskmon.log
o   mdnsd/mdnsd.log
o   ctssd/ctssd.log
o   agent/*
o   gipcd/gipcd.log  (11.2.0.2 new feature)
o   cvu/cvulog/*.log (11.2.0.2 new feature)
o   cvu/cvutrc/*     (11.2.0.2 new feature)
o   srvm/*
o   admin/*
o   acfs/*
o   crfmond/*
o   crflogd/*
o   racg/*
o   gnsd/* (if gns configured)