Clusterware Compatibility (Destructive) Testing

Destructive tests include forced failures by software and hardware while the system is running with either minimal or high workload. Oracle software - one or more of Oracle background processes is killed manually. OS software – one or more of the cluster daemons is killed manually or the system is forced to reboot. Hardware: Manual removal of network or disk connectivity or power supply.

There are two major categories of cluster compatibility tests:

 Clusterware (Destructive):

Starting with Oracle Database 10g, the certification and validation process has been enhanced to include hardware destructive tests executed under high system load.

   Cluster File System:

  Starting with Oracle Database 11g, the certification and validation process has been further enhanced to include a set of destructive and high availability tests, designed to verify the use of cluster file system to support the various Oracle Clusterware and Real Application Clusters components.

Useful adop options and syntax

1.Using analytics parameter in adop apply phase


$ adop phase=apply analytics=yes

Specifying this option will cause adop to run the following scripts and generate the associated output files (reports):

ADZDCMPED.sql - This script is used to display the differences between the run and patch editions, including new and changed objects. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>/adzdcmped.out.

ADZDSHOWED.sql - This script is used to display the editions in the system. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowed.out.

ADZDSHOWOBJS.sql - This script is used to display the summary of editioned objects per edition. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowobjs.out

ADZDSHOWSM.sql - This script is used to display the status report for the seed data manager. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowsm.out

Note: The analytics parameter should only be used when required, because of the extra processing needed.



2. flags=autoskip 

 e.g adop phase=apply patches=12345678  flags=autoskip 

 This option is an alternative for  "Continue as if it were successful"?  in adpatch, very useful in cases where patch failed to compile forms/reports and exiting.
 We need not restart the adop session just to compile a single forms (.fmb,.fmx) file.
 Make sure to review the autoskip.log logfile and fix the issues in autoskip log whenever you use autoskip flag in adop cycle.

3.  skipsyncerror=(yes|no)  [default: no]

 Specifies whether to ignore errors that may occur during incremental file system synchronization.  This might happen if you applied
 a patch in the previous patching cycle that had errors but decided to continue with the cutover.  When the patch is synchronized on
 the next patching cycle, the apply errors may occur again, but can be ignored.

4. wait_on_failed_job=(yes|no)  [default: no]

 Controls whether adop apply command exits when all workers have failed.  Instead of exiting, you can force adop to wait, and use the "adctrl" to retry failed jobs.

e.g adop phase=apply patches=7777 wait_on_failed_job=yes


5. Abort

The Online Patching Cycle can be aborted at any time prior to Cutover

e.g adop phase=abort

This is needed  if unrecoverable error happened or the user decides that patch is not needed.
If adop phase=apply failed, user should try abandon=yes first.
The abort command drops the database patch edition and  returns the system to normal runtime state.  Immediately following abort, you must also run a full cleanup and
fs_clone operation to fully remove effects of the failed online patching cycle.


            

What Happens in ADOP apply Phase

Apply Phase is executed in PATCH Filesystem
During Apply phase of an ADOP cycle, adop execute patch drivers to update Patch Edition
We can have multiple apply phases in an adop cycle, Multiple patches including customizations can be installed in this phase.

The production application is online and accessible to users. RUN filesystem is not affected by the changes.
Patches are applied to the copy (Patch Edition)
 Changes are made in the isolation of an Edition 
The running application is unaffected by these changes


workers 
specifies number of parallel workers, automatically calculated if not specified  
abandon 
abandons a patching session (if set to "yes") and must have an opposite specification to that of parameter "restart".
restart
 restarts a patching session (if set to "yes") and must have an opposite specification to that of parameter "abandon".

Example: $ adop phase=apply workers=4 abandon=yes restart=no patches=1234567,8909228



For more adop options and their usage Click here

Autopatch error: The worker should not have status 'Running' or 'Restarted' at this point

If you are restarting a failed patch session in oracle applications, sometimes you may encounter the error

AutoPatch error:
The worker should not have status 'Running' or 'Restarted' at this point.

Telling workers to quit...

All workers have quit.

Connecting to APPS......Connected successfully.

AutoPatch error:

Error running SQL and EXEC commands in parallel

Cause:

1. adpatch or adop process was killed from OS level while patch is being applied

2. Database shutdown or terminated



Solution:

Using adctrl utility, use option 4 to change the worker status to Failed


Review the messages above, then press [Return] to continue.

                    AD Controller Menu
     ---------------------------------------------------

     1.    Show worker status

     2.    Tell worker to restart a failed job

     3.    Tell worker to quit

     4.    Tell manager that a worker failed its job

     5.    Tell manager that a worker acknowledges quit

     6.    Restart a worker on the current machine

     7.    Exit


Enter your choice [1] : 4

Enter the worker number(s)/range(s) or 'all' for all workers,
or press [Return] to go back to the menu : all

Status changed to 'Failed' for worker 1.
Status changed to 'Failed' for worker 2.
Status changed to 'Failed' for worker 3.
Status changed to 'Failed' for worker 4.
Review the messages above, then press [Return] to continue.



select option "1. Show Worker Status" , The worker status will be "Failed" 

Now select option 2. "2. Tell worker to restart a failed job" 


Enter the worker number(s)/range(s) or 'all' for all workers,
or press [Return] to go back to the menu : all

Status changed to 'Fixed, restart' for worker 1.
Status changed to 'Fixed, restart' for worker 2.
Status changed to 'Fixed, restart' for worker 3.
Status changed to 'Fixed, restart' for worker 4.


Restart adpatch 

In 11i, R12.1.X version of oracle applications

when applying patch using adpatch , Select option Yes When it prompts for

'Do you wish to Continue with Previous adpatch Session'

In R12.2.x, Use restart and abandon parameters


adop phase=apply patches=123456 restart=yes abandon=no 

To restart the patch session from where it failed.