Resolving ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE

I was working on Database upgrade from 11.2.0.3 to 11.2.0.4 on a 2 node RAC instance.
After Installation of 11.2.0.4 software and connecting sqlplus in the new environment, Got the below error when trying to startup the database in upgrade mode


SQL> Startup upgrade
ORACLE instance started.

Total System Global Area 1.3262E+11 bytes
Fixed Size                  2304584 bytes
Variable Size            2.2481E+10 bytes
Database Buffers         1.1006E+11 bytes
Redo Buffers               74080256 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE
Process ID: 29206
Session ID: 2002 Serial number: 3


Issue Fix
=======

I observed from the initialization parameter file, the below values are set for cluster parameters

cluster_database         = TRUE
cluster_database_instances= 2



We need to set the value of cluster_database to FALSE using the below command


SQL> alter system set cluster_database=FALSE scope=spfile sid='*' ;


Reboot the Database for the changes to be effective.
Now you will see that value of cluster_database_instances parameter is automatically changed to 1,after setting cluster_database to FALSE.


Now, the database can be started in upgrade mode without any errors.
Revert back the parameter cluster_database to TRUE, after the upgrade is completed.
The following command can be used to set cluster_database to TRUE


SQL> alter system set cluster_database=TRUE scope=spfile sid='*' ;

Restart the database.

PROCESS and DISCARD Folders IMAP

While Troubleshooting workflow relates issues in EBS, we may need to clear PROCESS and DISCARD folders under workflow user.
The below commands are useful to move folders, connecting to imap


a)Shutdown Workflow Components
b) login to imap host(e.g server1.ebs.com) as applmgr user

  $ telnet server1.ebs.com 143
  
    You will get below messages:

    ----------------------------
Trying **.**.***.**

Connected to server1.ebs.com (**.**.***.**).
Escape character is '^]'.

* OK Dovecot ready.
    ----------------------------
  i login wfmgr *****
  

  i rename PROCESS process_<sys_date>
  
 i rename DISCARD discard_<sys_date>
  

 i create PROCESS
  
i create DISCARD

i subscribe PROCESS

 i subscribe DISCARD


 i unsubscribe process_<sys_date>

 i unsubscribe discard_<sys_date>
  

c) Startup Workflow Components

How to Check Patch Applied Status in Oracle Applications?

R11i,12.0.x,12.1.x
=============

From Oracle Applications Release 11i to 12.1.x, Patch application Status can be queried from ad_bugs or ad_applied_patches tables

Query to check whether patch is applied on a instance.

SQL>select bug_number,creation_date,last_update_date from apps.ad_bugs where bug_number='&patchnum';

SQL> select PATCH_NAME,CREATION_DATE from apps.ad_applied_patches where PATCH_NAME='&patchnum';

What is difference between above two Queries?


Suppose, we applied a patch which has multiple bugfixes.The details of all bugs fixed by the patch are listed in ad_bugs table.
In Contrast,ad_applied_patches table give info only about the patches which are applied through adpatch utility.


R12.2
=====

From Release 12.2, Online Patching utility(adop) is used for Patching application Filesystem.

Querying the data from ad_bugs or ad_applied_patches tables may not give correct information.Because, online patching can be aborted anytime prior to Cutover Phase.
ad_bugs or ad_applied_patches tables may contain entries for patches which are started through adop and later aborted.

The below query gives accurate Information about Patch Applied Status in R12.2

select AD_PATCH.IS_PATCH_APPLIED(\'$release\',\'$appltop_id\',\'$patch_no\',\'$language\') from dual;

example sql:
SELECT adb.bug_number,ad_patch.is_patch_applied('11i', 1045, adb.bug_number)
FROM ad_bugs adb
WHERE adb.bug_number in (22498647);

or for single app tier installations:
select ad_patch.is_patch_applied('R12',-1,22498647) from dual;

Sample Output:

EXPLICIT = applied
NOT APPLIED = not applied / aborted


Database
=========

Login to Database node as the database os user

Set environment using commands below

$export ORACLE_HOME=/u01/app/oracle/product/112

$export PATH=$PATH:$ORACLE_HOME/Opatch

$which opatch
==>It should give output like /u01/app/oracle/product/112/Opatch

$opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc

== This gives list of all patches applied to the ORACLE_HOME


$opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc|grep 988765


==> To check if a particular patch is applied to the ORACLE_HOME