ADOP Prepare Phase Fails with ORA-31011: XML parsing failed

While applying Patch on a test instance (R12.2.5), Prepare phase failed with error ORA-31011: XML parsing failed

Complete error message from adop logfile
==================================

[aptest12@localhost <RUN> ]$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Initializing.
    Run Edition context  : /test12/applmgr/fs2/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch edition context: /test12/applmgr/fs1/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch file system free space: 1130.34 GB

Validating system setup.
    [ERROR]     Failed to execute SQL statement:
select node_name from FND_OAM_CONTEXT_FILES
where NAME not in ('TEMPLATE','METADATA','config.txt') and
CTX_TYPE='A' and (status is null or upper(status) in ('S','F'))
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'run'
and EXTRACTVALUE(XMLType(TEXT),'//oa_service_group_status[@oa_var=''s_web_admin_status'']')='enabled'
and EXTRACTVALUE(XMLType(TEXT),'//oa_service_list/oa_service[@type=''admin_server'']/oa_service_status')='enabled'

    [ERROR]     Error Message:
    [ERROR]     ORA-31011: XML parsing failed
    [ERROR]     ORA-19202: Error occurred in XML processing
    [ERROR]     LPX-00229: input source is empty
    [ERROR]     ORA-06512: at "SYS.XMLTYPE", line 272
    [ERROR]     ORA-06512: at line 1 (DBD ERROR: error possibly near <*> indicator at char 208 in '    select node_name from FND_OAM_CONTEXT_FILES
    [ERROR]         where NAME not in ('TEMPLATE','METADATA','config.txt') and
    [ERROR]             CTX_TYPE='A' and (status is null or upper(status) in ('S','F'))
    [ERROR]             and EXTRACTVALUE(<*>XMLType(TEXT),'//file_edition_type') = 'run'
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),'//oa_service_group_status[@oa_var=''s_web_admin_status'']')='enabled'
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),'//oa_service_list/oa_service[@type=''admin_server'']/oa_service_status')='enabled'
    [ERROR]     ')
    [UNEXPECTED]Error occurred identifying Admin server node


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


Cause:

fnd_oam_context_files table has incorrect information  or the table is corrupted


Solution:


1. Login to sqlplus as applsys and take backup of fnd_oam_context_files table.


SQL> conn applsys
Enter password:
Connected.
SQL> create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;

Table created.

2. Truncate Table fnd_oam_context_files

SQL>  truncate table fnd_oam_context_files;

Table truncated.


3. Bring down MT Services

4. Run Autoconfig to populate fnd_oam_context_files  with right information.

Adpatch Worker Fails with "ORA-01031: insufficient privileges"

While Applying Patch on Customer environment, adpatch failed with error

Worker log has the below error.
=====================
GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

AD Worker error:
The following ORACLE error:

ORA-01031: insufficient privileges


occurred while executing the SQL statement:

GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

Error occurred in file

/test12/applmgr/1200/ad/12.0.0/patch/115/sql/ademusr.sql

with arguments '&systempwd &un_fnd &pw_fnd &un_apps &pw_apps'.

==============


Solution
=========

1.Login to oracle database server as oracle user

2. Connect to sqlplus as sysdba and run the failed sql statement
sqlplus / as sysdba

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role
  2  ;

Grant succeeded.

SQL>


3. Use adctrl to skip the failed worker (Option 8) and continue with Patching

Weblogic Admin Server Startup Fails with "weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid"

After Changing Weblogic Password in boot.properties, Admin Server Fails to Startup.
Below error is reported in Admin Server Logfile.

<May 10, 2017 10:10:18 AM CDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.> 


Solution
=======

1.Shutdown Admin Server and Nodemanager
2.Go to $DOMAIN_HOME/servers/Admin Server>/data
3.Delete the boot.properties
4.Delete the whole ldap folder
5.Start the node-manager
6.Start Admin Server

"lsnrctl: symbol lookup error: lsnrctl: undefined symbol: nlgh" while starting Oracle Database listener

After Applying PSU, database listener failed to start with below error

 $ lsnrctl start testdb
lsnrctl: symbol lookup error: lsnrctl: undefined symbol: nlgh


Solution
==========
1.Relink network executables using below command

$  make -kf ins_net_server.mk install
/bin/chmod 755 /testdb/oracle/product/11204/bin/
 - Linking tnslsnr
rm -f tnslsnr
gcc -o tnslsnr -m64 -z noexecstack -L/testdb/oracle/product/11204/network/lib/ -L/testdb/oracle/product/11204/lib/ -L/testdb/oracle/product/11204/lib/stubs/  /testdb/oracle/product/11204/network/lib/s0nsgl.o /testdb/oracle/product/11204/network/lib/snsglp.o -lclntsh  `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lztkg11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11   `cat /testdb/oracle/product/11204/lib/sysliblist` -Wl,-rpath,/testdb/oracle/product/11204/lib -lm    `cat /testdb/oracle/product/11204/lib/sysliblist` -ldl -lm   -L/testdb/oracle/product/11204/lib -lons -lnl11  -ln11 -lnlsnr11 -lnlsnrc11 -lnsgr11 -lnms -lncm11 -lnmsp -lpeer -lnro11 -ln11   -lnl11 -lnlsnr11 -lnlsnrc11 -lnms -lncm11 -lnmsp -ln11     -lm
mv -f /testdb/oracle/product/11204/bin/tnslsnr /testdb/oracle/product/11204/bin/tnslsnr0
mv tnslsnr /testdb/oracle/product/11204/bin/tnslsnr
/bin/chmod 751 /testdb/oracle/product/11204/bin/tnslsnr
 - Linking lsnrctl
rm -f lsnrctl
gcc -o lsnrctl -m64 -z noexecstack -L/testdb/oracle/product/11204/network/lib/ -L/testdb/oracle/product/11204/lib/ -L/testdb/oracle/product/11204/lib/stubs/  /testdb/oracle/product/11204/network/lib/s0nsglc.o -lclntsh  `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lztkg11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11   `cat /testdb/oracle/product/11204/lib/sysliblist` -Wl,-rpath,/testdb/oracle/product/11204/lib -lm    `cat /testdb/oracle/product/11204/lib/sysliblist` -ldl -lm   -L/testdb/oracle/product/11204/lib -lnl11  -ln11 -lnlsnr11 -lnlsnrc11 -lnsgr11 -lnms -lncm11 -lnmsp -lpeer -lnro11 -ln11   -lnl11 -lnlsnr11 -lnlsnrc11 -lnms -lncm11 -lnmsp -ln11     -lm
mv -f /testdb/oracle/product/11204/bin/lsnrctl /testdb/oracle/product/11204/bin/lsnrctl0
mv lsnrctl /testdb/oracle/product/11204/bin/lsnrctl
/bin/chmod 751 /testdb/oracle/product/11204/bin/lsnrctl


2. Restart Database Listener

$ lsnrctl start testdb

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 16-MAY-2016 02:00:20

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Starting /testdb/oracle/product/11204/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.4.0 - Production
System parameter file is /testdb/oracle/product/11204/network/admin/testdb/listener.ora
Log messages written to /testdb/log/network/testdb.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dblocalhost)(PORT=1523)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dblocalhost)(PORT=1523)))
STATUS of the LISTENER
------------------------
Alias                     testdb
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                16-MAY-2016 02:00:20
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /testdb/oracle/product/11204/network/admin/testdb/listener.ora
Listener Log File         /testdb/log/network/testdb.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dblocalhost)(PORT=1523)))
Services Summary...
Service "testdb" has 1 instance(s).
  Instance "testdb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

What are the daily activities of Oracle Applications DBA?

1. Maintaining Oracle Application File System and Database using AD Utilities.
2. Starting, stopping and troubleshooting issues related to Application server components.
3. Using adadmin to relink executables, generate forms/reports,generate jar files, compile/validate APPS schema.
4. Analyzing patch pre-requisites and Applying the Patches using AD Patch Utility on Application file System and Technology Stack.
5. Applying Mini Packs, Family Packs, Maintenance Pack and Rollup patches.
6. Applying database patches using opatch utility and troubleshooting issues related to OPatch.
7. System administration activities like creating users, assigning Responsibilities, defining custom managers.
8. Changing configuration files as per the requirement.
9. Changing APPS password and Application database users’ password for security.
10. Performing Healthchecks.
11. Splicing off-cycle products using adsplice utility.
12. Cloning of Oracle Applications 11i from Production to Test Instance.
13. Taking backups- physical, logical, RMAN and performing Restore, Recovery.
14. Troubleshooting issues related to concurrent managers.
15. Managing System Configuration using Autoconfig.
16. Modifying initialization parameters as per the requirement-PFILE AND SPFILE.
17. Reorganization of tables that are fragmented.
18. Creating users and assigning appropriate roles, privileges.
19. Managing tablespaces and data files.
20. Monitoring alert log file.
21. Troubleshooting ORA- errors.
22. Analyzing AWR reports.

How to enable Trace for Application User in Oracle Applications 11i/R12?

Enabling Tracing for a Application User makes diagnostics easier to analyze.Since only the sql operations performed by a specific User are Traced.


Steps to Enable Trace for Application User:

1.Login to Oracle Applications and select the System Administrator responsibility. 
Navigate to  Profile -->  System 

2.Search for Profile Option 'Initialization SQL Statement - Custom' .Also Specify the username whom you wish to enable tracing during Search.

3.
Under username column Update the below line

BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'USER_TRACE_IDENTIFER' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;

Note : Level cane be 4,8,12 depends on your requirement 


Save the Changes done in Profile Option

4. Be careful while updating the Profile Option, Any incorrect Syntax may Prevent user from Logging in

5. Inform User to Re-login and Perform the Activity which needs to be traced.

6. Trace Files can be found in the directory specified by "user_dump_directory" parameter on the database server.
Use the identifier given in Profile Option to search the trace file in udump directory.

7. After the issue is reproduced and required trace files are collected for analysis, disable Tracing from Front End.
Its recommended to disable the trace immediately after the diagnostics are collected, as it will generate huge logfiles.

8. Navigate to system=> Profile form under System Administrator Responsibility.

Query for the username  and Profile Option name "Initialization SQL Statement - Custom"

Remove the values and make it blank.

Save the changes.

Discoverer Login Fails with ORA-1017

Logging to discoverer url as sysadmin gives the following error.

ORA-1017: Invalid username/password
A connection error.  
- Oracle BI Discoverer is unable to authenticate using the password provided. This can happen due to an invalid password or because the password was lost while using back, forward, or refresh in your browser. Enter the password again to continue.
- Failed to connect to database - Unable to connect to Oracle Applications database (afscpgcs) 

Cause:

APPL_SERVER_ID parameter in .dbc file of Applications MT node(applmgr) and discoverer user is not matched


$diff test1_disco.dbc test1_mt.dbc

4c4
< APPL_SERVER_ID=178***********
---
> APPL_SERVER_ID=1A***********



Solution:

Copy the <SID>.dbc from $FND_TOP/secure or $FND_SECURE of Applications Middle tier to Disoverer user

Bounce Discoverer Services

Autopatch error: The worker should not have status 'Running' or 'Restarted' at this point

If you are restarting a failed patch session in oracle applications, sometimes you may encounter the error

AutoPatch error:
The worker should not have status 'Running' or 'Restarted' at this point.

Telling workers to quit...

All workers have quit.

Connecting to APPS......Connected successfully.

AutoPatch error:

Error running SQL and EXEC commands in parallel

Cause:

1. adpatch or adop process was killed from OS level while patch is being applied

2. Database shutdown or terminated



Solution:

Using adctrl utility, use option 4 to change the worker status to Failed


Review the messages above, then press [Return] to continue.

                    AD Controller Menu
     ---------------------------------------------------

     1.    Show worker status

     2.    Tell worker to restart a failed job

     3.    Tell worker to quit

     4.    Tell manager that a worker failed its job

     5.    Tell manager that a worker acknowledges quit

     6.    Restart a worker on the current machine

     7.    Exit


Enter your choice [1] : 4

Enter the worker number(s)/range(s) or 'all' for all workers,
or press [Return] to go back to the menu : all

Status changed to 'Failed' for worker 1.
Status changed to 'Failed' for worker 2.
Status changed to 'Failed' for worker 3.
Status changed to 'Failed' for worker 4.
Review the messages above, then press [Return] to continue.



select option "1. Show Worker Status" , The worker status will be "Failed" 

Now select option 2. "2. Tell worker to restart a failed job" 


Enter the worker number(s)/range(s) or 'all' for all workers,
or press [Return] to go back to the menu : all

Status changed to 'Fixed, restart' for worker 1.
Status changed to 'Fixed, restart' for worker 2.
Status changed to 'Fixed, restart' for worker 3.
Status changed to 'Fixed, restart' for worker 4.


Restart adpatch 

In 11i, R12.1.X version of oracle applications

when applying patch using adpatch , Select option Yes When it prompts for

'Do you wish to Continue with Previous adpatch Session'

In R12.2.x, Use restart and abandon parameters


adop phase=apply patches=123456 restart=yes abandon=no 

To restart the patch session from where it failed.

ADOP Patching Fails with aipspv(): Error setting up PDI list for upgrade

While Applying patch on a 12.2.5 environment using online patch cycle in adop, Autopatch Failed with error

AutoPatch error:
Invalid password for schema "CUST1" (oracle_id = 20087)

AutoPatch error:
Failed getting PDI list for product 'xx1'

AutoPatch error:
aipspv(): Error setting up PDI list for upgrade

Root Cause
=========

Password for Application schema is not changed using FNDCPASS

Solution
========

1.Change the password of custom application with FNDCPASS 

Syntax for FNDCPASS

$ FNDCPASS apps/<PASSWORD> 0 Y system/<PASSWORD> ORACLE CUST1 <NEWPASSWORD>
Log filename : L********.log


Report filename : O*****.out

2. Verify the password change is successful

$ cat L*******.log
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.2

Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

FNDCPASS:
+---------------------------------------------------------------------------+

Current system time is 10-MAR-2016 23:21:30

+---------------------------------------------------------------------------+

Working...
Password is changed successfully for user BOLINF.
FNDCPASS completed successfully.

+---------------------------------------------------------------------------+
Concurrent request completed successfully
Current system time is 10-MAR-2016 23:21:30

+---------------------------------------------------------------------------+



Please Note, In 12.2.x Environments FNDCPASS must be executed on RUN Filesystem.

3. Restart the online patching cycle 

While restarting adop apply phase after a previously failed session, use options abandon and restart


adop phase=apply patches=123456 restart=yes abandon=no 

[Use this command if you wish to continue the previous patch session from the step where it failed]

adop phase=apply patches=123456 restart=no abandon=yes


[Use this command if you wish to start a fresh patch session]




If changing password using FNDCPASS does not work, analyse further details about database user and application user.

Check if the database user and application user are defined correctly using the sql queries below

SQL> select ORACLE_ID from FND_ORACLE_USERID where oracle_username like '%CUST1%';

 ORACLE_ID
----------
     2008

SQL> select application_id from fnd_application where application_short_name = 'XX1';

APPLICATION_ID
--------------
         2003

SQL> select * from fnd_product_installations where oracle_id=2008;


If the user details are not correct from the output of above sql, Please correct them.

ORA-12504 While running Autoconfig on Database Server

After Oracle Database upgrade from 11.2.0.3 to 11.2.0.4, Autoconfig errored on Database server with the below error

[AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
      <filename>  <phase>  <return code where appropriate>

  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /test1/oracle/product/11204/appsutil/install/test1_host1
      afdbprf.sh              INSTE8_PRF         1

  [APPLY PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /test1/oracle/product/11204/appsutil/install/test1_host1
      adcrobj.sh              INSTE8_APPLY       1


AutoConfig is exiting with status 2

====Running the failed script has given the ORA-12504 error

java.lang.OutOfMemoryError: Java heap space error in Output PostProcessor

Possible causes for Java Heap Space error in OPP

1. There is limited heap space in the JVM to process the report
2. The Java application running is out of memory
3. The scalable feature isn't enabled
4. An older version of the JRE is being used in the configuration, 
5. There is insufficient space in the designated temp directory for processing the report, and reports are embedded with images which require a lot of space that unavailable.


The log will contain a java stack error similar to the following:
java.lang.reflect.InvocationTargetException
Caused by: java.lang.OutOfMemoryError: Java heap space
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:2869)
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:3000)
at oracle.xdo.parser.v2.XMLDocument.createTextNode(XMLDocument.java:735)
at oracle.xdo.parser.v2.DocumentBuilder.characters(DocumentBuilder.java:553)
at oracle.xdo.parser.v2.NonValidatingParser.reportCharacters(NonValidatingParser.java:1680)
at oracle.xdo.parser.v2.NonValidatingParser.parseText(NonValidatingParser.java:1391)
at oracle.xdo.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1268)


How to resolve Java Heap Space error in Output Post Processor?


Solution 1
==========


1. Determine what the heap size per OPP process is currently:
select DEVELOPER_PARAMETERS from FND_CP_SERVICES
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

2. The default should be:
J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m

3. Increase the Heap Space per Process to 1024:
update FND_CP_SERVICES
set DEVELOPER_PARAMETERS =
'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

4. Bring the managers down.
5. Please run the Concurrent Manager Recovery feature to address any Concurrent Manager / Concurrent Processing issues within the Oracle Application Manager.  NOTE: Regular use of the CP Analyzer in Note 1411723.1 can provide an Immediate Analysis and Output of a Concurrent Environment.
6. Bring the managers up again.

OR

Solution 2
==========
1. Log into applications with the System Administrator responsibility.
2. Navigate to Concurrent -> Program -> Define
3. Query the XML Publisher Template Re-Generator program
4. Set the following value for the Executable Options : -Xmx1024m
5. Save changes.
6. Retest the program.

OR


Solution 3
==========

Configure the XML Publisher Administrator Configuration settings.
1. As XML Publisher Administrator navigate to Administration->Configuration.
2. Under Temporary Directory pick a temporary file location on your concurrent processing node. This should be at least 5GB or 20x larger than largest XML data file you generate
3.  Under FO Processing, set:
o Use XML Publisher's XSLT processor set to True
o Enable scalable feature of XSLT processor set to False
o Enable XSLT runtime optimization set to True 

Applying Patch R12.AD.B.7 fails with ORA-1031 while running ademusr.sql

While Applying Patch 21314548:R12.AD.B.DELTA.7 using adpatch, Patch worker failed at running the script ademusr.sql.
To begin troubleshooting the issue,First check the worker id which was running the failed script.
We can get this information from Patch logfile[adpatch.log or u21314548.log] or console output while applying the patch.

Check the worker logfile[adworkxxx.log] for the failed worker

Errors in workerlog
===================
ORA-01031 occurs while Granting SELECT on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role For Patch 20395533 (Doc ID 2093641.1)


GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

AD Worker error:
The following ORACLE error:
ORA-01031: insufficient privileges
occurred while executing the SQL statement:

GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

Error occurred in file
/test1/applmgr/1213/ad/12.0.0/patch/115/sql/ademusr.sql
with arguments '&systempwd &un_fnd &pw_fnd &un_apps &pw_apps'.

Time when worker failed: Fri Mar 04 2016 09:48:00


The error occurred because worker is running the scripts as apps user, and APPS user does not have enough privileges to grant DBA_USERS_WITH_DEFPWD

Solution
========

1.Run the below sql statements to grant the role as system user.

SQL> show user
USER is "SYS"
SQL> GRANT select on DBA_USERS_WITH_DEFPWD to SYSTEM with grant option;

Grant succeeded.

SQL> show user
USER is "SYSTEM"
SQL> GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role;

Grant succeeded.

2. Restart the failed worker using adctrl.

3. Monitor Patching until completion.

Resolving ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE

I was working on Database upgrade from 11.2.0.3 to 11.2.0.4 on a 2 node RAC instance.
After Installation of 11.2.0.4 software and connecting sqlplus in the new environment, Got the below error when trying to startup the database in upgrade mode


SQL> Startup upgrade
ORACLE instance started.

Total System Global Area 1.3262E+11 bytes
Fixed Size                  2304584 bytes
Variable Size            2.2481E+10 bytes
Database Buffers         1.1006E+11 bytes
Redo Buffers               74080256 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE
Process ID: 29206
Session ID: 2002 Serial number: 3


Issue Fix
=======

I observed from the initialization parameter file, the below values are set for cluster parameters

cluster_database         = TRUE
cluster_database_instances= 2



We need to set the value of cluster_database to FALSE using the below command


SQL> alter system set cluster_database=FALSE scope=spfile sid='*' ;


Reboot the Database for the changes to be effective.
Now you will see that value of cluster_database_instances parameter is automatically changed to 1,after setting cluster_database to FALSE.


Now, the database can be started in upgrade mode without any errors.
Revert back the parameter cluster_database to TRUE, after the upgrade is completed.
The following command can be used to set cluster_database to TRUE


SQL> alter system set cluster_database=TRUE scope=spfile sid='*' ;

Restart the database.