Sunday 13 March 2016

ADOP Patching Fails with aipspv(): Error setting up PDI list for upgrade

While Applying patch on a 12.2.5 environment using online patch cycle in adop, Autopatch Failed with error

AutoPatch error:
Invalid password for schema "CUST1" (oracle_id = 20087)

AutoPatch error:
Failed getting PDI list for product 'xx1'

AutoPatch error:
aipspv(): Error setting up PDI list for upgrade

Root Cause
=========

Password for Application schema is not changed using FNDCPASS

Solution
========

1.Change the password of custom application with FNDCPASS 

Syntax for FNDCPASS

$ FNDCPASS apps/<PASSWORD> 0 Y system/<PASSWORD> ORACLE CUST1 <NEWPASSWORD>
Log filename : L********.log


Report filename : O*****.out

2. Verify the password change is successful

$ cat L*******.log
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.2

Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

FNDCPASS:
+---------------------------------------------------------------------------+

Current system time is 10-MAR-2016 23:21:30

+---------------------------------------------------------------------------+

Working...
Password is changed successfully for user BOLINF.
FNDCPASS completed successfully.

+---------------------------------------------------------------------------+
Concurrent request completed successfully
Current system time is 10-MAR-2016 23:21:30

+---------------------------------------------------------------------------+



Please Note, In 12.2.x Environments FNDCPASS must be executed on RUN Filesystem.

3. Restart the online patching cycle 

While restarting adop apply phase after a previously failed session, use options abandon and restart


adop phase=apply patches=123456 restart=yes abandon=no 

[Use this command if you wish to continue the previous patch session from the step where it failed]

adop phase=apply patches=123456 restart=no abandon=yes


[Use this command if you wish to start a fresh patch session]




If changing password using FNDCPASS does not work, analyse further details about database user and application user.

Check if the database user and application user are defined correctly using the sql queries below

SQL> select ORACLE_ID from FND_ORACLE_USERID where oracle_username like '%CUST1%';

 ORACLE_ID
----------
     2008

SQL> select application_id from fnd_application where application_short_name = 'XX1';

APPLICATION_ID
--------------
         2003

SQL> select * from fnd_product_installations where oracle_id=2008;


If the user details are not correct from the output of above sql, Please correct them.

Tuesday 8 March 2016

ORA-12504 While running Autoconfig on Database Server

After Oracle Database upgrade from 11.2.0.3 to 11.2.0.4, Autoconfig errored on Database server with the below error

[AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
      <filename>  <phase>  <return code where appropriate>

  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /test1/oracle/product/11204/appsutil/install/test1_host1
      afdbprf.sh              INSTE8_PRF         1

  [APPLY PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /test1/oracle/product/11204/appsutil/install/test1_host1
      adcrobj.sh              INSTE8_APPLY       1


AutoConfig is exiting with status 2

====Running the failed script has given the ORA-12504 error

java.lang.OutOfMemoryError: Java heap space error in Output PostProcessor

Possible causes for Java Heap Space error in OPP

1. There is limited heap space in the JVM to process the report
2. The Java application running is out of memory
3. The scalable feature isn't enabled
4. An older version of the JRE is being used in the configuration, 
5. There is insufficient space in the designated temp directory for processing the report, and reports are embedded with images which require a lot of space that unavailable.


The log will contain a java stack error similar to the following:
java.lang.reflect.InvocationTargetException
Caused by: java.lang.OutOfMemoryError: Java heap space
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:2869)
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:3000)
at oracle.xdo.parser.v2.XMLDocument.createTextNode(XMLDocument.java:735)
at oracle.xdo.parser.v2.DocumentBuilder.characters(DocumentBuilder.java:553)
at oracle.xdo.parser.v2.NonValidatingParser.reportCharacters(NonValidatingParser.java:1680)
at oracle.xdo.parser.v2.NonValidatingParser.parseText(NonValidatingParser.java:1391)
at oracle.xdo.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1268)


How to resolve Java Heap Space error in Output Post Processor?


Solution 1
==========


1. Determine what the heap size per OPP process is currently:
select DEVELOPER_PARAMETERS from FND_CP_SERVICES
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

2. The default should be:
J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m

3. Increase the Heap Space per Process to 1024:
update FND_CP_SERVICES
set DEVELOPER_PARAMETERS =
'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

4. Bring the managers down.
5. Please run the Concurrent Manager Recovery feature to address any Concurrent Manager / Concurrent Processing issues within the Oracle Application Manager.  NOTE: Regular use of the CP Analyzer in Note 1411723.1 can provide an Immediate Analysis and Output of a Concurrent Environment.
6. Bring the managers up again.

OR

Solution 2
==========
1. Log into applications with the System Administrator responsibility.
2. Navigate to Concurrent -> Program -> Define
3. Query the XML Publisher Template Re-Generator program
4. Set the following value for the Executable Options : -Xmx1024m
5. Save changes.
6. Retest the program.

OR


Solution 3
==========

Configure the XML Publisher Administrator Configuration settings.
1. As XML Publisher Administrator navigate to Administration->Configuration.
2. Under Temporary Directory pick a temporary file location on your concurrent processing node. This should be at least 5GB or 20x larger than largest XML data file you generate
3.  Under FO Processing, set:
o Use XML Publisher's XSLT processor set to True
o Enable scalable feature of XSLT processor set to False
o Enable XSLT runtime optimization set to True 

Sunday 6 March 2016

What Happens During ADOP Finalize Phase

The Finalize phase of adop cycle,takes care of below online activities

1.Perform the final operations that can be executed while the Application is online
2.Compile invalid objects
3.Generate derived objects
4.Pre-compute DDL to be run at Cutover

After Finalize phase, we can pause the adop online patching cycle until the appropriate downtime window is available.

Syntax for running finalize phase 

$ adop phase=finalize
$ adop phase=finalize finalize_mode=quick == This is Default Mode
$ adop phase=finalize finalize_mode=full    

Finalize can be run in 2 modes: “QUICK” or “FULL”
FULL gathers database dictionary statistics (not transaction tables statistics)
QUICK skips gathering database dictionary statistics.

Applying Patch R12.AD.B.7 fails with ORA-1031 while running ademusr.sql

While Applying Patch 21314548:R12.AD.B.DELTA.7 using adpatch, Patch worker failed at running the script ademusr.sql.
To begin troubleshooting the issue,First check the worker id which was running the failed script.
We can get this information from Patch logfile[adpatch.log or u21314548.log] or console output while applying the patch.

Check the worker logfile[adworkxxx.log] for the failed worker

Errors in workerlog
===================
ORA-01031 occurs while Granting SELECT on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role For Patch 20395533 (Doc ID 2093641.1)


GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

AD Worker error:
The following ORACLE error:
ORA-01031: insufficient privileges
occurred while executing the SQL statement:

GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

Error occurred in file
/test1/applmgr/1213/ad/12.0.0/patch/115/sql/ademusr.sql
with arguments '&systempwd &un_fnd &pw_fnd &un_apps &pw_apps'.

Time when worker failed: Fri Mar 04 2016 09:48:00


The error occurred because worker is running the scripts as apps user, and APPS user does not have enough privileges to grant DBA_USERS_WITH_DEFPWD

Solution
========

1.Run the below sql statements to grant the role as system user.

SQL> show user
USER is "SYS"
SQL> GRANT select on DBA_USERS_WITH_DEFPWD to SYSTEM with grant option;

Grant succeeded.

SQL> show user
USER is "SYSTEM"
SQL> GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role;

Grant succeeded.

2. Restart the failed worker using adctrl.

3. Monitor Patching until completion.

How to export List of members related to a OID Group into a LDIF


While adding new member to a OID group or making any changes to OID group, we need to backup the existing members details related to group.
The backup can be taken into a LDIF file using ODSM.

Steps to Take backup of OID group members.

1. Login to ODSM url of OID application.
2. Navigate to Data Browser Tab in ODSM Interface.
3. Expand the directories and select the group which needs to be modified
4. Right click on the selected group  click “Export LDIF” , option available at the top.
5. Click “OK” . After export is completed, It will generate a link to open the LDIF file.

Saturday 5 March 2016

Oracle Concurrent Manager FAQ's

1.What is PMON cycle and SLeep time in Concurrent Managers?


Sleep = The number of seconds ICM waits before checking new request from table FND_CONCURRENT_REQUEST.
Pmon = The number of sleep cycle ICM will wait before checking failed manager.
You can instruct the Internal Concurrent Manager to immediately verify the operating status of your individual concurrent managers, or to perform a PMON check.