What are the differences between Oracle Applications R12.1.x and R12.2.x

From DBA perspective, below are the changes in R12.2.x version of Oracle Applications, compared to R12.1.3.

1. Dual FS: Oracle Application Release 12.2 has dual filesystems(RUN FS and PATCH FS) to facilitate Online Patching.This Requires double the mountpoint space at MiddleTier Level.

2. Weblogic Server: Oracle HTTP Server in R12.1 is replaced by Oracle Weblogic Server.

10.1.3 Home is replaced by FMW_HOME.Weblogic Server Manages all oacore, forms and oafm servers in R12.2.

3. Online Patching : Dual Filesystem and EBR feature in 11gR2 Database facilitate online patching. Online Patching minimizes the downtime required for patching.adpatch utility in R12.1.x is replaced by adop utility. adop cycle runs in 5 Phases : prepare,apply,finalize,cutover and cleanup. adop internally calls adpatch utility to run the driver files associated with adop phases.

4. APPS Password Change: After Changing Apps Password in R12.2, we need to update the new password in EBS Datasource used by Weblogic Domain.To know more about apps password change in Oracle applications R12.2, check the below link.

http://www.appsdbadiaries.com/2016/02/changing-apps-password-in-r122.html

5. Cloning : While Cloning of R12.2.x adcfgclone instance needs to be run twice,  on RUN fs and PATCH fs.From 12.2.5 Version (OR R12.AD-TXK Delta 7), we have the dualfs option which creates both run fs and patch fs during clone process.

How to delete Managed Servers in R12.2

The procedure to increase/decrease the number of oacore/oafm servers in R12.2.X is different from R12.1.x.
Because in R12.2.x OPMN is replaced by Weblogic Server.


Steps to delete a managed Server in R12.2.x

1. Ensure there is no active adop cycle.

2. Shut down the managed server which needs to be deleted from the cluster/domain.

 $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop <MANAGED SERVER NAME>

Example:

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server5


3. Execute the script adProvisionEBS.pl  on the host where the managed server was created.

Source the RUN filesystem before running the command.

 perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-delete-managedserver \
-contextfile=<CONTEXT_FILE> -managedsrvname=<MANAGED_SERVER_NAME> \
-servicetype=<SERVICE_TYPE> -logfile=<LOGFILE>
Sample Output:

bash-4.1$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
> ebs-delete-managedserver \
> -contextfile=$CONTEXT_FILE -managedsrvname=oacore_server6 \
> -servicetype=oacore -logfile=$APPLRGF/TXK/delMS_oacoreserver6.log

Enter the APPS Schema password:
Enter the WebLogic AdminServer password:

ManagedServer oacore_server5 deleted successfully.

The above command deletes the managed server and also updates related parameters in context file


To Verify the entries are deleted in Context File,check the output of below command

grep -i oacore_server6 $CONTEXT_FILE


4.Execute the following command to delete details of the managed server from the OHS configuration files mod_wl_ohs.conf and apps.conf


$ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=removeMS \
-oacore=<host>.<domain>:<port> \
-oafm=<host>.<domain>:<port> \
-forms=<host>.<domain>:<port> \
-formsc4ws=<host>.<domain>:<port> \
-ekanban=<host>.<domain>:<port> \
-accessgate=<host>.<domain>:<port> \
-yms=<host>.<domain>:<port>



Sample Output:

bash-4.1$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
 -contextfile=$CONTEXT_FILE \
-configoption=removeMS -oacore=localhost.localdomain.com:7205


*** LOG FILE:
/test12/applmgr/fs1/inst/apps/test12_localhost/logs/appl/rgf/TXK/txkSetAppsConf_08009354.log

5. Restart HTTP Server

cd $ADMIN_SCRIPTS_HOME

adapcctl.sh stop

adapcctl.sh start

6. Repeat the above steps on PATCH filesystem.

adpreclone.pl error while running copyBinary.sh on a 12.2 Environment

In a 12.2.5 environment adpreclone.pl errors out while running copyBinary.sh adpreclone.pl on appsTier



$perl adpreclone.pl appsTier

                     Copyright (c) 2011, 2014 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adpreclone Version 120.31.12020000.20

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

 Checking the status of the Oracle WebLogic Administration Server....

 Running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:
perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress



 Setting the wls environment

Beginning application tier Stage - Thu Aug 18 02:43:26 2016

/test12/applmgr/fs1/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /test12/applmgr/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/test12/applmgr/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/test12/applmgr/fs1/EBSapps/comn/java/classes:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/share.jar:/test12/applmgr/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test12/applmgr/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/test12/applmgr/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test12/applmgr/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -stage /test12/applmgr/fs1/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg

Log file located at /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/StageAppsTier_08180243.log

  \     20% completed
ERROR while running Stage...
Thu Aug 18 02:50:43 2016

ERROR while running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress ...
Thu Aug 18 02:50:43 2016
[aptest12@vmohswrly128 <RUN> ]


adpreclone log files shows the below error message

Error Message:

START: Creating FMW archive.
Running /test12/applmgr/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /test12/applmgr/fs1/EBSapps/comn/clone/FMW/t2pjdk -al /test12/applmgr/fs1/EBSapps/comn/clone/FMW/FMW_Home.jar -smw /test12/applmgr/fs1/FMW_Home -ldl /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
Script Executed in 298222 milliseconds, returning status 1
ERROR: Script failed, exit code 1

Solution:

1. set the environment variable T2P_JAVA_OPTIONS as given below

$export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/test12/applmgr/temp"

2. Make sure enough free space is available in the temp directory provided above

3.Rerun adpreclone.pl on appsTier

ADOP Prepare Phase Fails with ORA-31011: XML parsing failed

While applying Patch on a test instance (R12.2.5), Prepare phase failed with error ORA-31011: XML parsing failed

Complete error message from adop logfile
==================================

[aptest12@localhost <RUN> ]$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Initializing.
    Run Edition context  : /test12/applmgr/fs2/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch edition context: /test12/applmgr/fs1/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch file system free space: 1130.34 GB

Validating system setup.
    [ERROR]     Failed to execute SQL statement:
select node_name from FND_OAM_CONTEXT_FILES
where NAME not in ('TEMPLATE','METADATA','config.txt') and
CTX_TYPE='A' and (status is null or upper(status) in ('S','F'))
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'run'
and EXTRACTVALUE(XMLType(TEXT),'//oa_service_group_status[@oa_var=''s_web_admin_status'']')='enabled'
and EXTRACTVALUE(XMLType(TEXT),'//oa_service_list/oa_service[@type=''admin_server'']/oa_service_status')='enabled'

    [ERROR]     Error Message:
    [ERROR]     ORA-31011: XML parsing failed
    [ERROR]     ORA-19202: Error occurred in XML processing
    [ERROR]     LPX-00229: input source is empty
    [ERROR]     ORA-06512: at "SYS.XMLTYPE", line 272
    [ERROR]     ORA-06512: at line 1 (DBD ERROR: error possibly near <*> indicator at char 208 in '    select node_name from FND_OAM_CONTEXT_FILES
    [ERROR]         where NAME not in ('TEMPLATE','METADATA','config.txt') and
    [ERROR]             CTX_TYPE='A' and (status is null or upper(status) in ('S','F'))
    [ERROR]             and EXTRACTVALUE(<*>XMLType(TEXT),'//file_edition_type') = 'run'
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),'//oa_service_group_status[@oa_var=''s_web_admin_status'']')='enabled'
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),'//oa_service_list/oa_service[@type=''admin_server'']/oa_service_status')='enabled'
    [ERROR]     ')
    [UNEXPECTED]Error occurred identifying Admin server node


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


Cause:

fnd_oam_context_files table has incorrect information  or the table is corrupted


Solution:


1. Login to sqlplus as applsys and take backup of fnd_oam_context_files table.


SQL> conn applsys
Enter password:
Connected.
SQL> create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;

Table created.

2. Truncate Table fnd_oam_context_files

SQL>  truncate table fnd_oam_context_files;

Table truncated.


3. Bring down MT Services

4. Run Autoconfig to populate fnd_oam_context_files  with right information.

What Happens during adop cutover Phase?

Cutover Phase Requires a brief downtime.

Following activities occur during Cutover:

+Users are logged off the system
+Finalize is run as part of cutover -if not run earlier in adop cycle
+Cutover phase cancels ADZPPATCH Concurrent Program
+Stops all Services on RUN and PATCH editions , switches filesystem and starts services on new RUN FS
+Flipping snapshots in run and patch editions.
+Users are brought back online on the patched system 

Environment is changed after cutover. Need to Re-source the env, for performing any actions after cutover.

Syntax:

$ adop phase=cutover

Multi-node Considerations
========================
By default cutover is invoked automatically on remote nodes.
If you want cutover individually:      
1. First node the cutover should be:           
$ adop phase=cutover allnodes=no       
2. For all other nodes you should do:        
$ adop phase=cutover allnodes=no action=nodb           
 * nodb because db work is already done


Cutover Parameters:
===================

mtrestart :Specifies whether or not to start Mid-Tier services at end of cutover, it allows you to perform some additional maintenance operations in the system before bringing it up.        - Takes values "yes" or "no".        - Default: yes
allnodes : Specifies whether or not to run on all the nodes.        - Takes values "yes" or "no".        - Default: yes
action: Specifies whether or not to take database action        - Takes values "db" or "nodb"        - Default: db

What is the difference between dbtechstack,dbtier and dbconfig modes in Cloning?

While Cloning Oracle Applications, adcgclone can be executed in 3 Modes on Database node.


1)perl adcfgclone.pl dbTier

 It configures the ORACLE_HOME on the target database node and  recreate the controlfiles.
 This is used in case of standby database/hot backups.   


2)perl adcfgclone.pl dbTechStack 

It configures the ORACLE_HOME on the target database tier node only. Relink the oracle home.

The below steps has to be performed manually
1. Create the Target Database control files.
2. Start the Target System Database in open mode
3. Run the library update script against the Database
cd $RDBMS_ORACLE_HOME/appsutil/install/[CONTEXT NAME]
sqlplus "/ as sysdba" @adupdlib.sql [libext]
 Where [libext] should be set to 'sl' for HP-UX, 'so' for any other UNIX platform, 
or 'dll' for Windows.

3)perl adcfgclone.pl dbconfig

It configures the database with context file. Database must be open while running adcfgclone in dbconfig mode.
e.g 
cd $RDBMS_ORACLE_HOME/appsutil/clone/bin
perl adcfgclone.pl dbconfig <Context_file>

Where Target Context File is:
$RDBMS_ORACLE_HOME/appsutil/<SID_HOSTNAME>.xml

What are the daily activities of Oracle Applications DBA?

1. Maintaining Oracle Application File System and Database using AD Utilities.
2. Starting, stopping and troubleshooting issues related to Application server components.
3. Using adadmin to relink executables, generate forms/reports,generate jar files, compile/validate APPS schema.
4. Analyzing patch pre-requisites and Applying the Patches using AD Patch Utility on Application file System and Technology Stack.
5. Applying Mini Packs, Family Packs, Maintenance Pack and Rollup patches.
6. Applying database patches using opatch utility and troubleshooting issues related to OPatch.
7. System administration activities like creating users, assigning Responsibilities, defining custom managers.
8. Changing configuration files as per the requirement.
9. Changing APPS password and Application database users’ password for security.
10. Performing Healthchecks.
11. Splicing off-cycle products using adsplice utility.
12. Cloning of Oracle Applications 11i from Production to Test Instance.
13. Taking backups- physical, logical, RMAN and performing Restore, Recovery.
14. Troubleshooting issues related to concurrent managers.
15. Managing System Configuration using Autoconfig.
16. Modifying initialization parameters as per the requirement-PFILE AND SPFILE.
17. Reorganization of tables that are fragmented.
18. Creating users and assigning appropriate roles, privileges.
19. Managing tablespaces and data files.
20. Monitoring alert log file.
21. Troubleshooting ORA- errors.
22. Analyzing AWR reports.

How to Recover lost APPS Password in Oracle Applications R12?

Have you ever forgot or lost apps password in R12. In Oracle Applications 11i, It is easy to recover apps password as it is stored in application files appsweb.cfg and wdbsvr.app.
Apps password is not available in the application files in R12. We can still recover the lost password by creating a decrypt function.

Steps to retreive apps password in R12:

1.Create function Using below syntax

SQL> create or replace FUNCTION apps.decrypt_pin_func(in_chr_key IN VARCHAR2,in_chr_encrypted_pin IN VARCHAR2) RETURN VARCHAR2
AS
LANGUAGE JAVA NAME ‘oracle.apps.fnd.security.WebSessionManagerProc.decrypt(java.lang.String,java.lang.String) return java.lang.String’;
4
5 /

Function created.


2. Find the encrypted password using below query

SQL> select ENCRYPTED_FOUNDATION_PASSWORD from apps.fnd_user where USER_NAME=’GUEST’;

ENCRYPTED_FOUNDATION_PASSWORD
——————————————————————————–
JH89E045BEA551FD01A8D71029ACA879F209U8BD898F451EBDB987C818E7608CE0CC0167BDCAF9D1D04D9C9CEE418CFE615A


3.Run the below query to decrypt the password


SQL> SELECT apps.decrypt_pin_func(‘GUEST/ORACLE’,’JH89E045BEA551FD01A8D71029ACA879F209U8BD898F451EBDB987C818E7608CE0CC0167BDCAF9D1D04D9C9CEE418CFE615A') from dual;

APPS.DECRYPT_PIN_FUNC(‘GUEST/ORACLE’,’ZG761B2FB6A2C49BC802F57E55AC26637388B4EA99
——————————————————————————–
PASSWORD


4. Verify Database Connection using the password retreived


SQL> conn apps/PASSWORD
Connected.

How to enable Trace for Application User in Oracle Applications 11i/R12?

Enabling Tracing for a Application User makes diagnostics easier to analyze.Since only the sql operations performed by a specific User are Traced.


Steps to Enable Trace for Application User:

1.Login to Oracle Applications and select the System Administrator responsibility. 
Navigate to  Profile -->  System 

2.Search for Profile Option 'Initialization SQL Statement - Custom' .Also Specify the username whom you wish to enable tracing during Search.

3.
Under username column Update the below line

BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'USER_TRACE_IDENTIFER' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;

Note : Level cane be 4,8,12 depends on your requirement 


Save the Changes done in Profile Option

4. Be careful while updating the Profile Option, Any incorrect Syntax may Prevent user from Logging in

5. Inform User to Re-login and Perform the Activity which needs to be traced.

6. Trace Files can be found in the directory specified by "user_dump_directory" parameter on the database server.
Use the identifier given in Profile Option to search the trace file in udump directory.

7. After the issue is reproduced and required trace files are collected for analysis, disable Tracing from Front End.
Its recommended to disable the trace immediately after the diagnostics are collected, as it will generate huge logfiles.

8. Navigate to system=> Profile form under System Administrator Responsibility.

Query for the username  and Profile Option name "Initialization SQL Statement - Custom"

Remove the values and make it blank.

Save the changes.

Environment Variables in oracle Applications R12.2

Below are the environment variables and their locations in R12.2 environment.

Useful adop options and syntax

1.Using analytics parameter in adop apply phase


$ adop phase=apply analytics=yes

Specifying this option will cause adop to run the following scripts and generate the associated output files (reports):

ADZDCMPED.sql - This script is used to display the differences between the run and patch editions, including new and changed objects. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>/adzdcmped.out.

ADZDSHOWED.sql - This script is used to display the editions in the system. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowed.out.

ADZDSHOWOBJS.sql - This script is used to display the summary of editioned objects per edition. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowobjs.out

ADZDSHOWSM.sql - This script is used to display the status report for the seed data manager. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowsm.out

Note: The analytics parameter should only be used when required, because of the extra processing needed.



2. flags=autoskip 

 e.g adop phase=apply patches=12345678  flags=autoskip 

 This option is an alternative for  "Continue as if it were successful"?  in adpatch, very useful in cases where patch failed to compile forms/reports and exiting.
 We need not restart the adop session just to compile a single forms (.fmb,.fmx) file.
 Make sure to review the autoskip.log logfile and fix the issues in autoskip log whenever you use autoskip flag in adop cycle.

3.  skipsyncerror=(yes|no)  [default: no]

 Specifies whether to ignore errors that may occur during incremental file system synchronization.  This might happen if you applied
 a patch in the previous patching cycle that had errors but decided to continue with the cutover.  When the patch is synchronized on
 the next patching cycle, the apply errors may occur again, but can be ignored.

4. wait_on_failed_job=(yes|no)  [default: no]

 Controls whether adop apply command exits when all workers have failed.  Instead of exiting, you can force adop to wait, and use the "adctrl" to retry failed jobs.

e.g adop phase=apply patches=7777 wait_on_failed_job=yes


5. Abort

The Online Patching Cycle can be aborted at any time prior to Cutover

e.g adop phase=abort

This is needed  if unrecoverable error happened or the user decides that patch is not needed.
If adop phase=apply failed, user should try abandon=yes first.
The abort command drops the database patch edition and  returns the system to normal runtime state.  Immediately following abort, you must also run a full cleanup and
fs_clone operation to fully remove effects of the failed online patching cycle.


            

What Happens in ADOP apply Phase

Apply Phase is executed in PATCH Filesystem
During Apply phase of an ADOP cycle, adop execute patch drivers to update Patch Edition
We can have multiple apply phases in an adop cycle, Multiple patches including customizations can be installed in this phase.

The production application is online and accessible to users. RUN filesystem is not affected by the changes.
Patches are applied to the copy (Patch Edition)
 Changes are made in the isolation of an Edition 
The running application is unaffected by these changes


workers 
specifies number of parallel workers, automatically calculated if not specified  
abandon 
abandons a patching session (if set to "yes") and must have an opposite specification to that of parameter "restart".
restart
 restarts a patching session (if set to "yes") and must have an opposite specification to that of parameter "abandon".

Example: $ adop phase=apply workers=4 abandon=yes restart=no patches=1234567,8909228



For more adop options and their usage Click here

Autopatch error: The worker should not have status 'Running' or 'Restarted' at this point

If you are restarting a failed patch session in oracle applications, sometimes you may encounter the error

AutoPatch error:
The worker should not have status 'Running' or 'Restarted' at this point.

Telling workers to quit...

All workers have quit.

Connecting to APPS......Connected successfully.

AutoPatch error:

Error running SQL and EXEC commands in parallel

Cause:

1. adpatch or adop process was killed from OS level while patch is being applied

2. Database shutdown or terminated



Solution:

Using adctrl utility, use option 4 to change the worker status to Failed


Review the messages above, then press [Return] to continue.

                    AD Controller Menu
     ---------------------------------------------------

     1.    Show worker status

     2.    Tell worker to restart a failed job

     3.    Tell worker to quit

     4.    Tell manager that a worker failed its job

     5.    Tell manager that a worker acknowledges quit

     6.    Restart a worker on the current machine

     7.    Exit


Enter your choice [1] : 4

Enter the worker number(s)/range(s) or 'all' for all workers,
or press [Return] to go back to the menu : all

Status changed to 'Failed' for worker 1.
Status changed to 'Failed' for worker 2.
Status changed to 'Failed' for worker 3.
Status changed to 'Failed' for worker 4.
Review the messages above, then press [Return] to continue.



select option "1. Show Worker Status" , The worker status will be "Failed" 

Now select option 2. "2. Tell worker to restart a failed job" 


Enter the worker number(s)/range(s) or 'all' for all workers,
or press [Return] to go back to the menu : all

Status changed to 'Fixed, restart' for worker 1.
Status changed to 'Fixed, restart' for worker 2.
Status changed to 'Fixed, restart' for worker 3.
Status changed to 'Fixed, restart' for worker 4.


Restart adpatch 

In 11i, R12.1.X version of oracle applications

when applying patch using adpatch , Select option Yes When it prompts for

'Do you wish to Continue with Previous adpatch Session'

In R12.2.x, Use restart and abandon parameters


adop phase=apply patches=123456 restart=yes abandon=no 

To restart the patch session from where it failed.

ADOP Patching Fails with aipspv(): Error setting up PDI list for upgrade

While Applying patch on a 12.2.5 environment using online patch cycle in adop, Autopatch Failed with error

AutoPatch error:
Invalid password for schema "CUST1" (oracle_id = 20087)

AutoPatch error:
Failed getting PDI list for product 'xx1'

AutoPatch error:
aipspv(): Error setting up PDI list for upgrade

Root Cause
=========

Password for Application schema is not changed using FNDCPASS

Solution
========

1.Change the password of custom application with FNDCPASS 

Syntax for FNDCPASS

$ FNDCPASS apps/<PASSWORD> 0 Y system/<PASSWORD> ORACLE CUST1 <NEWPASSWORD>
Log filename : L********.log


Report filename : O*****.out

2. Verify the password change is successful

$ cat L*******.log
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.2

Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

FNDCPASS:
+---------------------------------------------------------------------------+

Current system time is 10-MAR-2016 23:21:30

+---------------------------------------------------------------------------+

Working...
Password is changed successfully for user BOLINF.
FNDCPASS completed successfully.

+---------------------------------------------------------------------------+
Concurrent request completed successfully
Current system time is 10-MAR-2016 23:21:30

+---------------------------------------------------------------------------+



Please Note, In 12.2.x Environments FNDCPASS must be executed on RUN Filesystem.

3. Restart the online patching cycle 

While restarting adop apply phase after a previously failed session, use options abandon and restart


adop phase=apply patches=123456 restart=yes abandon=no 

[Use this command if you wish to continue the previous patch session from the step where it failed]

adop phase=apply patches=123456 restart=no abandon=yes


[Use this command if you wish to start a fresh patch session]




If changing password using FNDCPASS does not work, analyse further details about database user and application user.

Check if the database user and application user are defined correctly using the sql queries below

SQL> select ORACLE_ID from FND_ORACLE_USERID where oracle_username like '%CUST1%';

 ORACLE_ID
----------
     2008

SQL> select application_id from fnd_application where application_short_name = 'XX1';

APPLICATION_ID
--------------
         2003

SQL> select * from fnd_product_installations where oracle_id=2008;


If the user details are not correct from the output of above sql, Please correct them.

ORA-12504 While running Autoconfig on Database Server

After Oracle Database upgrade from 11.2.0.3 to 11.2.0.4, Autoconfig errored on Database server with the below error

[AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
      <filename>  <phase>  <return code where appropriate>

  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /test1/oracle/product/11204/appsutil/install/test1_host1
      afdbprf.sh              INSTE8_PRF         1

  [APPLY PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /test1/oracle/product/11204/appsutil/install/test1_host1
      adcrobj.sh              INSTE8_APPLY       1


AutoConfig is exiting with status 2

====Running the failed script has given the ORA-12504 error

java.lang.OutOfMemoryError: Java heap space error in Output PostProcessor

Possible causes for Java Heap Space error in OPP

1. There is limited heap space in the JVM to process the report
2. The Java application running is out of memory
3. The scalable feature isn't enabled
4. An older version of the JRE is being used in the configuration, 
5. There is insufficient space in the designated temp directory for processing the report, and reports are embedded with images which require a lot of space that unavailable.


The log will contain a java stack error similar to the following:
java.lang.reflect.InvocationTargetException
Caused by: java.lang.OutOfMemoryError: Java heap space
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:2869)
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:3000)
at oracle.xdo.parser.v2.XMLDocument.createTextNode(XMLDocument.java:735)
at oracle.xdo.parser.v2.DocumentBuilder.characters(DocumentBuilder.java:553)
at oracle.xdo.parser.v2.NonValidatingParser.reportCharacters(NonValidatingParser.java:1680)
at oracle.xdo.parser.v2.NonValidatingParser.parseText(NonValidatingParser.java:1391)
at oracle.xdo.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1268)


How to resolve Java Heap Space error in Output Post Processor?


Solution 1
==========


1. Determine what the heap size per OPP process is currently:
select DEVELOPER_PARAMETERS from FND_CP_SERVICES
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

2. The default should be:
J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m

3. Increase the Heap Space per Process to 1024:
update FND_CP_SERVICES
set DEVELOPER_PARAMETERS =
'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

4. Bring the managers down.
5. Please run the Concurrent Manager Recovery feature to address any Concurrent Manager / Concurrent Processing issues within the Oracle Application Manager.  NOTE: Regular use of the CP Analyzer in Note 1411723.1 can provide an Immediate Analysis and Output of a Concurrent Environment.
6. Bring the managers up again.

OR

Solution 2
==========
1. Log into applications with the System Administrator responsibility.
2. Navigate to Concurrent -> Program -> Define
3. Query the XML Publisher Template Re-Generator program
4. Set the following value for the Executable Options : -Xmx1024m
5. Save changes.
6. Retest the program.

OR


Solution 3
==========

Configure the XML Publisher Administrator Configuration settings.
1. As XML Publisher Administrator navigate to Administration->Configuration.
2. Under Temporary Directory pick a temporary file location on your concurrent processing node. This should be at least 5GB or 20x larger than largest XML data file you generate
3.  Under FO Processing, set:
o Use XML Publisher's XSLT processor set to True
o Enable scalable feature of XSLT processor set to False
o Enable XSLT runtime optimization set to True 

Downtime Mode feature in ADOP Patching

Downtime mode is introduced in E-Business Suite Release 12.2, to minimize the duration of 12.2 upgrade process.This feature is available from AD.Delta.5 Release Update Pack.

Before applying patch in downtime mode, all Application services must be shutdown.Downtime mode does not use online patching cycle phases,hence it applied patches quickly compared to online patching. However, EBS application will be down for longer time in downtime mode.Make sure that no adop cycle is in progress while applying patch in downtime mode, Otherwise it will give error.

To apply patch in downtime mode. Follow the below steps

1. Shut down Application Services

2. Apply patch using the below command

 $ adop phase=apply patches=<patch_number> apply_mode=downtime

3. Startup Application Services


Following Scenarios require to apply patch in downtime mode in R12.2

1. All Patching that is part of 12.2 upgrade process. Once the upgrade is complete and users are online, all subsequent patching on a production system should use online mode, unless stated in patch ReadMe.

2. Single-node development or test environments, where production support and high availability are not required.

Limitations of downtime mode

1. We cannot test the patch application prior to cutover as in online patching cycle.
2. We cannot rollback patch applied using downtime mode.
3. Downtime mode is not applicable for Multi-node environments

How to Unlock Objects Statistics?

While running Concurrent Program, Gather Schema Statitics the request failed with error
20005: object statistics are locked (stattype = ALL)***

Logfile:

start of log messages from FND_FILE

+----------------------------------------------------------------------

-----+

In GATHER_SCHEMA_STATS , schema_name= APPS percent= 10 degree = 2 

internal_flag= NOBACKUP

Error #1: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.CZ_MESSAGE_QENTRIES***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #2: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.ECX_INQUEUE***ORA-20005: 

object statistics are locked (stattype = ALL)***

Error #3: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.ECX_IN_OAG_Q_TABLE***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #4: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.ECX_OUTQUEUE***ORA-20005: 

object statistics are locked (stattype = ALL)***

Error #5: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_CRMXRWD_REQ_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #6: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_CRMXRWD_RES_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #7: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_CRMXRWL_REQ_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #8: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_CRMXRWL_RES_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #9: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_GTREQ_QTBL***ORA-20005: 

object statistics are locked (stattype = ALL)***

Error #10: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_GTRES_QTBL***ORA-20005: 

object statistics are locked (stattype = ALL)***

Error #11: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_HM000A_REQ_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #12: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_HM000A_RES_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #13: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_JTF_REQ_Q_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #14: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_JTF_RES_Q_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #15: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_RAPID_BP_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #16: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_RAPID_B_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #17: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_RAPID_ER_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #18: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_RAPID_MP_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #19: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_FM_RAPID_M_QTBL***ORA-

20005: object statistics are locked (stattype = ALL)***

Error #20: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_IH_BULK_QTBL***ORA-20005: 

object statistics are locked (stattype = ALL)***

Error #21: ERROR: While GATHER_TABLE_STATS:

                        object_name=APPS.JTF_PF_LOGGING_TABLE***ORA-

20005: object statistics are locked (stattype = ALL)***

+----------------------------------------------------------------------

-----+

End of log messages from FND_FILE

+----------------------------------------------------------------------

-----+

Solution:

This error comes when analyzing tables in oracle. Basic issue in this error is  when you
import table without data i.e structure/schema only, oracle will lock table statistics.
You can view all the locked tables in schema by executing following query:

select table_name, stattype_locked 
from dba_tab_statistics 
where owner = ‘APPS’ and stattype_locked is not null;

Then how to unlock them, following query will help us to sort this issue:

select ‘exec DBMS_STATS.UNLOCK_TABLE_STATS (”’|| owner ||”’,”’|| table_name 
||”’);’ from dba_tab_statistics where owner = ‘MBS’ and stattype_locked is not null;

or you can do that for every single table:

exec DBMS_STATS.UNLOCK_TABLE_STATS(‘owner’,'table name’);