HTTP 404 error after logging into Oracle EBS 12.2.6 url

Issue Description

We recently upgraded the Oracle EBS Version to 12.2.6 from 12.1.3. After Upgrade, one user reported he is getting erro HTTP 404 when he login to the application url.



Cause:

User has set his preferences to load a Custom Start Page after login. The custom html was not available after the upgrade.Hence he is getting error HTTP 404 as in screenshot. He is unable to reset the preferences as the page is not loading.

Solution:

In this case, only option is to modify the profile option 'Application Start Page' for the specific user  to a correct value.

Oracle EBS 12.2.x: ADOP error during Cutover phase

Background of the issue: I was applying hrblobal driver in a multi-node Production environment. Our Production application EBS has one Primary node and one DMZ Node.

After applying hrglobal driver using the online patching cycle, Cutover phase thrown below error message

==

Applying patch(es): <6390830> on node EBSDMZ failed.
If you choose to proceed with cutover, node EBSDMZ will be marked as abandoned.
Do you want adop to continue processing on completed nodes [y/n]? n
    As per user choice, adop will not proceed further with processing.

==

If we Provide input as 'Y' above, EBSDMZ node will be abandoned and we need to register the DMZ node again.


Solution:

1.Apply Patch on DMZ node using options=nodb and restart the cutover phase

On DMZ Node
=====
Source the patch filesystem
adop phase=apply patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv allnodes=NO action=nodb options=forceapply

2. Restart Cutover phase on Master node

Oracle Applications Patching :How to Run hrglobal driver in Oracle EBS 12.2.X Version.

Applying hrglobal driver is needed during an oracle applications upgrade or when the  payroll data is required to be analyzed as per latest code pack.

Below are the steps to apply hrglobal driver in a Oracle EBS environment (12.2.x)

Before applying hrglobal driver download the latest hrglobal patch from My Oracle Support.

Start an Oracle online EBS Patching cycle.

1.adop phase=prepare

2. Apply latest hrglobal patch

adop phase=apply patches=<patchnum>

3. Run Datainstall and hrglobal driver using below commands

ADOP utillity is intelligent enough to switch to Patch Filesystem depending on the ADOP phase being run.
For any Other manual operations, we need to set the environment to point to Patch Filesystem explicitly.

. ./EBSapps.env PATCH

+Run DataInstall

java oracle.apps.per.DataInstall apps appspassword thin test.domain.com:1521:DEV

Select the Required localisations and save the changes.

Example of Data Install Changes Summary is shown below.



          DataInstall - Actions confirmation

Do you really wish to exit and save your changes?

      [Y]      - Yes, save then exit
      [N]      - No, don't save but exit
      [Return] - To return to the DataInstall Main Menu

Enter your choice (for example Y) : Y


          DataInstall - Actions summary
          -----------------------------

The following actions will be performed:


Localisation         Product(s)               Leg. Data? Action
-------------------- ------------------------ ---------- -------------
Global               Human Resources          Installed  Install
United Arab Emirates Human Resources          Installed  Install
United Arab Emirates Payroll                  Installed  Install

Localisation   College Data? Action
-------------- ------------- -------------
United Kingdom
United States

Option                         Data?         Action
------------------------------ ------------- -------------
JIT/Geocode


Legislation                            Action
------------------------------         -------------
ALL  All Legislations


++Apply hrglobal.drv using below syntax.

adop phase=apply patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv workers=8

4. adop phase=finalize

5. adop phase=cutover

6. adop phase=cleanup

Oracle EBS Concurrent Manager Troubleshooting

In Oracle EBS version 12.2.6, Internal concurrent manager had an unexpected Outage.
ICM logfile has the error message  ORA-04068: existing state of packages has been discarded
====================

Shutting down Internal Concurrent Manager : 30-JAN-2018 12:09:54

List of errors encountered:
.............................................................................

_ 1 _
Error in DBMS lock request by handle. (ROUTINE=AFPGMG) (REASON=ORACLE
error 4068 in afpdlrh

Cause: afpdlrh failed due to ORA-04068: existing state of packages has
been discarded
ORA-04061: existing state of package body "APPS.FND_DCP" has been
invalidate


==========================================
Issue Resolution

1. Shut down the Concurrent manager completely using adcmctl.sh

cd $ADMIN_SCRIPTS_HOME

adcmctl.sh stop apps/<appspassword>

2. Compile the invalids in database

SQL>alter package FND_DCP compile body;

OR

Run the utlrp.sql script to recompile all the invalid objects in the database.

3. Startup Concurrent manager using adcmctl.sh

cd $ADMIN_SCRIPTS_HOME

adcmctl.sh start apps/<appspassword>

4. Verify Oracle EBS Concurrent manager is running fine.

ADOP Options: cleanup_mode

ADOP cleanup_mode

Run with adop cleanup phase.

Provides control over the extent of cleanup operations.

If no value is specified, cleanup is performed in standard mode, which does the same as quick mode but also drops obsolete code objects.

Values: quick/full

Purpose:

cleanup_mode=quick performs the minimum necessary cleanup to complete the online patching cycle. Use this mode if you want to start a new online patching cycle as soon as possible.

cleanup_mode=full performs all possible cleanup tasks for maximum recovery of space from unused database editions, objects and columns. Full cleanup is required after aborting an online patching cycle.

Usage:

adop phase=cleanup cleanup_mode=full

adop phase=cleanup cleanup_mode=quick

ADOP Options: actualize_all

Whenever adop prepare phase is initiated, a new patch edition is created in the database.During Online patching (ADOP) : An additional column ZD_EDITION_NAME is populated in the seed tables.As we do more online patching cycles, the number of entries for database editions will increase. This affects system performance and also increases cleanup time.When the number of old database editions reaches 25 or more, you should consider dropping all old database editions by running the adop actualize_all phase and then performing a full cleanup.

To drop older database editions, follow the below steps:

$ adop phase=prepare
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cutover
$ adop phase=cleanup cleanup_mode=full

How to Restore context file in oracle applications?

Scenario: Your Applications context file is either accidentally deleted or Corrupted. You do not have any latest backup of context file available.

Below are the steps to recreate context file 

The Applications context file can be retrieved by running the adclonectx.pl script.

perl /clone/bin/adclonectx.pl retrieve

Above script prompts for Database connection Info, apps username/password , Available Context files and the location to which context files can be restored.When prompted for the context file to be retrieved, select the option of retrieving the Applications tier context file that has been lost and retrieve it to the default location as in $CONTEXT_FILE parameter.

The above command can be used only when INST_TOP the is still intact. Incase if INST_TOP is also lost, the Applications tier context file may be retrieved as follows:

Execute the following command on the Database tier:

perl /appsutil/clone/bin/adclonectx.pl retrieve

When prompted for the context file to be retrieved, select the option of retrieving the Applications tier context file that has been lost.Choose the location to any writeable location and copy the restored context file to the value specified in $CONTEXT_FILE parameter on Application Tier.

To retrieve the Database tier context file,

Execute the following command on the Database tier:

perl /appsutil/clone/bin/adclonectx.pl retrieve

When prompted for the context file to be retrieved, select the Database tier context file and retrieve it to the default location ($CONTEXT_FILE on DB node).

Oracle Applications R12.2.5 New Features: dualfs in cloning

From Oracle Applications Release 12.2.5 Version, we have dualfs option for cloning applications tier.We can use this feature in earlier 12.2.x versions also, if AD-TXK Delta 7 patch is applied.Before AD-TXK Delta 7, it was required to run adcfgclone twice on run and patch filesystems.Using dualfs option,we can create both run and patch fs at one time.

-bash$ perl adcfgclone.pl appsTier dualfs

                     Copyright (c) 2002, 2015 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adcfgclone Version 120.63.12020000.56

Enter the APPS password :

Enter the Weblogic AdminServer password :

Do you want to add a node (yes/no) [no] :


Running: Context clone...

Log file located at /u01/VISN/fs1/EBSapps/comn/clone/bin/CloneContext_1023194859.log

Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [appsmt] :

Target System Database SID : VISN

Target System Database Server Node [r12erpt] : appsdb

Target System Database Domain Name [domain.com] :

Target System Base Directory : /u01/VISN

Target System Base Directory set to /u01/VISN

Target System Current File System Base set to /u01/VISN/fs1

Target System Other File System Base set to /u01/VISN/fs2

Target System Fusion Middleware Home set to /u01/VISN/fs1/FMW_Home
Target System Other File System Fusion Middleware Home set to /u01/VISN/fs2/FMW_Home

Target System Web Oracle Home set to /u01/VISN/fs1/FMW_Home/webtier
Target System Other File System Web Oracle Home set to /u01/VISN/fs2/FMW_Home/webtier

Target System Appl TOP set to /u01/VISN/fs1/EBSapps/appl
Target System Other File System Appl TOP set to /u01/VISN/fs2/EBSapps/appl

Target System COMMON TOP set to /u01/VISN/fs1/EBSapps/comn
Target System Other File System COMMON TOP set to /u01/VISN/fs2/EBSapps/comn

Target System Instance Home Directory [V] : /u01/VISN

Target System Current File System Instance Top set to /u01/VISN/fs1/inst/apps/VISN_appsmt

Do you want to preserve the Display [appsmt:0.0] (y/n)  : n

Target System Display [r12erpt:0.0] : appsmt:3.0

Target System Root Service [enabled] :

Target System Web Entry Point Services [enabled] :

Target System Web Application Services [enabled] :

Target System Batch Processing Services [enabled] :

Target System Other Services [disabled] :


Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 88

Checking the port pool 88
done: Port Pool 88 is free
Report file located at /u01/VISN/fs1/inst/apps/VISN_appsmt/admin/out/portpool.lst

UTL_FILE_DIR on database tier consists of the following directories.

1. /usr/tmp
2. /usr/tmp
3./u01/VISN/product/12.1.0/appsutil/outbound/VISN_appsmt
4. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1] : 3
The new APPL_TOP context file has been created :
  /u01/VISN/fs1/inst/apps/VISN_appsmt/appl/admin/VISN_appsmt.xml
Check Clone Context logfile /u01/VISN/fs1/EBSapps/comn/clone/bin/CloneContext_1023194859.log for details.

Creating Patch file system context file.....

Log file located at /u01/VISN/fs1/EBSapps/comn/clone/bin/CloneContextPatch_1023194973.log

Target System Other File System Instance Top set to /u01/VISN/fs2/inst/apps/VISN_appsmt

Target System Port Pool [0-99] : 89

Checking the port pool 89
done: Port Pool 89 is free
Report file located at /u01/VISN/fs2/inst/apps/VISN_appsmt/admin/out/portpool.lst
The new APPL_TOP context file has been created :
 /u01/VISN/fs2/inst/apps/VISN_appsmt/appl/admin/VISN_appsmt.xml
Check Clone Context logfile /u01/VISN/fs1/EBSapps/comn/clone/bin/CloneContextPatch_********.log for details.

FMW Pre-requisite check log file location : /u01/VISN/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log
Running: FMW pre-req check...

Configuring: Run file system....
LogFile located at /u01/VISN/fs1/inst/apps/VISN_appsmt/admin/log/clone/run/RCloneApplyAppstier_*******.log

What are the differences between Oracle Applications R12.1.x and R12.2.x

From DBA perspective, below are the changes in R12.2.x version of Oracle Applications, compared to R12.1.3.

1. Dual FS: Oracle Application Release 12.2 has dual filesystems(RUN FS and PATCH FS) to facilitate Online Patching.This Requires double the mountpoint space at MiddleTier Level.

2. Weblogic Server: Oracle HTTP Server in R12.1 is replaced by Oracle Weblogic Server.

10.1.3 Home is replaced by FMW_HOME.Weblogic Server Manages all oacore, forms and oafm servers in R12.2.

3. Online Patching : Dual Filesystem and EBR feature in 11gR2 Database facilitate online patching. Online Patching minimizes the downtime required for patching.adpatch utility in R12.1.x is replaced by adop utility. adop cycle runs in 5 Phases : prepare,apply,finalize,cutover and cleanup. adop internally calls adpatch utility to run the driver files associated with adop phases.

4. APPS Password Change: After Changing Apps Password in R12.2, we need to update the new password in EBS Datasource used by Weblogic Domain.To know more about apps password change in Oracle applications R12.2, check the below link.

http://www.appsdbadiaries.com/2016/02/changing-apps-password-in-r122.html

5. Cloning : While Cloning of R12.2.x adcfgclone instance needs to be run twice,  on RUN fs and PATCH fs.From 12.2.5 Version (OR R12.AD-TXK Delta 7), we have the dualfs option which creates both run fs and patch fs during clone process.

ADOP Useful Options :skipsyncerror

There are Various options available with Oracle Applications Online Patching Utility (ADOP). Here , we will see the benefits of skipsyncerror option.

Scenario: 

Your Previous  adop session failed while applying patches and you did not find any solution to fix the errors, Oracle Support Provided a new patch to fix the issue. You will get errors while synchronization and need to ignore those errors.

Usage: adop phase=prepare skipsyncerror=yes


skipsyncerror Details

This option is used along with prepare phase.

Purpose:

This feature enables the user to specify that any synchronization errors in the prepare phase are expected to be fixed automatically in the synchronization that takes place with subsequent patches.

Values: yes/no

Default value is 'no'. Set the value to 'yes' in order to work around synchronization failures that may occur when patches that failed to apply correctly in a previous patching cycle are synchronized during the prepare phase.

How to delete Managed Servers in R12.2

The procedure to increase/decrease the number of oacore/oafm servers in R12.2.X is different from R12.1.x.
Because in R12.2.x OPMN is replaced by Weblogic Server.


Steps to delete a managed Server in R12.2.x

1. Ensure there is no active adop cycle.

2. Shut down the managed server which needs to be deleted from the cluster/domain.

 $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop <MANAGED SERVER NAME>

Example:

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server5


3. Execute the script adProvisionEBS.pl  on the host where the managed server was created.

Source the RUN filesystem before running the command.

 perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-delete-managedserver \
-contextfile=<CONTEXT_FILE> -managedsrvname=<MANAGED_SERVER_NAME> \
-servicetype=<SERVICE_TYPE> -logfile=<LOGFILE>
Sample Output:

bash-4.1$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
> ebs-delete-managedserver \
> -contextfile=$CONTEXT_FILE -managedsrvname=oacore_server6 \
> -servicetype=oacore -logfile=$APPLRGF/TXK/delMS_oacoreserver6.log

Enter the APPS Schema password:
Enter the WebLogic AdminServer password:

ManagedServer oacore_server5 deleted successfully.

The above command deletes the managed server and also updates related parameters in context file


To Verify the entries are deleted in Context File,check the output of below command

grep -i oacore_server6 $CONTEXT_FILE


4.Execute the following command to delete details of the managed server from the OHS configuration files mod_wl_ohs.conf and apps.conf


$ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=removeMS \
-oacore=<host>.<domain>:<port> \
-oafm=<host>.<domain>:<port> \
-forms=<host>.<domain>:<port> \
-formsc4ws=<host>.<domain>:<port> \
-ekanban=<host>.<domain>:<port> \
-accessgate=<host>.<domain>:<port> \
-yms=<host>.<domain>:<port>



Sample Output:

bash-4.1$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
 -contextfile=$CONTEXT_FILE \
-configoption=removeMS -oacore=localhost.localdomain.com:7205


*** LOG FILE:
/test12/applmgr/fs1/inst/apps/test12_localhost/logs/appl/rgf/TXK/txkSetAppsConf_08009354.log

5. Restart HTTP Server

cd $ADMIN_SCRIPTS_HOME

adapcctl.sh stop

adapcctl.sh start

6. Repeat the above steps on PATCH filesystem.