ADOP fs_clone failing with error : "Could not find patch context file from database"

 Issue:

FS_CLONE is failing with error :Could not find patch context file from database
Error details:
$adop phase=fs_clone
==============
Checking for existing adop sessions.
    No pending session exists.
    Starting new adop session.
===========================================================================
ADOP (C.Delta.12)
Session ID: 43
Node: test1.domain1.com
Phase: fs_clone
Log: /test/erp/apps/fs_ne/EBSapps/log/adop/32/20220121_145748/adop.log
===========================================================================
Verifying existence of context files in database.
    [UNEXPECTED] Could not find patch context file from database
    [UNEXPECTED]Patch edition context file not found in database for host test1.domain1.com

Summary report for current adop session:
    Node test1.domain1.com:
       - Fs_clone status:   Not Started
    For more details, run the command: adop -status -detail

[STATEMENT] Please run adopscanlog utility, using the command
"adopscanlog -latest=yes"
to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 2 (Fail)

=========================
Cause:
===
Patch context file not found in fnd_oam_context_files tables.

Solution:
=======
1.Source the patch file system and run autoconfig on patch filesystem to update context file in database.
or
2. Run CtxSynchronizer to upload patch context file to database.

$ $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=<patch context file name and location> logfile=/tmp/patchctxupload.log

Provide apps password when prompted.

"APP-FND-02704: Unable to alter user APPS_NE to change password" while changing APPS password using FNDCPASS

 Issue: 

Changing apps password using FNDCPASS fails with below error

APP-FND-02704: Unable to alter user APPS_NE to change password.
Oracle error 28003:  has been detected in alterpassword2.

Command to change APPS Password.[Ensure to shut down Application services before changing apps password]
FNDCPASS apps/<apps_pwd> 0 Y system/<password> SYSTEM APPLSYS <new_apps_password>
Error Message from Logfile:
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.2
Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.
FNDCPASS:
+---------------------------------------------------------------------------+
Current system time is 10-APR-2020 18:43:36
+---------------------------------------------------------------------------+
Arguments
   FNDCPASS system/***** SYSTEM APPLSYS *****
+----------------------------------------------------------------------------+
Working...
APP-FND-02704: Unable to alter user APPS_NE to change password.
Oracle error 28003:  has been detected in alterpassword2.
+---------------------------------------------------------------------------+
Concurrent request completed
Current system time is 10-APR-2020 18:43:39
+---------------------------------------------------------------------------+

Cause:
Issue is due to Password_verify_function enables in dba_profiles for APPS,APPLSYS,APPS_NE users

Solution:

Disable the password verify function for APPS,APPLSYS,APPS_NE users in database profiles.
1. Identify the database profile for APPS,APPLSYS,APPS_NE users using below query.

SELECT USERNAME,PROFILE FROM DBA_USERS where USERNAME in ('APPS','APPLSYS','APPS_NE');
SELECT PROFILE,VALUE FROM DBA_PROFILES WHERE  PROFILE='&profile for user';

2. Disable password verify function

alter profile <profile_name> limit PASSWORD_VERIFY_FUNCTION null;

3. Run the FNDCPASS command to change password.

FNDCPASS apps/<apps_pwd> 0 Y system/<password> SYSTEM APPLSYS <new_apps_password>

4. After changing apps password, run autoconfig on Middle tier and update EBS datasource for R12.2.x

You can follow the below steps to update apps password in EBS Datasource:

http://www.appsdbadiaries.com/2022/02/how-to-update-apps-password-in-ebs.html

5. Startup Application services

4. Revert back the changes done in step 2

How to change weblogic password from Backend in EBS 12.2.x?

On EBS 12.2.X environments , if you are on R12.AD.C.Delta.7 and R12.TXK.C.Delta.7 or later, we can change Weblogic admin server password using the below script.

1. Startup admin server. Do not start any other services

$. ./EBSapps.env RUN
$ cd $ADMIN_SCRIPTS_HOME
$adadminsrvctl.sh start

2. Run the below script  

perl $FND_TOP/patch/115/bin/txkUpdateEBSDomain.pl -action=updateAdminPassword

[The script will prompt for Current and New Weblogic Password and restarts Weblogic admin server using the new password.]

3. Run fs_clone to propagate the changes to the alternate file system (Patch File system).

$adop phase=fs_clone

How to Update APPS password in EBS Datasource (EBS 12.2.x)?

 From EBS 12.2.x versions, whenever we change apps password we need to update the new apps password in EBS Datasource.
Otherwise managed servers wont come up.
Steps to update apps password in Weblogic datasource for EBS 12.2x, post AD-TXK 7:
1. Start Only Admin server,Do not start any other services
$. ./EBSapps.env RUN
$ cd $ADMIN_SCRIPTS_HOME
$adadminsrvctl.sh start

2. Update apps password in Datasource from backend:
$perl $FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl
When prompted select the "updateDSPassword" option.
Sample screen output:

perl $FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl
Please select from list of valid options
        updateDSPassword - Update WebLogic Datasource Password
        updateDSJdbcUrl  - Update WebLogic Datasource Connection String
Enter Your Choice : updateDSPassword
Enter the full path of Applications Context File [DEFAULT -]:
Enter weblogic admin server password:
Enter the APPS user password:

Note: We can also update the apps password in EBS Datasource from Weblogic console. 

adpreclone.pl Error on Applications Tier

Issue:

adpreclone.pl appsTier on EBS 12.2.x environment fails with error

ERROR: The context variable s_apps_jdbc_connect_descriptor is null

Cause:

Value of parameter "s_apps_jdbc_connect_descriptor" is null in Applications Context file.

Solution:

1. Take Backup of context file

2. Update the value of parameter "s_apps_jdbc_connect_descriptor" with JDBC connect string to connect to the database.

3. Run Autoconfig on Applications node.

4. Restart adpreclone.pl appsTier

Autoconfig Error : ORA-01422: exact fetch returns more than requested number of rows

Issue: Autoconfig on EBS 12.2 environment fails while running script txkGenADOPWrapper.pl

Below is the error message  from logfile

=======================

Script Name    : txkGenADOPWrapper.pl

Script Version : 120.0.12020000.3

ERROR DESCRIPTION:

(*******FATAL ERROR*******

PROGRAM : (/test/app/fs1/inst/apps/test_mtnode1/admin/install/txkGenADOPWrapper.pl)

TIME    : Wed Nov 25 04:22:34 2020

FUNCTION: TXK::SQLPLUS::_doExecute [ Level 3 ]

MESSAGES:

SQLPLUS error: buffer=

 

SQL*Plus: Release 10.1.0.5.0 - Production on Wed Nov 25 04:22:33 2020

 

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

 

SQL> SQL> Connected.

SQL> SELECT ad_zd_adop.get_node_type('mtnode1') FROM DUAL

       *

ERROR at line 1:

ORA-01422: exact fetch returns more than requested number of rows

ORA-06512: at "APPS.AD_ZD_ADOP", line 3010

=======================

Cause:

FND_NODES table has incorrect node information.

Found an extra row in FND_NODES table where node_name is null after clone.

Solution:

1. Clean up FND_NODES table

Login to database as apps user

sqlplus apps/<password>

    

sql>exec FND_CONC_CLONE.SETUP_CLEAN;

2.Run Autoconfig on Database node(s).

cd $ORACLE_HOME/appsutil/bin

adconfig.sh

[Provide Database Context file  and apps Password]

3. Run Autoconfig on apps tier

Login to Application node 

Source the RUN filesystem environment file.

cd $AD_TOP/bin

adconfig.sh

[Provide Applictions Context file and apps Password]

RMAN-03009: failure of allocate command on ORA_AUX_SBT_TAPE_1 channel

This is an error during Database Clone using RMAN. I was trying to restore database using RMAN Backup 

RMAN-00571: ===========================================================

RMAN-00569: ===============
ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command
RMAN-05501: aborting duplication of target database

RMAN-03009: failure of allocate command on ORA_AUX_SBT_TAPE_1 channel 
ORA-19554: error allocating device, device type: SBT_TAPE, device name:

ORA-27211: Failed to load Media Management Library

Additional information: 

Cause: Backup Configuration was modified.

Solution:
Use the below command to Resolve the error and continue with Database Restore

rman auxiliary / 
run
{
allocate auxiliary channel c1 type disk;
allocate auxiliary channel c2 type disk;
duplicate database to 'DEV' backup location '/Daily_Backup/Backup_16042018';
}
exit;
EOF

adcfgclone.pl script failed during instantiate OHS Stage

While cloning 12.2.6 environment, adcfgclone.pl script failed during instantiate OHS Stage.

Issue is because opmn components were unable to start up with below error messages.

globalInitNLS: NLS boot file not found or invalid
 -- default linked-in boot block used
XML parser init: error 201.
globalInitNLS: NLS boot file not found or invalid
 -- default linked-in boot block used

Further analysis on the clone logfiles for OHS creation shown that ORA_NLS10 parameter is unset before cloning.

Resolution:

$unset ORA_NLS10

+Restart the failed script

Oracle Applications R12.2.5 New Features: dualfs in cloning

From Oracle Applications Release 12.2.5 Version, we have dualfs option for cloning applications tier.We can use this feature in earlier 12.2.x versions also, if AD-TXK Delta 7 patch is applied.Before AD-TXK Delta 7, it was required to run adcfgclone twice on run and patch filesystems.Using dualfs option,we can create both run and patch fs at one time.

-bash$ perl adcfgclone.pl appsTier dualfs

                     Copyright (c) 2002, 2015 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adcfgclone Version 120.63.12020000.56

Enter the APPS password :

Enter the Weblogic AdminServer password :

Do you want to add a node (yes/no) [no] :


Running: Context clone...

Log file located at /u01/VISN/fs1/EBSapps/comn/clone/bin/CloneContext_1023194859.log

Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [appsmt] :

Target System Database SID : VISN

Target System Database Server Node [r12erpt] : appsdb

Target System Database Domain Name [domain.com] :

Target System Base Directory : /u01/VISN

Target System Base Directory set to /u01/VISN

Target System Current File System Base set to /u01/VISN/fs1

Target System Other File System Base set to /u01/VISN/fs2

Target System Fusion Middleware Home set to /u01/VISN/fs1/FMW_Home
Target System Other File System Fusion Middleware Home set to /u01/VISN/fs2/FMW_Home

Target System Web Oracle Home set to /u01/VISN/fs1/FMW_Home/webtier
Target System Other File System Web Oracle Home set to /u01/VISN/fs2/FMW_Home/webtier

Target System Appl TOP set to /u01/VISN/fs1/EBSapps/appl
Target System Other File System Appl TOP set to /u01/VISN/fs2/EBSapps/appl

Target System COMMON TOP set to /u01/VISN/fs1/EBSapps/comn
Target System Other File System COMMON TOP set to /u01/VISN/fs2/EBSapps/comn

Target System Instance Home Directory [V] : /u01/VISN

Target System Current File System Instance Top set to /u01/VISN/fs1/inst/apps/VISN_appsmt

Do you want to preserve the Display [appsmt:0.0] (y/n)  : n

Target System Display [r12erpt:0.0] : appsmt:3.0

Target System Root Service [enabled] :

Target System Web Entry Point Services [enabled] :

Target System Web Application Services [enabled] :

Target System Batch Processing Services [enabled] :

Target System Other Services [disabled] :


Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 88

Checking the port pool 88
done: Port Pool 88 is free
Report file located at /u01/VISN/fs1/inst/apps/VISN_appsmt/admin/out/portpool.lst

UTL_FILE_DIR on database tier consists of the following directories.

1. /usr/tmp
2. /usr/tmp
3./u01/VISN/product/12.1.0/appsutil/outbound/VISN_appsmt
4. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1] : 3
The new APPL_TOP context file has been created :
  /u01/VISN/fs1/inst/apps/VISN_appsmt/appl/admin/VISN_appsmt.xml
Check Clone Context logfile /u01/VISN/fs1/EBSapps/comn/clone/bin/CloneContext_1023194859.log for details.

Creating Patch file system context file.....

Log file located at /u01/VISN/fs1/EBSapps/comn/clone/bin/CloneContextPatch_1023194973.log

Target System Other File System Instance Top set to /u01/VISN/fs2/inst/apps/VISN_appsmt

Target System Port Pool [0-99] : 89

Checking the port pool 89
done: Port Pool 89 is free
Report file located at /u01/VISN/fs2/inst/apps/VISN_appsmt/admin/out/portpool.lst
The new APPL_TOP context file has been created :
 /u01/VISN/fs2/inst/apps/VISN_appsmt/appl/admin/VISN_appsmt.xml
Check Clone Context logfile /u01/VISN/fs1/EBSapps/comn/clone/bin/CloneContextPatch_********.log for details.

FMW Pre-requisite check log file location : /u01/VISN/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log
Running: FMW pre-req check...

Configuring: Run file system....
LogFile located at /u01/VISN/fs1/inst/apps/VISN_appsmt/admin/log/clone/run/RCloneApplyAppstier_*******.log

adpreclone.pl error while running copyBinary.sh on a 12.2 Environment

In a 12.2.5 environment adpreclone.pl errors out while running copyBinary.sh adpreclone.pl on appsTier



$perl adpreclone.pl appsTier

                     Copyright (c) 2011, 2014 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adpreclone Version 120.31.12020000.20

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

 Checking the status of the Oracle WebLogic Administration Server....

 Running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:
perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress



 Setting the wls environment

Beginning application tier Stage - Thu Aug 18 02:43:26 2016

/test12/applmgr/fs1/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /test12/applmgr/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/test12/applmgr/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/test12/applmgr/fs1/EBSapps/comn/java/classes:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/share.jar:/test12/applmgr/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test12/applmgr/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/test12/applmgr/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test12/applmgr/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -stage /test12/applmgr/fs1/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg

Log file located at /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/StageAppsTier_08180243.log

  \     20% completed
ERROR while running Stage...
Thu Aug 18 02:50:43 2016

ERROR while running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress ...
Thu Aug 18 02:50:43 2016
[aptest12@vmohswrly128 <RUN> ]


adpreclone log files shows the below error message

Error Message:

START: Creating FMW archive.
Running /test12/applmgr/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /test12/applmgr/fs1/EBSapps/comn/clone/FMW/t2pjdk -al /test12/applmgr/fs1/EBSapps/comn/clone/FMW/FMW_Home.jar -smw /test12/applmgr/fs1/FMW_Home -ldl /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
Script Executed in 298222 milliseconds, returning status 1
ERROR: Script failed, exit code 1

Solution:

1. set the environment variable T2P_JAVA_OPTIONS as given below

$export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/test12/applmgr/temp"

2. Make sure enough free space is available in the temp directory provided above

3.Rerun adpreclone.pl on appsTier

What is the difference between dbtechstack,dbtier and dbconfig modes in Cloning?

While Cloning Oracle Applications, adcgclone can be executed in 3 Modes on Database node.


1)perl adcfgclone.pl dbTier

 It configures the ORACLE_HOME on the target database node and  recreate the controlfiles.
 This is used in case of standby database/hot backups.   


2)perl adcfgclone.pl dbTechStack 

It configures the ORACLE_HOME on the target database tier node only. Relink the oracle home.

The below steps has to be performed manually
1. Create the Target Database control files.
2. Start the Target System Database in open mode
3. Run the library update script against the Database
cd $RDBMS_ORACLE_HOME/appsutil/install/[CONTEXT NAME]
sqlplus "/ as sysdba" @adupdlib.sql [libext]
 Where [libext] should be set to 'sl' for HP-UX, 'so' for any other UNIX platform, 
or 'dll' for Windows.

3)perl adcfgclone.pl dbconfig

It configures the database with context file. Database must be open while running adcfgclone in dbconfig mode.
e.g 
cd $RDBMS_ORACLE_HOME/appsutil/clone/bin
perl adcfgclone.pl dbconfig <Context_file>

Where Target Context File is:
$RDBMS_ORACLE_HOME/appsutil/<SID_HOSTNAME>.xml