How to delete Managed Servers in R12.2

The procedure to increase/decrease the number of oacore/oafm servers in R12.2.X is different from R12.1.x.
Because in R12.2.x OPMN is replaced by Weblogic Server.


Steps to delete a managed Server in R12.2.x

1. Ensure there is no active adop cycle.

2. Shut down the managed server which needs to be deleted from the cluster/domain.

 $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop <MANAGED SERVER NAME>

Example:

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server5


3. Execute the script adProvisionEBS.pl  on the host where the managed server was created.

Source the RUN filesystem before running the command.

 perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-delete-managedserver \
-contextfile=<CONTEXT_FILE> -managedsrvname=<MANAGED_SERVER_NAME> \
-servicetype=<SERVICE_TYPE> -logfile=<LOGFILE>
Sample Output:

bash-4.1$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
> ebs-delete-managedserver \
> -contextfile=$CONTEXT_FILE -managedsrvname=oacore_server6 \
> -servicetype=oacore -logfile=$APPLRGF/TXK/delMS_oacoreserver6.log

Enter the APPS Schema password:
Enter the WebLogic AdminServer password:

ManagedServer oacore_server5 deleted successfully.

The above command deletes the managed server and also updates related parameters in context file


To Verify the entries are deleted in Context File,check the output of below command

grep -i oacore_server6 $CONTEXT_FILE


4.Execute the following command to delete details of the managed server from the OHS configuration files mod_wl_ohs.conf and apps.conf


$ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=removeMS \
-oacore=<host>.<domain>:<port> \
-oafm=<host>.<domain>:<port> \
-forms=<host>.<domain>:<port> \
-formsc4ws=<host>.<domain>:<port> \
-ekanban=<host>.<domain>:<port> \
-accessgate=<host>.<domain>:<port> \
-yms=<host>.<domain>:<port>



Sample Output:

bash-4.1$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
 -contextfile=$CONTEXT_FILE \
-configoption=removeMS -oacore=localhost.localdomain.com:7205


*** LOG FILE:
/test12/applmgr/fs1/inst/apps/test12_localhost/logs/appl/rgf/TXK/txkSetAppsConf_08009354.log

5. Restart HTTP Server

cd $ADMIN_SCRIPTS_HOME

adapcctl.sh stop

adapcctl.sh start

6. Repeat the above steps on PATCH filesystem.

adpreclone.pl error while running copyBinary.sh on a 12.2 Environment

In a 12.2.5 environment adpreclone.pl errors out while running copyBinary.sh adpreclone.pl on appsTier



$perl adpreclone.pl appsTier

                     Copyright (c) 2011, 2014 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adpreclone Version 120.31.12020000.20

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

 Checking the status of the Oracle WebLogic Administration Server....

 Running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:
perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress



 Setting the wls environment

Beginning application tier Stage - Thu Aug 18 02:43:26 2016

/test12/applmgr/fs1/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /test12/applmgr/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/test12/applmgr/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/test12/applmgr/fs1/EBSapps/comn/java/classes:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/share.jar:/test12/applmgr/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test12/applmgr/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/test12/applmgr/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test12/applmgr/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -stage /test12/applmgr/fs1/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg

Log file located at /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/StageAppsTier_08180243.log

  \     20% completed
ERROR while running Stage...
Thu Aug 18 02:50:43 2016

ERROR while running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress ...
Thu Aug 18 02:50:43 2016
[aptest12@vmohswrly128 <RUN> ]


adpreclone log files shows the below error message

Error Message:

START: Creating FMW archive.
Running /test12/applmgr/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /test12/applmgr/fs1/EBSapps/comn/clone/FMW/t2pjdk -al /test12/applmgr/fs1/EBSapps/comn/clone/FMW/FMW_Home.jar -smw /test12/applmgr/fs1/FMW_Home -ldl /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
Script Executed in 298222 milliseconds, returning status 1
ERROR: Script failed, exit code 1

Solution:

1. set the environment variable T2P_JAVA_OPTIONS as given below

$export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/test12/applmgr/temp"

2. Make sure enough free space is available in the temp directory provided above

3.Rerun adpreclone.pl on appsTier

ADOP Prepare Phase Fails with ORA-31011: XML parsing failed

While applying Patch on a test instance (R12.2.5), Prepare phase failed with error ORA-31011: XML parsing failed

Complete error message from adop logfile
==================================

[aptest12@localhost <RUN> ]$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Initializing.
    Run Edition context  : /test12/applmgr/fs2/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch edition context: /test12/applmgr/fs1/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch file system free space: 1130.34 GB

Validating system setup.
    [ERROR]     Failed to execute SQL statement:
select node_name from FND_OAM_CONTEXT_FILES
where NAME not in ('TEMPLATE','METADATA','config.txt') and
CTX_TYPE='A' and (status is null or upper(status) in ('S','F'))
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'run'
and EXTRACTVALUE(XMLType(TEXT),'//oa_service_group_status[@oa_var=''s_web_admin_status'']')='enabled'
and EXTRACTVALUE(XMLType(TEXT),'//oa_service_list/oa_service[@type=''admin_server'']/oa_service_status')='enabled'

    [ERROR]     Error Message:
    [ERROR]     ORA-31011: XML parsing failed
    [ERROR]     ORA-19202: Error occurred in XML processing
    [ERROR]     LPX-00229: input source is empty
    [ERROR]     ORA-06512: at "SYS.XMLTYPE", line 272
    [ERROR]     ORA-06512: at line 1 (DBD ERROR: error possibly near <*> indicator at char 208 in '    select node_name from FND_OAM_CONTEXT_FILES
    [ERROR]         where NAME not in ('TEMPLATE','METADATA','config.txt') and
    [ERROR]             CTX_TYPE='A' and (status is null or upper(status) in ('S','F'))
    [ERROR]             and EXTRACTVALUE(<*>XMLType(TEXT),'//file_edition_type') = 'run'
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),'//oa_service_group_status[@oa_var=''s_web_admin_status'']')='enabled'
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),'//oa_service_list/oa_service[@type=''admin_server'']/oa_service_status')='enabled'
    [ERROR]     ')
    [UNEXPECTED]Error occurred identifying Admin server node


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


Cause:

fnd_oam_context_files table has incorrect information  or the table is corrupted


Solution:


1. Login to sqlplus as applsys and take backup of fnd_oam_context_files table.


SQL> conn applsys
Enter password:
Connected.
SQL> create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;

Table created.

2. Truncate Table fnd_oam_context_files

SQL>  truncate table fnd_oam_context_files;

Table truncated.


3. Bring down MT Services

4. Run Autoconfig to populate fnd_oam_context_files  with right information.

Adpatch Worker Fails with "ORA-01031: insufficient privileges"

While Applying Patch on Customer environment, adpatch failed with error

Worker log has the below error.
=====================
GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

AD Worker error:
The following ORACLE error:

ORA-01031: insufficient privileges


occurred while executing the SQL statement:

GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

Error occurred in file

/test12/applmgr/1200/ad/12.0.0/patch/115/sql/ademusr.sql

with arguments '&systempwd &un_fnd &pw_fnd &un_apps &pw_apps'.

==============


Solution
=========

1.Login to oracle database server as oracle user

2. Connect to sqlplus as sysdba and run the failed sql statement
sqlplus / as sysdba

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role
  2  ;

Grant succeeded.

SQL>


3. Use adctrl to skip the failed worker (Option 8) and continue with Patching

Weblogic Admin Server Startup Fails with "weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid"

After Changing Weblogic Password in boot.properties, Admin Server Fails to Startup.
Below error is reported in Admin Server Logfile.

<May 10, 2017 10:10:18 AM CDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.> 


Solution
=======

1.Shutdown Admin Server and Nodemanager
2.Go to $DOMAIN_HOME/servers/Admin Server>/data
3.Delete the boot.properties
4.Delete the whole ldap folder
5.Start the node-manager
6.Start Admin Server

"lsnrctl: symbol lookup error: lsnrctl: undefined symbol: nlgh" while starting Oracle Database listener

After Applying PSU, database listener failed to start with below error

 $ lsnrctl start testdb
lsnrctl: symbol lookup error: lsnrctl: undefined symbol: nlgh


Solution
==========
1.Relink network executables using below command

$  make -kf ins_net_server.mk install
/bin/chmod 755 /testdb/oracle/product/11204/bin/
 - Linking tnslsnr
rm -f tnslsnr
gcc -o tnslsnr -m64 -z noexecstack -L/testdb/oracle/product/11204/network/lib/ -L/testdb/oracle/product/11204/lib/ -L/testdb/oracle/product/11204/lib/stubs/  /testdb/oracle/product/11204/network/lib/s0nsgl.o /testdb/oracle/product/11204/network/lib/snsglp.o -lclntsh  `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lztkg11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11   `cat /testdb/oracle/product/11204/lib/sysliblist` -Wl,-rpath,/testdb/oracle/product/11204/lib -lm    `cat /testdb/oracle/product/11204/lib/sysliblist` -ldl -lm   -L/testdb/oracle/product/11204/lib -lons -lnl11  -ln11 -lnlsnr11 -lnlsnrc11 -lnsgr11 -lnms -lncm11 -lnmsp -lpeer -lnro11 -ln11   -lnl11 -lnlsnr11 -lnlsnrc11 -lnms -lncm11 -lnmsp -ln11     -lm
mv -f /testdb/oracle/product/11204/bin/tnslsnr /testdb/oracle/product/11204/bin/tnslsnr0
mv tnslsnr /testdb/oracle/product/11204/bin/tnslsnr
/bin/chmod 751 /testdb/oracle/product/11204/bin/tnslsnr
 - Linking lsnrctl
rm -f lsnrctl
gcc -o lsnrctl -m64 -z noexecstack -L/testdb/oracle/product/11204/network/lib/ -L/testdb/oracle/product/11204/lib/ -L/testdb/oracle/product/11204/lib/stubs/  /testdb/oracle/product/11204/network/lib/s0nsglc.o -lclntsh  `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lztkg11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /testdb/oracle/product/11204/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11   `cat /testdb/oracle/product/11204/lib/sysliblist` -Wl,-rpath,/testdb/oracle/product/11204/lib -lm    `cat /testdb/oracle/product/11204/lib/sysliblist` -ldl -lm   -L/testdb/oracle/product/11204/lib -lnl11  -ln11 -lnlsnr11 -lnlsnrc11 -lnsgr11 -lnms -lncm11 -lnmsp -lpeer -lnro11 -ln11   -lnl11 -lnlsnr11 -lnlsnrc11 -lnms -lncm11 -lnmsp -ln11     -lm
mv -f /testdb/oracle/product/11204/bin/lsnrctl /testdb/oracle/product/11204/bin/lsnrctl0
mv lsnrctl /testdb/oracle/product/11204/bin/lsnrctl
/bin/chmod 751 /testdb/oracle/product/11204/bin/lsnrctl


2. Restart Database Listener

$ lsnrctl start testdb

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 16-MAY-2016 02:00:20

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Starting /testdb/oracle/product/11204/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.4.0 - Production
System parameter file is /testdb/oracle/product/11204/network/admin/testdb/listener.ora
Log messages written to /testdb/log/network/testdb.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dblocalhost)(PORT=1523)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dblocalhost)(PORT=1523)))
STATUS of the LISTENER
------------------------
Alias                     testdb
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                16-MAY-2016 02:00:20
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /testdb/oracle/product/11204/network/admin/testdb/listener.ora
Listener Log File         /testdb/log/network/testdb.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dblocalhost)(PORT=1523)))
Services Summary...
Service "testdb" has 1 instance(s).
  Instance "testdb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

What Happens during adop cutover Phase?

Cutover Phase Requires a brief downtime.

Following activities occur during Cutover:

+Users are logged off the system
+Finalize is run as part of cutover -if not run earlier in adop cycle
+Cutover phase cancels ADZPPATCH Concurrent Program
+Stops all Services on RUN and PATCH editions , switches filesystem and starts services on new RUN FS
+Flipping snapshots in run and patch editions.
+Users are brought back online on the patched system 

Environment is changed after cutover. Need to Re-source the env, for performing any actions after cutover.

Syntax:

$ adop phase=cutover

Multi-node Considerations
========================
By default cutover is invoked automatically on remote nodes.
If you want cutover individually:      
1. First node the cutover should be:           
$ adop phase=cutover allnodes=no       
2. For all other nodes you should do:        
$ adop phase=cutover allnodes=no action=nodb           
 * nodb because db work is already done


Cutover Parameters:
===================

mtrestart :Specifies whether or not to start Mid-Tier services at end of cutover, it allows you to perform some additional maintenance operations in the system before bringing it up.        - Takes values "yes" or "no".        - Default: yes
allnodes : Specifies whether or not to run on all the nodes.        - Takes values "yes" or "no".        - Default: yes
action: Specifies whether or not to take database action        - Takes values "db" or "nodb"        - Default: db