HTTP 404 error after logging into Oracle EBS 12.2.6 url

Issue Description

We recently upgraded the Oracle EBS Version to 12.2.6 from 12.1.3. After Upgrade, one user reported he is getting erro HTTP 404 when he login to the application url.



Cause:

User has set his preferences to load a Custom Start Page after login. The custom html was not available after the upgrade.Hence he is getting error HTTP 404 as in screenshot. He is unable to reset the preferences as the page is not loading.

Solution:

In this case, only option is to modify the profile option 'Application Start Page' for the specific user  to a correct value.

Oracle EBS 12.2.x: ADOP error during Cutover phase

Background of the issue: I was applying hrblobal driver in a multi-node Production environment. Our Production application EBS has one Primary node and one DMZ Node.

After applying hrglobal driver using the online patching cycle, Cutover phase thrown below error message

==

Applying patch(es): <6390830> on node EBSDMZ failed.
If you choose to proceed with cutover, node EBSDMZ will be marked as abandoned.
Do you want adop to continue processing on completed nodes [y/n]? n
    As per user choice, adop will not proceed further with processing.

==

If we Provide input as 'Y' above, EBSDMZ node will be abandoned and we need to register the DMZ node again.


Solution:

1.Apply Patch on DMZ node using options=nodb and restart the cutover phase

On DMZ Node
=====
Source the patch filesystem
adop phase=apply patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv allnodes=NO action=nodb options=forceapply

2. Restart Cutover phase on Master node

ORA-32771: Cannot Add File To Bigfile Tablespace

When trying to add a datafile to a tablespace , got the error – ORA-32771: cannot add file to bigfile tablespace.

SQL> ALTER TABLESPACE BIG_TBSP1 add datafile ‘+DATA/df01.dbf’ size 130G;
ALTER TABLESPACE BIG_TBSP1 add datafile ‘+DATA/df01.dbf’ size 130G;
*
ERROR at line 1:
ORA-32771: cannot add file to bigfile tablespace

SOLUTION:

A bigfile tablespace can contain only one datafile and no other datafile can be added to that.

SQL> select BIGFILE from dba_tablespaces where tablespace_name=’BIG_TBSP1’;

BIGFILE
———————– —
YES

How to increase space in Bigfile Tablespace?


BIGFILE tablespace stores its data in a single datafile with a much larger capacity.

We can resize the size of the datafile in BIGFILE tablespace using ALTER DATABASE Command


ALTER DATABASE DATAFILE '/+DATA/df0101.dbf' RESIZE 180G;

Since BIGFILE Tablespace has only one datafile, there is no need to identify the datafile and increase its size.
We can use ALTER TABLESPACE command to resize at the tablespace level.

ALTER TABLESPACE BIG_TBSP1 RESIZE 180G;

Enable ArchiveLog Mode on Oracle RAC database fails with ORA-00265

On a test RAC environment, while archiving is being enabled, "ALTER DATABASE ARCHIVELOG" command errored with ORA-00265: instance recovery required, cannot set ARCHIVELOG mode

Below are the steps performed to enable Archivelog mode on a 3-node RAC database environment.

test0115:TEST1011 $ srvctl status database -d TEST101
Instance TEST1011 is running on node test0115
Instance TEST1012 is running on node test0116
Instance TEST1013 is running on node test0117
test0115:TEST1011 $
test0115:TEST1011 $ srvctl stop database -d TEST101


test0115:TEST1011 $ srvctl status database -d TEST101
Instance TEST1011 is not running on node test0115
Instance TEST1012 is not running on node test0116
Instance TEST1013 is not running on node test0117
test0115:TEST1011 $


test0115:TEST1011 $ srvctl start database -d TEST101 -o mount
test0115:TEST1011 $  srvctl status database -d TEST101
Instance TEST1011 is running on node test0115
Instance TEST1012 is running on node test0116
Instance TEST1013 is running on node test0117
test0115:TEST1011 $


test0115:TEST1011 $ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat May 9 14:13:54 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters and Automatic Storage Management options

SQL> select name,open_mode from gv$database;

NAME      OPEN_MODE
--------- --------------------
TEST101   MOUNTED
TEST101   MOUNTED
TEST101   MOUNTED

SQL>SQL> alter database archivelog;
alter database archivelog
*
ERROR at line 1:
ORA-00265: instance recovery required, cannot set ARCHIVELOG mode

Solution:
=========

1. Shutdown the database cleanly

SQL> shutdown immediate (on 3 RAC nodes)

2. Startup the database in mount stage

SQL> startup mount

3. Enable Archivelog mode

SQL> Alter database archivelog;

Database altered.

4. Open the database

SQL> Alter database open.

5. Verify archielog mode is enabled.

SQL> ARCHIVE LOG LIST;

Impdp fails with error ORA-31604

Issue:

When tried to Import a table from one Oracle database to another using IMPDP utility,encountered the error ORA-31604: invalid transform NAME parameter "MODIFY" for object type PROCACT_INSTANCE in function ADD_TRANSFORM

Impdp logfile has the below error:

Starting "<LOGIN_SCHEMA>"."SYS_IMPORT_FULL_10":  <LOGIN>/******** parfile=<PARFILE_NAME>.par logfile=<LOG_NAME>.log dumpfile=<DUMPFILE_NAME>.dmp parallel=1

Processing object type TABLE_EXPORT/TABLE/PROCACT_INSTANCE

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.LOAD_MD_TRANSFORMS []

ORA-31604: invalid transform NAME parameter "MODIFY" for object type PROCACT_INSTANCE in function ADD_TRANSFORM

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86

ORA-06512: at "SYS.KUPW$WORKER", line 8996

Cause: This is a bug in Oracle Database versio 11g, fixed in Oracle Database version 12c

Solution:

As a workaround, implement any of the below solutions:

1. Use an additional parameter that is exclude=PROCACT_INSTANCE during impdp

2. Redo the export with exclude=PROCACT_INSTANCE and perform import using new dumpfiles.

The use of the exclude=PROCACT_SYSTEM will exclude the resource manager objects such as resource plans and groups. We need to re-create resource plans and groups after impdp is successfully completed.

Oracle Recovery Manager Troubleshooting ::ORA-27211: Failed to load Media Management Library

This is an error during Database Clone using RMAN.

I was trying to restore database using RMAN Backup

RMAN-00571: ===========================================================

RMAN-00569: ===============
ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command
RMAN-05501: aborting duplication of target database

RMAN-03009: failure of allocate command on ORA_AUX_SBT_TAPE_1 channel 
ORA-19554: error allocating device, device type: SBT_TAPE, device name:

ORA-27211: Failed to load Media Management Library

Additional information: 

Cause: Backup Configuration was modified.

Solution:
Use the below command to Resolve the error and continue with Database Restore

rman auxiliary / 
run
{
allocate auxiliary channel c1 type disk;
allocate auxiliary channel c2 type disk;
duplicate database to 'DEV' backup location '/Daily_Backup/Backup_16042018';
}
exit;
EOF

How to Resolve ORA-12514 error While accessing Oracle database link?

Our client had requirement to pull data from MSSQL server to Oracle Database. I installed Oracle Gateway Server for MSSQL and configured database link to access the MSSQL Database.
When trying to access a heterogenous database link between Oracle Database 12c and MSSQL database, some users were getting the below error
ORA-12154: TNS:could not resolve the connect identifier specified.

Analysis

Database link was created on the Oracle Database environment to access data from MSSQL Server.
Database link was created using below command

create public database link DBLINK_INTG connect to "MYSQL" identified by "Password" using 'dg4msql';

Error

SQL>select sysdate from dual@DBLINK_INTG;
ORA-12154: TNS:could not resolve the connect identifier specified
12154. 00000 -  "TNS:could not resolve the connect identifier specified"
*Cause:    A connection to a database or other service was requested using
           a connect identifier, and the connect identifier specified could not
           be resolved into a connect descriptor using one of the naming methods
           configured. For example, if the type of connect identifier used was a
           net service name then the net service name could not be found in a
           naming method repository, or the repository could not be
           located or reached.
*Action:   
           - If you are using local naming (TNSNAMES.ORA file):
           - Make sure that "TNSNAMES" is listed as one of the values of the
           NAMES.DIRECTORY_PATH parameter in the Oracle Net profile
           (SQLNET.ORA)
           - Verify that a TNSNAMES.ORA file exists and is in the proper
           directory and is accessible.
           - Check that the net service name used as the connect identifier
           exists in the TNSNAMES.ORA file.
           - Make sure there are no syntax errors anywhere in the TNSNAMES.ORA
           file.  Look for unmatched parentheses or stray characters. Errors
           in a TNSNAMES.ORA file may make it unusable.
           - If you are using directory naming:
           - Verify that "LDAP" is listed as one of the values of the
           NAMES.DIRETORY_PATH parameter in the Oracle Net profile
           (SQLNET.ORA).
           - Verify that the LDAP directory server is up and that it is
           accessible.
           - Verify that the net service name or database name used as the
           connect identifier is configured in the directory.
           - Verify that the default context being used is correct by
           specifying a fully qualified net service name or a full LDAP DN
           as the connect identifier
           - If you are using easy connect naming:
           - Verify that "EZCONNECT" is listed as one of the values of the
           NAMES.DIRETORY_PATH parameter in the Oracle Net profile
           (SQLNET.ORA).
          - Make sure the host, port and service name specified
           are correct.



Solution

Recreate Heterogenous Database link using below command


create public database link DBLINK_INTG connect to "MYSQL" identified by "Password" using '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=Test1.domain.com)(PORT=1524))(CONNECT_DATA=(SID=dg4msql))(HS=OK))'
  

Oracle Applications Patching : ADOP Prepare phase failed due to missing information in FND_NODES.

Applying Patches to Oracle EBS and Oracle Database is one of the main activities for an Oracle database administration.We Often get errors during Oracle patching, which could be due to environment issue or a configuration issue or an Oracle Bug.

Checking Patch logfiles and Oracle Patching worker logfiles is very important to identify and troubleshoot the patching error.

Recently, while i am applying patch on a Production environment, Prepare phase in Oracle EBS online patching failed due to below  error

Error Details:Information missing in FND_NODES table for one or more application tier nodes.


=================================================
ADOP (C.Delta.9)
Session ID: 19
Node: prodapp01
Phase: prepare
Log: /applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/adop.log
===========================================================================

Validating configuration on node: [prodapp01].
    Log: /applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/prepare/validate/rwerpprodapp01
[ERROR]: Information missing in FND_NODES table for one or more application tier nodes. For details, refer to log file on the relevant node.
[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
[WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.
[WARNING]: Found invalid cross references in FS config files.
    [UNEXPECTED]Error occurred running "perl /applmgr/Oracle/PRODAPP/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/applmgr/Oracle/PRODAPP/fs2/inst/apps/PRODAPP_prodapp01/appl/admin/PRODAPP_prodapp01.xml -patchctxfile=/applmgr/Oracle/PRODAPP/fs1/inst/apps/PRODAPP_prodapp01/appl/admin/PRODAPP_prodapp01.xml -phase=prepare -logloc=/applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/prepare/validate/prodapp01 -promptmsg=hide"
    [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on prodapp01


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 1 (Fail)
=================================================


Analysis
The environment on which issue is reported has two middle Tier nodes. Primary node is the Master node and DMZ tier is the slave node.
Due to autoconfig failure on Oracle EBS Slave node, Support_Web  column is not updated in FND_NODES after autoconfig execution.
SUPPORT_WEB column for DMZ node is null in FND_NODES


Solution

1. Bring down Middle Tier Services
2. Run autoconfig on all the Middle Tier nodes
3. Startup the Middle Tier Services
4. Verify that FND_NODES table has correct information
5. Restart adop Prepare Phase

How to Restore context file in oracle applications?

Scenario: Your Applications context file is either accidentally deleted or Corrupted. You do not have any latest backup of context file available.

Below are the steps to recreate context file 

The Applications context file can be retrieved by running the adclonectx.pl script.

perl /clone/bin/adclonectx.pl retrieve

Above script prompts for Database connection Info, apps username/password , Available Context files and the location to which context files can be restored.When prompted for the context file to be retrieved, select the option of retrieving the Applications tier context file that has been lost and retrieve it to the default location as in $CONTEXT_FILE parameter.

The above command can be used only when INST_TOP the is still intact. Incase if INST_TOP is also lost, the Applications tier context file may be retrieved as follows:

Execute the following command on the Database tier:

perl /appsutil/clone/bin/adclonectx.pl retrieve

When prompted for the context file to be retrieved, select the option of retrieving the Applications tier context file that has been lost.Choose the location to any writeable location and copy the restored context file to the value specified in $CONTEXT_FILE parameter on Application Tier.

To retrieve the Database tier context file,

Execute the following command on the Database tier:

perl /appsutil/clone/bin/adclonectx.pl retrieve

When prompted for the context file to be retrieved, select the Database tier context file and retrieve it to the default location ($CONTEXT_FILE on DB node).

Weblogic Managed Servers Crashed due to weblogic.socket.MaxMessageSizeExceededException

ISSUE
=====

Weblogic Managed Servers are crashing frequently .

Admin Server Logfile has the below error message.

 <IOException occurred on socket: Socket[addr=test1.localhost.com/180.81.41.219,port=1345,localport=47093]
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'.
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'
at weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMuxableSocket.java:230)
at weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:351)
at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:989)
at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:920)
at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:897)
at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:215)
at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:177)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) 

Due to above error, Admin Server is going into WARNING Status and Managed Servers are disconnected from ADMIN Server.

SOLUTION
========

1. Increase Maxmessage size from WLS console


Login to weblogic console url as weblogic admin user


WebLogic Console -> Servers ->Admin Server ->  Protocols -> General -> Max Message 
Size

Increase Max Message Size to  20000000 (Any value higher than Incoming message size from error message)

2.Restart MT Services

adpreclone.pl error while running copyBinary.sh on a 12.2 Environment

In a 12.2.5 environment adpreclone.pl errors out while running copyBinary.sh adpreclone.pl on appsTier



$perl adpreclone.pl appsTier

                     Copyright (c) 2011, 2014 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adpreclone Version 120.31.12020000.20

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

 Checking the status of the Oracle WebLogic Administration Server....

 Running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:
perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress



 Setting the wls environment

Beginning application tier Stage - Thu Aug 18 02:43:26 2016

/test12/applmgr/fs1/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /test12/applmgr/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/test12/applmgr/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/test12/applmgr/fs1/EBSapps/comn/java/classes:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/share.jar:/test12/applmgr/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test12/applmgr/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/test12/applmgr/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test12/applmgr/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -stage /test12/applmgr/fs1/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg

Log file located at /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/StageAppsTier_08180243.log

  \     20% completed
ERROR while running Stage...
Thu Aug 18 02:50:43 2016

ERROR while running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress ...
Thu Aug 18 02:50:43 2016
[aptest12@vmohswrly128 <RUN> ]


adpreclone log files shows the below error message

Error Message:

START: Creating FMW archive.
Running /test12/applmgr/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /test12/applmgr/fs1/EBSapps/comn/clone/FMW/t2pjdk -al /test12/applmgr/fs1/EBSapps/comn/clone/FMW/FMW_Home.jar -smw /test12/applmgr/fs1/FMW_Home -ldl /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
Script Executed in 298222 milliseconds, returning status 1
ERROR: Script failed, exit code 1

Solution:

1. set the environment variable T2P_JAVA_OPTIONS as given below

$export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/test12/applmgr/temp"

2. Make sure enough free space is available in the temp directory provided above

3.Rerun adpreclone.pl on appsTier