How to change weblogic password from Backend in EBS 12.2.x?

On EBS 12.2.X environments , if you are on R12.AD.C.Delta.7 and R12.TXK.C.Delta.7 or later, we can change Weblogic admin server password using the below script.

1. Startup admin server. Do not start any other services

$. ./EBSapps.env RUN
$ cd $ADMIN_SCRIPTS_HOME
$adadminsrvctl.sh start

2. Run the below script  

perl $FND_TOP/patch/115/bin/txkUpdateEBSDomain.pl -action=updateAdminPassword

[The script will prompt for Current and New Weblogic Password and restarts Weblogic admin server using the new password.]

3. Run fs_clone to propagate the changes to the alternate file system (Patch File system).

$adop phase=fs_clone

EBS 12.2.6 login Page is taking long time to load

EBS Login Performance Issues:

After providing username and password in the login page of EBS 12.2.6, loading home page is taking very long time to load.

weblogic console url shows that one of the oacore servers, oacore_server2 is in WARNING state due to Max number of Stuck Threads reached.

Error Message from oacore log
=============================

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "605" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 605929 ms
[
GET /OA_HTML/OA.jsp?page=/oracle/apps/per/perimage/webui/PerImgViewPG&transactionid=82152123&language_code=US&pCalledFrom=PERWSIMG&OAMC=N&oas=Wnrjc-sK-Ns6nz5o7H7HIg..&pPersonId=16520&payLegislationCode=AE&CallFromForm='Y' HTTP/1.1
Accept: */*
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
Cookie: oracle.uix=0^^GMT+4:00^p; treemenu1=0; JSESSIONID=tg0hURm_tL89fEex9hDrVrvC1rLBjpTn3fc2pZ89o7Hpb7jR7gVi!222603901; JsessionIDForms=Lh0hURqNRRpfmiTouQ_0kF2MiYYn5x8j8cW5AhjSBMEyOEfvp1oa!-851768; DPRPROD=l6hjdVQuX3p1xvWrRtPjEW8SAj
ECID-Context: 1.005NsDH9rp8FCC0_zxo2yW0002km000083;kXjE
Connection: Keep-Alive
X-Forwarded-For: 10.11.32.120
Proxy-Client-IP: 10.11.32.120
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 222603901

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
        java.net.SocketInputStream.socketRead0(Native Method)
        java.net.SocketInputStream.read(SocketInputStream.java:152)
        java.net.SocketInputStream.read(SocketInputStream.java:122)
        oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:730)



Solution1
=====

Temporary workaround is to Restart the oacore servers to release the stuck threads

Solution2
========

1. Increase the value of paramter "StuckThreadMaxTime" TO 3600 Seconds from the weblogic Console.

Log in to WebLogic console 

Click on the Admin server >> Click on the Configuration tab >> Tuning tab and set the below values 

Weblogic Managed Servers Crashed due to weblogic.socket.MaxMessageSizeExceededException

ISSUE
=====

Weblogic Managed Servers are crashing frequently .

Admin Server Logfile has the below error message.

 <IOException occurred on socket: Socket[addr=test1.localhost.com/180.81.41.219,port=1345,localport=47093]
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'.
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'
at weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMuxableSocket.java:230)
at weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:351)
at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:989)
at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:920)
at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:897)
at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:215)
at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:177)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) 

Due to above error, Admin Server is going into WARNING Status and Managed Servers are disconnected from ADMIN Server.

SOLUTION
========

1. Increase Maxmessage size from WLS console


Login to weblogic console url as weblogic admin user


WebLogic Console -> Servers ->Admin Server ->  Protocols -> General -> Max Message 
Size

Increase Max Message Size to  20000000 (Any value higher than Incoming message size from error message)

2.Restart MT Services

How to delete Managed Servers in R12.2

The procedure to increase/decrease the number of oacore/oafm servers in R12.2.X is different from R12.1.x.
Because in R12.2.x OPMN is replaced by Weblogic Server.


Steps to delete a managed Server in R12.2.x

1. Ensure there is no active adop cycle.

2. Shut down the managed server which needs to be deleted from the cluster/domain.

 $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop <MANAGED SERVER NAME>

Example:

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server5


3. Execute the script adProvisionEBS.pl  on the host where the managed server was created.

Source the RUN filesystem before running the command.

 perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-delete-managedserver \
-contextfile=<CONTEXT_FILE> -managedsrvname=<MANAGED_SERVER_NAME> \
-servicetype=<SERVICE_TYPE> -logfile=<LOGFILE>
Sample Output:

bash-4.1$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
> ebs-delete-managedserver \
> -contextfile=$CONTEXT_FILE -managedsrvname=oacore_server6 \
> -servicetype=oacore -logfile=$APPLRGF/TXK/delMS_oacoreserver6.log

Enter the APPS Schema password:
Enter the WebLogic AdminServer password:

ManagedServer oacore_server5 deleted successfully.

The above command deletes the managed server and also updates related parameters in context file


To Verify the entries are deleted in Context File,check the output of below command

grep -i oacore_server6 $CONTEXT_FILE


4.Execute the following command to delete details of the managed server from the OHS configuration files mod_wl_ohs.conf and apps.conf


$ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=removeMS \
-oacore=<host>.<domain>:<port> \
-oafm=<host>.<domain>:<port> \
-forms=<host>.<domain>:<port> \
-formsc4ws=<host>.<domain>:<port> \
-ekanban=<host>.<domain>:<port> \
-accessgate=<host>.<domain>:<port> \
-yms=<host>.<domain>:<port>



Sample Output:

bash-4.1$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
 -contextfile=$CONTEXT_FILE \
-configoption=removeMS -oacore=localhost.localdomain.com:7205


*** LOG FILE:
/test12/applmgr/fs1/inst/apps/test12_localhost/logs/appl/rgf/TXK/txkSetAppsConf_08009354.log

5. Restart HTTP Server

cd $ADMIN_SCRIPTS_HOME

adapcctl.sh stop

adapcctl.sh start

6. Repeat the above steps on PATCH filesystem.

Weblogic Admin Server Startup Fails with "weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid"

After Changing Weblogic Password in boot.properties, Admin Server Fails to Startup.
Below error is reported in Admin Server Logfile.

<May 10, 2017 10:10:18 AM CDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.> 


Solution
=======

1.Shutdown Admin Server and Nodemanager
2.Go to $DOMAIN_HOME/servers/Admin Server>/data
3.Delete the boot.properties
4.Delete the whole ldap folder
5.Start the node-manager
6.Start Admin Server

How to export List of members related to a OID Group into a LDIF

While adding new member to a OID group or making any changes to OID group, we need to backup the existing members details related to group.
The backup can be taken into a LDIF file using ODSM.

Steps to Take backup of OID group members.

1. Login to ODSM url of OID application.
2. Navigate to Data Browser Tab in ODSM Interface.
3. Expand the directories and select the group which needs to be modified
4. Right click on the selected group  click “Export LDIF” , option available at the top.
5. Click “OK” . After export is completed, It will generate a link to open the LDIF file.

Changing APPS Password in R12.2

APPS Password in R12.2 can be changed either using AFPASSWD or FNDCPASSS utility. AFPASSWD is an enhancement to FNDCPASS.
To change APPS password , follow the below steps

1. Shut down MT Services

2.Use the syntax below to change password.

$AFPASSWD -c apps -s APPLSYS

Sample Output: Enter the ORACLE password of Application Object Library ‘APPSUSER’:
Connected successfully to APPS.
Enter the password for your ‘SYSTEM’ ORACLE schema:
Connected successfully to SYSTEM.  Working…
Enter new password for user:
Verify new password for user:
Working…
AFPASSWD completed successfully.

Note that, to change APPS password we need to give APPLSYS username in AFPASSWD syntax.
APPLSYS and APPS share the same password. Changing APPLSYS password using AFPASSWD utility changes APPS Password also.

3. After the password is changed, start only adminserver on RUN filesystem  using the script adadminsrvctl.sh.

Update the “apps” password in WLS Datasource as follows:
Log in to WLS Administration Console.
Click Lock & Edit in Change Center.
In the Domain Structure tree, expand Services, then select Data Sources.
On the “Summary of JDBC Data Sources” page, select EBSDataSource.
On the “Settings for EBSDataSource” page, select the Connection Pool tab.
Enter the new password in the “Password” field.
Enter the new password in the “Confirm Password” field.
Click Save.
Click Activate Changes in Change Center.

4. Start all the application services using adstrtal.sh

Troubleshooting "Nodemanager Associated with the machine is not reachable"

While trying to start Managed servers from the Administration Console, you may receive the below error

For server soa_server1, the Node Manager associated with machine Machine1 is not reachable. 
All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.


To fix this issue.

1) Check is the nodemanager is running at OS level using command

$ps -ef|grep -i nodemanager 

2) Verify the nodemanager logfile has no errors

3)Check if the managed server is properly associated with the machine

4) Check the nodemanager configuration(credentials/ Hostname/Port number) details are correctly updated in Console

In the Change Center of the Administration Console, click Lock & Edit (see Use the Change Center).
In the left pane of the Console, expand Environment and select Machines.
Select the machine for which you want to configure Node Manager.
Select Configuration > Node Manager.
In the Type field, select the Node Manager type from the drop-down list.
For more information on Node Manager types, see Node Manager Administrator's Guide.
Note: The values you provide here must correspond to the values you used to configured the Node Manager instance.
In the Listen Address field, enter the DNS name or IP address upon which Node Manager listens.
If you identify the Listen Address by IP address, you must disable Host Name Verification on Administration Servers that will access Node Manager. For more information and instructions, see Using Host Name Verification in Managing WebLogic Security.
Enter a value in the Listen Port field. This is the port where Node Manager listens for incoming requests.
If you have set the Type field to SSH or RSH, you should specify values in the Node Manager Home and Shell Command fields.
For more information on configuring Node Manager using SSH or RSH, see Node Manager Administrator's Guide.
Click Save.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes. 
Not all changes take effect immediately—some require a restart (see Use the Change Center).

Fusion Middleware FAQ's

1.What is the use of nodemanager in Weblogic?

Nodemanager enables us to startup/shutdown/maintain Managed Servers from console.
It is a Java utility that runs as separate process from WebLogic Server.
It is recommended to enable nodemanager in Clustered domains & High Availability Environments.

• Monitors server availability and can restart failed servers
• Can be used to migrate servers on a failed machine to Can be used to migrate servers on a failed machine to another machine

2.How to Check if nodemanager is running?

$ps -ef|grep -i nodemanager


3.How to Start/Stop Nodemanager?

Use the script to startup Nodemanager located in $WL_HOME\server\bin

$./startNode Manager.sh

or

$nohup ./startNodeManager.sh


To stop nodemanager process, get the process id using the below command

$ps -ef|grep -i nodemanager

Kill the java process related to nodemanager at OS level

or Simple close the command shell in which nodemanager is started