Troubleshooting "Nodemanager Associated with the machine is not reachable"

While trying to start Managed servers from the Administration Console, you may receive the below error

For server soa_server1, the Node Manager associated with machine Machine1 is not reachable. 
All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.


To fix this issue.

1) Check is the nodemanager is running at OS level using command

$ps -ef|grep -i nodemanager 

2) Verify the nodemanager logfile has no errors

3)Check if the managed server is properly associated with the machine

4) Check the nodemanager configuration(credentials/ Hostname/Port number) details are correctly updated in Console

In the Change Center of the Administration Console, click Lock & Edit (see Use the Change Center).
In the left pane of the Console, expand Environment and select Machines.
Select the machine for which you want to configure Node Manager.
Select Configuration > Node Manager.
In the Type field, select the Node Manager type from the drop-down list.
For more information on Node Manager types, see Node Manager Administrator's Guide.
Note: The values you provide here must correspond to the values you used to configured the Node Manager instance.
In the Listen Address field, enter the DNS name or IP address upon which Node Manager listens.
If you identify the Listen Address by IP address, you must disable Host Name Verification on Administration Servers that will access Node Manager. For more information and instructions, see Using Host Name Verification in Managing WebLogic Security.
Enter a value in the Listen Port field. This is the port where Node Manager listens for incoming requests.
If you have set the Type field to SSH or RSH, you should specify values in the Node Manager Home and Shell Command fields.
For more information on configuring Node Manager using SSH or RSH, see Node Manager Administrator's Guide.
Click Save.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes. 
Not all changes take effect immediately—some require a restart (see Use the Change Center).

Fusion Middleware FAQ's

1.What is the use of nodemanager in Weblogic?

Nodemanager enables us to startup/shutdown/maintain Managed Servers from console.
It is a Java utility that runs as separate process from WebLogic Server.
It is recommended to enable nodemanager in Clustered domains & High Availability Environments.

• Monitors server availability and can restart failed servers
• Can be used to migrate servers on a failed machine to Can be used to migrate servers on a failed machine to another machine

2.How to Check if nodemanager is running?

$ps -ef|grep -i nodemanager


3.How to Start/Stop Nodemanager?

Use the script to startup Nodemanager located in $WL_HOME\server\bin

$./startNode Manager.sh

or

$nohup ./startNodeManager.sh


To stop nodemanager process, get the process id using the below command

$ps -ef|grep -i nodemanager

Kill the java process related to nodemanager at OS level

or Simple close the command shell in which nodemanager is started

Online Patching Steps in R12.2 (ADOP)

Basics of online Patching in R12.2


Online patching uses the latest feature of the Oracle database 11gR2 which is called “Edition Based
Redefinition” and also uses multiple file systems on the application side.
While online Patching with adop is in progress, users can use the application and database on RUN Filesystem.
Patch is applied on the alternate filesystem, which is an exact copy of RUN filesystem,called PATCH filesystem.
Users are switched to PATCH filesystem after the patching is complete.
Application is only offline during the Cutover phase.Downtime is now redefined as Cutover.


ADOP Patching Steps
1. Download the patch and unzip in PATCH_TOP directory.

2. Prepare the system for Patching

Source the RUN environment file
cd /test12/applmgr
. ./EBSapps.env RUN

adop phase=prepare

3. Source the PATCH Environment & apply patches:
cd /test12/applmgr
. ./EBSapps.env PATCH
==> Apply patch 19697098

Patch Location: /test12/applmgr/patches
adop phase=apply patchtop=/test12/applmgr/patches  patches=19697098

4..Run finalize.
adop phase=finalize

5. Cutover Phase

    $ adop phase=cutover
   
6. Cleanup old editions

    $ adop phase=cleanup
   
7. Synchronize RUN and PATCH filesystems.Start Fs_clone

adop phase=fs_clone

How to Change Oracle Database User Password?

Scenarios which require password Change:

1. Its recommended to change passwords periodically for security concerns.

2. User has forgotten/lost the password and unable to login to the system.

3. User wants to change password for any operational Purpose


Oracle database allows user to change database user's password using the below queries

+Logging in as the user

SQL> ALTER USER ME IDENTIFIED BY <NEW_PASSWORD>;

+Logging in as SYS/SYSTEM User

SQL> ALTER USER <USERNAME> IDENTIFIED BY <PASSWORD>;


There can be some cases when the DBA need to change password and revert back to older value after the dba action is completed.
Below test case helps in understanding how to change db user password and restore to older password value in Oracle 10g.

1. Change oracle user password using alter command

 SQL> conn system/manager
 Connected. 

SQL>  alter user TESTUSER identified by password1 
 User altered. 

2. Test the password is working

SQL> conn TESTUSER/password1;
 Connected.

3.Retreive the encrypted password for user

SQL> conn system/manager; 
Connected.

SQL> alter user TESTUSER identified by values '6057000499B128C3'; 
User altered. 

select username, password from dba_users where username = 'TESTUSER'; 
  
USERNAME                       PASSWORD 
------------------------------ ------------------------------ 
TESTUSER                            DB78866145D4E1C3 


4. Change the value of TESTUSER password to a new value

SQL> conn system/manager
Connected. 
SQL>  
SQL>  alter user TESTUSER identified by password2; 

User altered. 

 5. Verify the new password is working

SQL> conn TESTUSER/password2; 
Connected. 

6. Restore the older password using the encrypted password which we retreived from dba_users;
SQL> conn system/manager@dev; 
Connected. 
SQL> alter user TESTUSER identified by values 'DB78866145D4E1C3'; 

User altered. 

7. Verify that you are able to connect using old password.

SQL> conn TESTUSER/password2@dev;

ORA-1017 invalid username/password ;logon denied

SQL> conn TESTUSER/password1@dev;
Connected. 

How to Purge Workflow Notifications from Mailer Queue, so that Email is not sent to Users?

What to do when Workflow Notification  Mailer was not running properly for a certain period of time and accumulated all the emails in Workflow Queue.
We can Purge the workflow Mails which are not required by updating mail_status='SENT' for unwanted notifications in wf_notifications table.

Here are a couple of reasons to Purge Workflow Notifications

a)The end user may not want to receive outdated email.
b)The Workflow Queue has accumulated a lot of emails during the issue period, and processing has become very slow.

Below are the steps to Purge Workflow Notifications which are not required

1.First Take a Backup of wf_notifications table  and stop the Workflow Notification Mailer

2. Verify notifications from WF_NOTIFICATIONS table that has potential for being sent

SQL> select notification_id, recipient_role, message_type, message_name, status, mail_status
     from wf_notifications
     where status in ('OPEN', 'CANCELED')
     And Mail_Status In ('MAIL', 'INVALID')
     order by notification_id;
   
3. Narrow down the emails which are not to be sent by the Mailer from a specific date range.

SQL> select notification_id, begin_date, recipient_role, message_type, message_name, status, mail_status
     from wf_notifications
     Where Status In ('OPEN', 'CANCELED')
     And Mail_Status In ('MAIL', 'INVALID')
     and begin_date < sysdate-30              -- List only emails older than 30 days ago
     order by notification_id;

4. Set Mail_status='SENT' in WF_NOTIFICATIONS

SQL>update WF_NOTIFICATIONS set mail_status = 'SENT'
where mail_status in ('MAIL','INVALID')
and Status In ('OPEN', 'CANCELED')
and begin_date < sysdate-30;      

SQL>Commit;


This will update  notifications ,which are older than 30 days and waiting to be sent by the mailer to SENT.
Therefore the notifications will not get emailed when the Mailer is restarted.

5.Run the script $FND_TOP/patch/115/sql/wfntfqup.sql to purge the WF_NOTIFICATION_OUT queue and rebuild it with data currently in the WF_NOTIFICATIONS table. This is what purges all notifications waiting in the queue to be sent.  It will then populate the queue with the current data in the wf_notifications table.
Since you have changed the mail_status = 'SENT" it will not enqueue these messages again.. Only the ones where mail_status = 'MAIL' and status = 'OPEN' will be placed in the WF_NOTIFICATION_OUT queue and sent by the mailer. (or CANCELED and INVALID if certain concurrent reports are run)

Example :
$ sqlplus usr/passwd@db @$FND_TOP/patch/115/sql/wfntfqup.sql APPSusr APPSpw FNDusr


Example Syntax:
$ sqlplus apps/***@db @$FND_TOP/patch/115/sql/wfntfqup.sql apps ***** applsys


6.Now start the Workflow Notification Mailer.

Resolving ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE

I was working on Database upgrade from 11.2.0.3 to 11.2.0.4 on a 2 node RAC instance.
After Installation of 11.2.0.4 software and connecting sqlplus in the new environment, Got the below error when trying to startup the database in upgrade mode


SQL> Startup upgrade
ORACLE instance started.

Total System Global Area 1.3262E+11 bytes
Fixed Size                  2304584 bytes
Variable Size            2.2481E+10 bytes
Database Buffers         1.1006E+11 bytes
Redo Buffers               74080256 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE
Process ID: 29206
Session ID: 2002 Serial number: 3


Issue Fix
=======

I observed from the initialization parameter file, the below values are set for cluster parameters

cluster_database         = TRUE
cluster_database_instances= 2



We need to set the value of cluster_database to FALSE using the below command


SQL> alter system set cluster_database=FALSE scope=spfile sid='*' ;


Reboot the Database for the changes to be effective.
Now you will see that value of cluster_database_instances parameter is automatically changed to 1,after setting cluster_database to FALSE.


Now, the database can be started in upgrade mode without any errors.
Revert back the parameter cluster_database to TRUE, after the upgrade is completed.
The following command can be used to set cluster_database to TRUE


SQL> alter system set cluster_database=TRUE scope=spfile sid='*' ;

Restart the database.

PROCESS and DISCARD Folders IMAP

While Troubleshooting workflow relates issues in EBS, we may need to clear PROCESS and DISCARD folders under workflow user.
The below commands are useful to move folders, connecting to imap


a)Shutdown Workflow Components
b) login to imap host(e.g server1.ebs.com) as applmgr user

  $ telnet server1.ebs.com 143
  
    You will get below messages:

    ----------------------------
Trying **.**.***.**

Connected to server1.ebs.com (**.**.***.**).
Escape character is '^]'.

* OK Dovecot ready.
    ----------------------------
  i login wfmgr *****
  

  i rename PROCESS process_<sys_date>
  
 i rename DISCARD discard_<sys_date>
  

 i create PROCESS
  
i create DISCARD

i subscribe PROCESS

 i subscribe DISCARD


 i unsubscribe process_<sys_date>

 i unsubscribe discard_<sys_date>
  

c) Startup Workflow Components