adop phase=fs_clone fails during Validation of /etc/hosts

 Issue:  Running fs_clone using command adop phase=fs_clone fails during txkADOPValidations.pl script

Error Details:

Error message from ADOPValidations log file

[ERROR]: Required content is missing from /etc/hosts file.

Cause:  

There are duplicate entries for localhost in /etc/hosts file in one of the slave nodes.

Solution:

Correct the entries in /etc/hosts and re-run fs_clone

Follow steps mentioned in MOS document:

adop phase=fs_clone Ends with "[ERROR]: Required content is missing from /etc/hosts file" (Doc ID 2815387.1)

ADOP fs_clone failing with error : "Could not find patch context file from database"

 Issue:

FS_CLONE is failing with error :Could not find patch context file from database
Error details:
$adop phase=fs_clone
==============
Checking for existing adop sessions.
    No pending session exists.
    Starting new adop session.
===========================================================================
ADOP (C.Delta.12)
Session ID: 43
Node: test1.domain1.com
Phase: fs_clone
Log: /test/erp/apps/fs_ne/EBSapps/log/adop/32/20220121_145748/adop.log
===========================================================================
Verifying existence of context files in database.
    [UNEXPECTED] Could not find patch context file from database
    [UNEXPECTED]Patch edition context file not found in database for host test1.domain1.com

Summary report for current adop session:
    Node test1.domain1.com:
       - Fs_clone status:   Not Started
    For more details, run the command: adop -status -detail

[STATEMENT] Please run adopscanlog utility, using the command
"adopscanlog -latest=yes"
to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 2 (Fail)

=========================
Cause:
===
Patch context file not found in fnd_oam_context_files tables.

Solution:
=======
1.Source the patch file system and run autoconfig on patch filesystem to update context file in database.
or
2. Run CtxSynchronizer to upload patch context file to database.

$ $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=<patch context file name and location> logfile=/tmp/patchctxupload.log

Provide apps password when prompted.

EBSO 12.2.x : Query to check status of ADOP session

Check Status of adop session using below sql query:

set pagesize 200;
set linesize 160;
col PREPARE_STATUS format a15
col APPLY_STATUS format a15
col CUTOVER_STATUS format a15
col ABORT_STATUS format a15
col STATUS format a15
select NODE_NAME,ADOP_SESSION_ID, PREPARE_STATUS , APPLY_STATUS  ,CUTOVER_STATUS , CLEANUP_STATUS , ABORT_STATUS , STATUS
from AD_ADOP_SESSIONS
order by ADOP_SESSION_ID;

Decoding Cutover status:

cutover_status='Y' 'COMPLETED'
cutover_status not in ('N','Y','X') and status='F' 'FAILED'
cutover_status='0' 'CUTOVER STARTED'
cutover_status='1' 'SERVICES SHUTDOWN COMPLETED'
cutover_status='3' 'DB CUTOVER COMPLETED'
cutover_status='D' 'FLIP SNAPSHOTS COMPLETED'
cutover_status='4' 'FS CUTOVER COMPLETED'
cutover_status='5' 'ADMIN STARTUP COMPLETED'
cutover_status='6' 'SERVICES STARTUP COMPLETED'
cutover_status='N' 'NOT STARTED'
cutover_status='X' 'NOT APPLICABLE'

EBS 12.2.X Patching: ADOP Cutover failed while running txkADOPCutOverPhaseCtrlScript.pl

Issue:

 During an adop online patching cycle, cutover phase failed while running the script txkADOPCutOverPhaseCtrlScript.pl

  [PROCEDURE]  Starting Middle Tier Services
    [UNEXPECTED]Error occurred running "perl /test/app/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -contextfile=/test/app/fs1/inst/apps/EBSTEST_testebsmt111/appl/admin/EBSTEST_testebsmt111.xml -patchcontextfile=/test/app/fs1/2/inst/apps/EBSTEST_testebsmt111/appl/admin/EBSTEST_testebsmt111.xml -promptmsg=hide -console=off -sessionid=14 -timestamp=20201108_194155 -outdir=/test/app/fs_ne/EBSapps/log/adop/14/20201108_194155/cutover/testebsmt111 -action=forcestartup"
    [UNEXPECTED]Cutover phase has failed.

Cause: 

Review of all adop and cutover log files shows that Cutover phase failed after filesystem switch and ADMIN server startup,oacore services could not be started after cutover due to port conflict.

Resolution:

1. Check Status of adop session using below sql query:
set pagesize 200;
set linesize 160;
col PREPARE_STATUS format a15
col APPLY_STATUS format a15
col CUTOVER_STATUS format a15
col ABORT_STATUS format a15
col STATUS format a15
select NODE_NAME,ADOP_SESSION_ID, PREPARE_STATUS , APPLY_STATUS  ,CUTOVER_STATUS , CLEANUP_STATUS , ABORT_STATUS , STATUS
from AD_ADOP_SESSIONS
order by ADOP_SESSION_ID;
For my issue, cutover status was 5, which means ADMIN startup is completed.

2. Restart the cutover phase with mtrestart=no option

adop phase=cutover mtrestart=no

3. Fix the issues related to services startup on RUN filesystem (which is switched after cutover).

4. Startup the services

5. Perform cleanup phase and complete the adop cycle
adop phase=cleanup 

How to Run hrglobal driver in Oracle EBS 12.2.X Version?

Steps to apply hrglobal driver in Oracle EBS 12.2.x version


1. Start an Oracle online EBS Patching cycle.

adop phase=prepare

2.Source the patch filesystem environment file

ADOP utillity is intelligent enough to switch to Patch Filesystem depending on the ADOP phase being run.For any Other manual operations, we need to set the environment to point to Patch Filesystem explicitly.

. ./EBSapps.env PATCH

3. Run DataInstall

java oracle.apps.per.DataInstall apps appspassword thin test.domain.com:1521:DEV

Select the Required localisations and save the changes.

Example of Data Install Changes Summary is shown below.



          DataInstall - Actions confirmation

Do you really wish to exit and save your changes?

      [Y]      - Yes, save then exit
      [N]      - No, don't save but exit
      [Return] - To return to the DataInstall Main Menu

Enter your choice (for example Y) : Y


          DataInstall - Actions summary
          -----------------------------

The following actions will be performed:


Localisation         Product(s)               Leg. Data? Action
-------------------- ------------------------ ---------- -------------
Global               Human Resources          Installed  Install
United Arab Emirates Human Resources          Installed  Install
United Arab Emirates Payroll                  Installed  Install

Localisation   College Data? Action
-------------- ------------- -------------
United Kingdom
United States

Option                         Data?         Action
------------------------------ ------------- -------------
JIT/Geocode


Legislation                            Action
------------------------------         -------------
ALL  All Legislations


4. Apply hrglobal.drv using below syntax.

adop phase=apply patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv workers=8

Complete the ADOP patching cycle by running finalize,cutover and cleanup phases

5. adop phase=finalize

6. adop phase=cutover

7. adop phase=cleanup

8. Run fs_clone to synchronize run and patch filesystems

adop phase=fs_clone

Oracle EBS 12.2.x: ADOP error during Cutover phase

Background of the issue: I was applying hrblobal driver in a multi-node Production environment. Our Production application EBS has one Primary node and one DMZ Node.

After applying hrglobal driver using the online patching cycle, Cutover phase thrown below error message

==

Applying patch(es): <6390830> on node EBSDMZ failed.
If you choose to proceed with cutover, node EBSDMZ will be marked as abandoned.
Do you want adop to continue processing on completed nodes [y/n]? n
    As per user choice, adop will not proceed further with processing.

==

If we Provide input as 'Y' above, EBSDMZ node will be abandoned and we need to register the DMZ node again.


Solution:

1.Apply Patch on DMZ node using options=nodb and restart the cutover phase

On DMZ Node
=====
Source the patch filesystem
adop phase=apply patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv allnodes=NO action=nodb options=forceapply

2. Restart Cutover phase on Master node

Oracle Applications Patching :How to Run hrglobal driver in Oracle EBS 12.2.X Version.

Applying hrglobal driver is needed during an oracle applications upgrade or when the  payroll data is required to be analyzed as per latest code pack.

Below are the steps to apply hrglobal driver in a Oracle EBS environment (12.2.x)

Before applying hrglobal driver download the latest hrglobal patch from My Oracle Support.

Start an Oracle online EBS Patching cycle.

1.adop phase=prepare

2. Apply latest hrglobal patch

adop phase=apply patches=<patchnum>

3. Run Datainstall and hrglobal driver using below commands

ADOP utillity is intelligent enough to switch to Patch Filesystem depending on the ADOP phase being run.
For any Other manual operations, we need to set the environment to point to Patch Filesystem explicitly.

. ./EBSapps.env PATCH

+Run DataInstall

java oracle.apps.per.DataInstall apps appspassword thin test.domain.com:1521:DEV

Select the Required localisations and save the changes.

Example of Data Install Changes Summary is shown below.



          DataInstall - Actions confirmation

Do you really wish to exit and save your changes?

      [Y]      - Yes, save then exit
      [N]      - No, don't save but exit
      [Return] - To return to the DataInstall Main Menu

Enter your choice (for example Y) : Y


          DataInstall - Actions summary
          -----------------------------

The following actions will be performed:


Localisation         Product(s)               Leg. Data? Action
-------------------- ------------------------ ---------- -------------
Global               Human Resources          Installed  Install
United Arab Emirates Human Resources          Installed  Install
United Arab Emirates Payroll                  Installed  Install

Localisation   College Data? Action
-------------- ------------- -------------
United Kingdom
United States

Option                         Data?         Action
------------------------------ ------------- -------------
JIT/Geocode


Legislation                            Action
------------------------------         -------------
ALL  All Legislations


++Apply hrglobal.drv using below syntax.

adop phase=apply patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv workers=8

4. adop phase=finalize

5. adop phase=cutover

6. adop phase=cleanup

ADOP Options: cleanup_mode

ADOP cleanup_mode

Run with adop cleanup phase.

Provides control over the extent of cleanup operations.

If no value is specified, cleanup is performed in standard mode, which does the same as quick mode but also drops obsolete code objects.

Values: quick/full

Purpose:

cleanup_mode=quick performs the minimum necessary cleanup to complete the online patching cycle. Use this mode if you want to start a new online patching cycle as soon as possible.

cleanup_mode=full performs all possible cleanup tasks for maximum recovery of space from unused database editions, objects and columns. Full cleanup is required after aborting an online patching cycle.

Usage:

adop phase=cleanup cleanup_mode=full

adop phase=cleanup cleanup_mode=quick

ADOP Options: actualize_all

Whenever adop prepare phase is initiated, a new patch edition is created in the database.During Online patching (ADOP) : An additional column ZD_EDITION_NAME is populated in the seed tables.As we do more online patching cycles, the number of entries for database editions will increase. This affects system performance and also increases cleanup time.When the number of old database editions reaches 25 or more, you should consider dropping all old database editions by running the adop actualize_all phase and then performing a full cleanup.

To drop older database editions, follow the below steps:

$ adop phase=prepare
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cutover
$ adop phase=cleanup cleanup_mode=full

What are the differences between Oracle Applications R12.1.x and R12.2.x

From DBA perspective, below are the changes in R12.2.x version of Oracle Applications, compared to R12.1.3.

1. Dual FS: Oracle Application Release 12.2 has dual filesystems(RUN FS and PATCH FS) to facilitate Online Patching.This Requires double the mountpoint space at MiddleTier Level.

2. Weblogic Server: Oracle HTTP Server in R12.1 is replaced by Oracle Weblogic Server.

10.1.3 Home is replaced by FMW_HOME.Weblogic Server Manages all oacore, forms and oafm servers in R12.2.

3. Online Patching : Dual Filesystem and EBR feature in 11gR2 Database facilitate online patching. Online Patching minimizes the downtime required for patching.adpatch utility in R12.1.x is replaced by adop utility. adop cycle runs in 5 Phases : prepare,apply,finalize,cutover and cleanup. adop internally calls adpatch utility to run the driver files associated with adop phases.

4. APPS Password Change: After Changing Apps Password in R12.2, we need to update the new password in EBS Datasource used by Weblogic Domain.To know more about apps password change in Oracle applications R12.2, check the below link.

http://www.appsdbadiaries.com/2016/02/changing-apps-password-in-r122.html

5. Cloning : While Cloning of R12.2.x adcfgclone instance needs to be run twice,  on RUN fs and PATCH fs.From 12.2.5 Version (OR R12.AD-TXK Delta 7), we have the dualfs option which creates both run fs and patch fs during clone process.

ADOP Useful Options :skipsyncerror

There are Various options available with Oracle Applications Online Patching Utility (ADOP). Here , we will see the benefits of skipsyncerror option.

Scenario: 

Your Previous  adop session failed while applying patches and you did not find any solution to fix the errors, Oracle Support Provided a new patch to fix the issue. You will get errors while synchronization and need to ignore those errors.

Usage: adop phase=prepare skipsyncerror=yes


skipsyncerror Details

This option is used along with prepare phase.

Purpose:

This feature enables the user to specify that any synchronization errors in the prepare phase are expected to be fixed automatically in the synchronization that takes place with subsequent patches.

Values: yes/no

Default value is 'no'. Set the value to 'yes' in order to work around synchronization failures that may occur when patches that failed to apply correctly in a previous patching cycle are synchronized during the prepare phase.

How to delete Managed Servers in R12.2

The procedure to increase/decrease the number of oacore/oafm servers in R12.2.X is different from R12.1.x.
Because in R12.2.x OPMN is replaced by Weblogic Server.


Steps to delete a managed Server in R12.2.x

1. Ensure there is no active adop cycle.

2. Shut down the managed server which needs to be deleted from the cluster/domain.

 $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop <MANAGED SERVER NAME>

Example:

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server5


3. Execute the script adProvisionEBS.pl  on the host where the managed server was created.

Source the RUN filesystem before running the command.

 perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-delete-managedserver \
-contextfile=<CONTEXT_FILE> -managedsrvname=<MANAGED_SERVER_NAME> \
-servicetype=<SERVICE_TYPE> -logfile=<LOGFILE>
Sample Output:

bash-4.1$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
> ebs-delete-managedserver \
> -contextfile=$CONTEXT_FILE -managedsrvname=oacore_server6 \
> -servicetype=oacore -logfile=$APPLRGF/TXK/delMS_oacoreserver6.log

Enter the APPS Schema password:
Enter the WebLogic AdminServer password:

ManagedServer oacore_server5 deleted successfully.

The above command deletes the managed server and also updates related parameters in context file


To Verify the entries are deleted in Context File,check the output of below command

grep -i oacore_server6 $CONTEXT_FILE


4.Execute the following command to delete details of the managed server from the OHS configuration files mod_wl_ohs.conf and apps.conf


$ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=removeMS \
-oacore=<host>.<domain>:<port> \
-oafm=<host>.<domain>:<port> \
-forms=<host>.<domain>:<port> \
-formsc4ws=<host>.<domain>:<port> \
-ekanban=<host>.<domain>:<port> \
-accessgate=<host>.<domain>:<port> \
-yms=<host>.<domain>:<port>



Sample Output:

bash-4.1$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
 -contextfile=$CONTEXT_FILE \
-configoption=removeMS -oacore=localhost.localdomain.com:7205


*** LOG FILE:
/test12/applmgr/fs1/inst/apps/test12_localhost/logs/appl/rgf/TXK/txkSetAppsConf_08009354.log

5. Restart HTTP Server

cd $ADMIN_SCRIPTS_HOME

adapcctl.sh stop

adapcctl.sh start

6. Repeat the above steps on PATCH filesystem.

What Happens during adop cutover Phase?

Cutover Phase Requires a brief downtime.

Following activities occur during Cutover:

+Users are logged off the system
+Finalize is run as part of cutover -if not run earlier in adop cycle
+Cutover phase cancels ADZPPATCH Concurrent Program
+Stops all Services on RUN and PATCH editions , switches filesystem and starts services on new RUN FS
+Flipping snapshots in run and patch editions.
+Users are brought back online on the patched system 

Environment is changed after cutover. Need to Re-source the env, for performing any actions after cutover.

Syntax:

$ adop phase=cutover

Multi-node Considerations
========================
By default cutover is invoked automatically on remote nodes.
If you want cutover individually:      
1. First node the cutover should be:           
$ adop phase=cutover allnodes=no       
2. For all other nodes you should do:        
$ adop phase=cutover allnodes=no action=nodb           
 * nodb because db work is already done


Cutover Parameters:
===================

mtrestart :Specifies whether or not to start Mid-Tier services at end of cutover, it allows you to perform some additional maintenance operations in the system before bringing it up.        - Takes values "yes" or "no".        - Default: yes
allnodes : Specifies whether or not to run on all the nodes.        - Takes values "yes" or "no".        - Default: yes
action: Specifies whether or not to take database action        - Takes values "db" or "nodb"        - Default: db

Useful adop options and syntax

1.Using analytics parameter in adop apply phase


$ adop phase=apply analytics=yes

Specifying this option will cause adop to run the following scripts and generate the associated output files (reports):

ADZDCMPED.sql - This script is used to display the differences between the run and patch editions, including new and changed objects. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>/adzdcmped.out.

ADZDSHOWED.sql - This script is used to display the editions in the system. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowed.out.

ADZDSHOWOBJS.sql - This script is used to display the summary of editioned objects per edition. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowobjs.out

ADZDSHOWSM.sql - This script is used to display the status report for the seed data manager. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowsm.out

Note: The analytics parameter should only be used when required, because of the extra processing needed.



2. flags=autoskip 

 e.g adop phase=apply patches=12345678  flags=autoskip 

 This option is an alternative for  "Continue as if it were successful"?  in adpatch, very useful in cases where patch failed to compile forms/reports and exiting.
 We need not restart the adop session just to compile a single forms (.fmb,.fmx) file.
 Make sure to review the autoskip.log logfile and fix the issues in autoskip log whenever you use autoskip flag in adop cycle.

3.  skipsyncerror=(yes|no)  [default: no]

 Specifies whether to ignore errors that may occur during incremental file system synchronization.  This might happen if you applied
 a patch in the previous patching cycle that had errors but decided to continue with the cutover.  When the patch is synchronized on
 the next patching cycle, the apply errors may occur again, but can be ignored.

4. wait_on_failed_job=(yes|no)  [default: no]

 Controls whether adop apply command exits when all workers have failed.  Instead of exiting, you can force adop to wait, and use the "adctrl" to retry failed jobs.

e.g adop phase=apply patches=7777 wait_on_failed_job=yes


5. Abort

The Online Patching Cycle can be aborted at any time prior to Cutover

e.g adop phase=abort

This is needed  if unrecoverable error happened or the user decides that patch is not needed.
If adop phase=apply failed, user should try abandon=yes first.
The abort command drops the database patch edition and  returns the system to normal runtime state.  Immediately following abort, you must also run a full cleanup and
fs_clone operation to fully remove effects of the failed online patching cycle.


            

What Happens in ADOP apply Phase

Apply Phase is executed in PATCH Filesystem
During Apply phase of an ADOP cycle, adop execute patch drivers to update Patch Edition
We can have multiple apply phases in an adop cycle, Multiple patches including customizations can be installed in this phase.

The production application is online and accessible to users. RUN filesystem is not affected by the changes.
Patches are applied to the copy (Patch Edition)
 Changes are made in the isolation of an Edition 
The running application is unaffected by these changes


workers 
specifies number of parallel workers, automatically calculated if not specified  
abandon 
abandons a patching session (if set to "yes") and must have an opposite specification to that of parameter "restart".
restart
 restarts a patching session (if set to "yes") and must have an opposite specification to that of parameter "abandon".

Example: $ adop phase=apply workers=4 abandon=yes restart=no patches=1234567,8909228



For more adop options and their usage Click here

What Happens During ADOP Finalize Phase

The Finalize phase of adop cycle,takes care of below online activities

1.Perform the final operations that can be executed while the Application is online
2.Compile invalid objects
3.Generate derived objects
4.Pre-compute DDL to be run at Cutover

After Finalize phase, we can pause the adop online patching cycle until the appropriate downtime window is available.

Syntax for running finalize phase 

$ adop phase=finalize
$ adop phase=finalize finalize_mode=quick == This is Default Mode
$ adop phase=finalize finalize_mode=full    

Finalize can be run in 2 modes: “QUICK” or “FULL”
FULL gathers database dictionary statistics (not transaction tables statistics)
QUICK skips gathering database dictionary statistics.

Downtime Mode feature in ADOP Patching

Downtime mode is introduced in E-Business Suite Release 12.2, to minimize the duration of 12.2 upgrade process.This feature is available from AD.Delta.5 Release Update Pack.

Before applying patch in downtime mode, all Application services must be shutdown.Downtime mode does not use online patching cycle phases,hence it applied patches quickly compared to online patching. However, EBS application will be down for longer time in downtime mode.Make sure that no adop cycle is in progress while applying patch in downtime mode, Otherwise it will give error.

To apply patch in downtime mode. Follow the below steps

1. Shut down Application Services

2. Apply patch using the below command

 $ adop phase=apply patches=<patch_number> apply_mode=downtime

3. Startup Application Services


Following Scenarios require to apply patch in downtime mode in R12.2

1. All Patching that is part of 12.2 upgrade process. Once the upgrade is complete and users are online, all subsequent patching on a production system should use online mode, unless stated in patch ReadMe.

2. Single-node development or test environments, where production support and high availability are not required.

Limitations of downtime mode

1. We cannot test the patch application prior to cutover as in online patching cycle.
2. We cannot rollback patch applied using downtime mode.
3. Downtime mode is not applicable for Multi-node environments

What Happens during adop prepare Phase?

The online patching cycle in R12.2  which uses ADOP  utility applies patch in 5 Phases
The phases are

PREPARE
APPLY
FINALIZE
CUTOVER
CLEANUP

PREPARE PHASE:

1)Prepares the Filesystem

During Prepare phase,RUN and PATCH filesystems are synchronized.By Default, Synchronization is done in incremental mode.
Full Synchronization is also available, if needed.

Online Patching Steps in R12.2 (ADOP)

Basics of online Patching in R12.2


Online patching uses the latest feature of the Oracle database 11gR2 which is called “Edition Based
Redefinition” and also uses multiple file systems on the application side.
While online Patching with adop is in progress, users can use the application and database on RUN Filesystem.
Patch is applied on the alternate filesystem, which is an exact copy of RUN filesystem,called PATCH filesystem.
Users are switched to PATCH filesystem after the patching is complete.
Application is only offline during the Cutover phase.Downtime is now redefined as Cutover.


ADOP Patching Steps
1. Download the patch and unzip in PATCH_TOP directory.

2. Prepare the system for Patching

Source the RUN environment file
cd /test12/applmgr
. ./EBSapps.env RUN

adop phase=prepare

3. Source the PATCH Environment & apply patches:
cd /test12/applmgr
. ./EBSapps.env PATCH
==> Apply patch 19697098

Patch Location: /test12/applmgr/patches
adop phase=apply patchtop=/test12/applmgr/patches  patches=19697098

4..Run finalize.
adop phase=finalize

5. Cutover Phase

    $ adop phase=cutover
   
6. Cleanup old editions

    $ adop phase=cleanup
   
7. Synchronize RUN and PATCH filesystems.Start Fs_clone

adop phase=fs_clone