Blog

On a cloned environment, adop prepare phase is failing with [ERROR] ETCC not run in the database node

adop prepare phase failed with below error

[ERROR] ETCC not run in the database node <node name>

Validating system setup.
Node registry is valid.
[WARNING] SYS.DUAL table has incorrect privileges
Shut down the application tier services and then run the following commands in the order shown
sqlplus / as sysdba @ $AD_TOP/patch/115/sql/ADFIXUSER.sql
sqlplus apps/ @ $AD_TOP/sql/adutlrcmp.sql
[ERROR] ETCC not run in the database node calrdbuat03
The EBS Technology Codelevel Checker needs to be run on the database node.
It is available as Patch 17537119.
[WARNING] MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh.

The node name in table APPS.TXK_TCC_RESULTS and /etc/hosts can be different, causing an adop failure to identify the database host name correctly.

Solution:

  1. Validate /etc/hosts file has entries with a correct format. IP Address hostname.domain hostname.
  2. Validate entries in APPS.TXK_TCC_RESULTS and make updates if needed
  3. update APPLSYS.TXK_TCC_RESULTS set node_name=’uatfin’ where node_name =’uatfin.node.com’;
  4. Validate the s_dbhost value in the database context file ${CONTEXT_FILE} and ensured it has only NODE_NAME and not the domain. Validate the same in applications context file ${CONTEXT_FILE} as well.
  5. Downloaded the latest version of ETCC and ran on DB node.
  6. Ran autoconfig on DB node.
  7. Ran autoconfig on Apps node.
  8. Retry prepare phase

 

 

How to reset password from opc user from console connection

(I had to do this as my servers were not connecting via putty and original ppk keys. Hence I wanted to check if it is a server issue or network issue apart from the fact tracert to the servers were timing out all throughout)

First create a OCI serial console connection

  1. Open your OCI Dashboard Using your Web Browser.
  2. Open the Cloud Menu (Hamburger Icon –  ) and select “Compute”.
  3. On the “Instances” list Select the instance you would like to connect o using “Serial connection”.
  4. Scroll down and Select the “Console Connections” option.
  5. Press the “Create local connection” button.
  6. On the Popup window, Click on “Generate a keypair for me” and save public and private keys on your local PC.
  7. In few mins, you will see an active console connection.

Now connect using SSH

8. Copy the Console connection string (according to your desired method – SSH / VNC).

9. Open your Terminal / SSH Client and Paste the copied command. I used windows command prompt

Important

Don’t forget the -i switch. it is very important, especially if your SSH keys is stored on a different location or if it is a specific key you generated for this scenario.

The correct syntax will be to add the -i after the ssh word and add the path to your key file, afterwords the rest of the command will remain on it’s original form.

Note: When using single quotes with ProxyCommand=, I got Bad stdio forwarding specification ‘%h:%p’

C:\Users\heema>ssh -i ssh-key-2022-11-29.key -o ProxyCommand=’ssh -i ssh-key-2022-11-29.key -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.phx.anyhqljtxqk2jiackf2rbglisl2h7utrq6teoox57aylau5ptfs6dzdoc4iq@instance-console.us-phoenix-1.oci.oraclecloud.com” ocid1.instance.oc1.phx.anyhqljtxqk2jiacggemi6p7xgy4auhfgbmszhuq4li3ksi52yinh25edn5qC:\Users\heema>ssh -i ssh-key-2022-11-29.key -o ProxyCommand=”ssh -i ssh-key-2022-11-29.key -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.phx.anyhqljtxqk2jiackf2rbglisl2h7utrq6teoox57aylau5ptfs6dzdoc4iq@instance-console.us-phoenix-1.oci.oraclecloud.com’ ocid1.instance.oc1.phx.anyhqljtxqk2jiacggemi6p7xgy4auhfgbmszhuq4li3ksi52yinh25edn5q
Bad stdio forwarding specification ‘%h:%p’

C:\Users\heema>

Solution was to use double quotes as below

C:\Users\heema>ssh -i ssh-key-2022-11-29.key -o ProxyCommand=”ssh -i ssh-key-2022-11-29.key -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.phx.anyhqljtxqk2jiackf2rbglisl2h7utrq6teoox57aylau5ptfs6dzdoc4iq@instance-console.us-phoenix-1.oci.oraclecloud.com” ocid1.instance.oc1.phx.anyhqljtxqk2jiacggemi6p7xgy4auhfgbmszhuq4li3ksi52yinh25edn5q

Now we are connected to the server and it will ask for username/password. Since we dont have ‘opc’ password, we will follow below steps to create a new password

With your open console connection, you want to go back to the webpage where you built the console connection and reboot your instance, by clicking on the Reboot button.
Once you click Reboot go back to the terminal window and click on it to make it the active application. You’ll see the Linux stop and reboot routine scroll past the console eventually you’ll see the GRUB menu – From the GRUB menu take the following steps:

  1. As soon as you see the GRUB boot menu, use the up/down arrows to stop the process, enabling you to use the boot menu.
  2. In the boot menu, highlight the top item in the menu, and press e to edit the boot entry.
  3. In edit mode, scroll down until you reach the line that starts with either linuxefi and add the following to the line: init=/bin/bash

4. You don’t need to save the file. Reboot the instance using keyboard shortcut CTRL+X

5. After the instance has rebooted, you’ll see the Bash prompt – Now you have cosmic powers to the instance, play safe and only do what you understand. Maybe even take a snapshot before if you care about the instance.

6. Run the following command to load the Linux policies to preserve the files you’re about to modify: /usr/sbin/load_policy -i  //NOTE// If you skip this step you will not able to login to your instance via SSH or console and you’ll be sad.

7. The following command will remount the root partition into read and write mode so you can modify the password files : /bin/mount -o remount, rw /

8. The following command will reset the password of the OPC default user account: sudo passwd opc

9. After the password has been reset you’re done. Reboot the machine with the following command : sudo reboot -f

Now you’ve done all of the needful and you should be to login to your Linux instance from the console after the reboot.

After adcfgclone on Rel 12.2, managed servers were starting in “admin” mode

The Node Manager is already up.

ERROR: Unable to start up the managed server oacore_server1

ERROR: Unable to start up the managed server oacore_server3

Server specific logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server1/logs

11/02/22-09:07:49 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

Server specific logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server3/logs

11/02/22-09:07:49 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

ERROR: Unable to start up the managed server oacore_server2

Server specific logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server2/logs

11/02/22-09:07:50 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

ERROR: Unable to start up the managed server oacore_server4

Server specific logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server4/logs

11/02/22-09:07:52 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

After creating a new EBS instance using Rapidclone, managed servers such as oacore_server2/oacore_server3/oacore_server4 and forms_server1 fail to start and their status on WLS console are not “RUNNING” but “ADMIN” while oafm_server1 is “RUNNING”.

ORA-01017 is shown in the log files of oacore_server1 and forms_server1 as follows.

###<Nov 26, 2021 11:45:44 AM JST> <Error> <Deployer> <hostname.domain> <oacore_server1> <[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <> <1637894744672> <BEA-149205> <Failed to initialize the application ‘EBSDataSource’ due to error weblogic.application.ModuleException:
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:327)
at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)
at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:45)
at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:648)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:191)
at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:44)
at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:154)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:61)
at weblogic.deploy.internal.targetserver.SystemResourceDeployment.prepare(SystemResourceDeployment.java:55)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:39)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:191)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:180)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:96)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: weblogic.common.resourcepool.ResourceSystemException:
Could not connect to ‘oracle.jdbc.OracleDriver’.

The returned message is: ORA-01017: invalid username/password; logon denied

It is likely that the login or password is not valid.
It is also possible that something else is invalid in
the configuration or that the database is not available.
at weblogic.jdbc.common.internal.JDBCUtil.parseException(JDBCUtil.java:301)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:383)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:241)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1322)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1239)
at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:250)The


The customer confirmed followings.

  – The initial parameter SEC_CASE_SENSITIVE_LOGON is set to FALSE.
  – They can connect to Database using sqlplus by password both in upper and lower case.

Solution

Reset the password of EBSDataSource as follows.

(1) Login to WLS Console and proceed as follows.

      Domain Structure > Services > Data Sources > EBSDataSource > Configuration TAB > Connection Pool Sub-TAB

(2) Push “Lock & Edit” button in Change Center.
     Then set APPS schema’s password for “Password” & “Confirm Password” fields and push “Save” button.

(3) Push “Activate Chang” button in Change Center.

(4) Stop oacore_server1 and forms_server1 as follows.

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server2
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server3
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server4
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop forms_server1


(5) Start  oacore_server1 and forms_server1 as follows.

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server2
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server3
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server4
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start forms_server1


(6) Check if the status of “oacore_server1” and “forms_server1” has been changed to “RUNNING”.

Reference: Status of “oacore_server1” And “forms_sever1” Is “ADMIN” Not “RUNNING” on EBS Created by Rapidclone (Doc ID 2824612.1)

fs_clone issue on a cloned environment

Verifying existence of context files in database.

Checking if adop can continue with available nodes in the configuration.

Log: /u01/install/APPS/fs_ne/EBSapps/log/adop/7/20221108_074858/fs_clone/am501amerap0005

txkADOPEvalSrvStatus.pl returned SUCCESS

Validating configuration on admin node: [

![image][pasted-2022.11.08-15.57.05.png]

am501amerap0005].

Output: /u01/install/APPS/fs_ne/EBSapps/log/adop/7/20221108_074858/fs_clone/validate/remote_execution_result_level1.xml

[ERROR] txkADOPValidation failed on Node: “am501amerap0005”

[ERROR] txkADOPValidation failed or is incomplete on Admin node: am501amerap0005

[ERROR] Unable to continue processing on other available nodes: am501amerap0006

[UNEXPECTED]Error running “txkADOPValidation” on node(s): am501amerap0005.

Summary report for current adop session:

Node am501amerap0005:

– Fs_clone status: Failed

Node am501amerap0006:

– Fs_clone status: Not Started

For more details, run the command: adop -status -detail

[STATEMENT] Please run adopscanlog utility, using the command

“adopscanlog -latest=yes”

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 2 (Fail)

[applmgr@AM501AMERAP0005 ~]$

Cause

The global inventory on secondary node was corrupt or missing on secondary node(s) where the txkADOPValidation was failing. ADOPValidation Log files showed below error for secondary node

ERROR: Either /u01/app/oraInventory/ContentsXML/inventory.xml file doesn’t exist or doesn’t have READ permission.

Solution

Copy the contents of /etc/oraInst.loc from Primary to Secondary. Take a tar ball of global inventory from Primary to Secondary nodes. Ensure permissions are same as primary node.

Re-run the fs_clone

adop prepare phase failing with below error

[STATEMENT] ICM status cannot be determined.
[STATEMENT] [END 2022/11/01 10:48:57] Check if ICM is running
[UNEXPECTED]Error determining status of Internal Concurrent Manager
[UNEXPECTED]Error calling runConcurrentProgram subroutine.

I was running “adop phase=prepare” from a directory that contained a “*.dbc” file

Solution

Move the dbc file to a backup directory(if possible) or run adop prepare command from a different directory where “*.dbc” file is not presetn