Blog

While starting EBS services after “perl adcfgclone.pl appsTier dualfs” , received below errors

Invalid server name oacore_server1 

Invalid server name forms_server1 

Fix

1.  Removed that managed server oacore_server1 

[applmgr@AM501AMERAP0005 scripts]$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \ 

> ebs-delete-managedserver \ 

> -contextfile=$CONTEXT_FILE \ 

> -managedsrvname=oacore_server1 \ 

> -servicetype=oacore \ 

> -logfile=/home/applmgr/oacore_server2_remove.log 

Enter the APPS Schema password: 

Enter the WebLogic AdminServer password: 

ManagedServer oacore_server1 deleted 

2. Add the managed server oacore_server1 

[applmgr@AM501AMERAP0005 scripts]$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \ 

> ebs-create-managedserver -contextfile=$CONTEXT_FILE \ 

> -managedsrvname=oacore_server1 -servicetype=oacore \ 

> -managedsrvport=7201 -logfile=/home/applmgr/oacore_server1_add.log 

Enter the APPS Schema password: 

Enter the WebLogic AdminServer password: 

The logfile for this session is located at /home/applmgr/oacore_server1_add.log 

ManagedServer oacore_server1 created successfully. 

[applmgr@AM501AMERAP0005 scripts]$ 

3. Sync it with .conf file 

applmgr@AM501AMERAP0005 scripts]$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE \ 

> -configoption=addMS -oacore=oacore_server1:7201 

Enter apps schema password : 

4. Start oacore1 server from admanagedsrvctl.sh 

   

Repeat above steps 1 – 4 for forms_server1 (commands below)

perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \ 

ebs-delete-managedserver \          

-contextfile=$CONTEXT_FILE \ 

-managedsrvname=forms_server1 \ 

 -servicetype=forms \ 

-logfile=/home/applmgr/forms_server1_remove.log 

perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \ 

ebs-create-managedserver -contextfile=$CONTEXT_FILE \ 

-managedsrvname=forms_server1 -servicetype=forms \ 

-managedsrvport=7401 -logfile=/home/applmgr/forms_server1_add.log

 

perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE \ 

-configoption=addMS -oacore=oacore_server1:7401 

Start form_server1 server from admanagedsrvctl.sh 

xla_distribution_links_n1.xdf worker is running long without any error in the log file, it is trying to create index and taking long time.

Reading language information from applUS.txt …

  Temporarily resetting CLASSPATH to:

  “/u01/install/APPS/fs1/EBSapps/appl/ad/12.0.0/java/adjava.zip:/u01/install/APPS/fs1/EBSapps/comn/util/jdk32/lib/dt.jar:/u01/install/APPS/fs1/EBSapps/comn/util/jdk32/lib/tools.jar:/u01/install/APPS/fs1/EBSapps/comn/util/jdk32/jre/lib/rt.jar:/u01/install/APPS/fs1/EBSapps/comn/shared-libs/ebs-3rdparty/WEB-INF/lib/ebs3rdpartyManifest.jar:/u01/install/APPS/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-fmw/WEB-INF/lib/ebsFMWManifest.jar:/u01/install/APPS/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/install/APPS/fs1/FMW_Home/oracle_common/modules/oracle.uix_11.1.1/uix2.jar:/u01/install/APPS/fs1/EBSapps/comn/java/classes:/u01/install/APPS/fs1/EBSapps/10.1.2/forms/java:/u01/install/APPS/fs1/EBSapps/10.1.2/forms/java/frmall.jar:/u01/install/APPS/fs1/EBSapps/10.1.2/jlib/ewt3.jar:/u01/install/APPS/fs1/EBSapps/10.1.2/j2ee/OC4J_BI_Forms/applications/formsapp/formsweb/WEB-INF/lib/frmsrv.jar”

  Calling /u01/install/APPS/fs1/EBSapps/comn/util/jdk32/jre/bin/java …

Invoking Utility FndXdfCmp …

Class: oracle.apps.fnd.odf2.FndXdfCmp

Method: applyXDF

Arguments: &un_xla &pw_xla &un_apps &pw_apps &jdbc_protocol &jdbc_db_addr index &fullpath_xla_patch/115/xdf_xla_distribution_links_n1.xdf &fullpath_fnd_patch/115/xdf_xsl

**** Reseting package states in the current session ****

TimeStamp : [Thu Sep 01 22:04:31 PDT 2022]

Session details

=========

Its ACTIVE and creating INDEX.

=========================================================

Sid , Serial#     : 1617 , 35952

Server/Shadow     : 18437

Client/Foreground : 1234

Terminal / Machine: unknown / calraptst03.harborfreight.com

Username……..: APPS

Osuser……….: applmgr

Program………: JDBC Thin Client

Module……….: JDBC Thin Client

Status……….: ACTIVE

Action……….:

Wait_time…….: 0

State ……….: WAITING

Wait Event …..: db file scattered read

Seconds_in_wait.: 0

Pga_alloc….:    432.60 Mb

Pga_used…..:    427.39 Mb

Pga_free…..:      4.63 Mb

Pga_max……:    432.61 Mb

Lock / Latch.: None/ None

Blocking Sessi=on:

Blocking Session Status: NO HOLDER

Latch Spin…: None

Logon Time…: Tue 06-Sep-22 13:38:13

Last Call….: Tue 06-Sep-22 13:38:13  ->    277.2 Mins     4.6 Hours   .2 Days

Sql Address. : 00000000716A5A50

Sql Hash…. : 2915616157

Prev Sql Hash: 3569781485

Trans Status : ACTIVE

Trans Active : 00000001EE5EB2F0

Undo Generation: 1 Blocks

Changed Blocks : 433613 Blocks

…………    Current  Sql Statment   ……………..:

=========================================================

CREATE INDEX “XLA”.”XLA~DISTRIBUTION~LINKS~N1″ ON “XLA”.”XLA_DISTRIBUTION_LINKS”

(“SOURCE_DISTRIBUTION_TYPE”, “SOURCE_DISTRIBUTION_ID_NUM_1”, NVL(“SOURCE_DISTRI

BUTION_ID_NUM_2″,(-99)), “SOURCE_DISTRIBUTION_ID_NUM_2”)    PCTFREE 10 INITRANS

11 MAXTRANS 255    STORAGE(   BUFFER_POOL DEFAULT) LOCAL  ( PARTITION AP TABLESP

ACE APPS_TS_TX_IDX ,  PARTITION AR TABLESPACE APPS_TS_TX_IDX ,  PARTITION PA TAB

LESPACE APPS_TS_TX_IDX ,  PARTITION PO TABLESPACE APPS_TS_TX_IDX ,  PARTITION PN

TABLESPACE APPS_TS_TX_IDX ,  PARTITION OFA TABLESPACE APPS_TS_TX_IDX ,  PARTITI

ON OKL TABLESPACE APPS_TS_TX_IDX ,  PARTITION OZF TABLESPACE APPS_TS_TX_IDX ,  P

ARTITION CST TABLESPACE APPS_TS_TX_IDX ,  PARTITION PSB TABLESPACE APPS_TS_TX_ID

X ,  PARTITION CE TABLESPACE APPS_TS_TX_IDX ,  PARTITION FUN TABLESPACE APPS_TS_

TX_IDX ,  PARTITION FV TABLESPACE APPS_TS_TX_IDX ,  PARTITION GMF TABLESPACE APP

S_TS_TX_IDX ,  PARTITION LNS TABLESPACE APPS_TS_TX_IDX ,  PARTITION PAY TABLESPA

CE APPS_TS_TX_IDX ,  PARTITION IGI TABLE

=========================================================

…………….  Prev Sql Statment  ………………:

=========================================================

insert into sys.aud$( sessionid,entryid,statement,ntimestamp#, userid,userhost,t

erminal,action#,returncode, obj$creator,obj$name,auth$privileges,auth$grantee, n

ew$owner,new$name,ses$actions,ses$tid,logoff$pread, logoff$lwrite,logoff$dead,co

mment$text,spare1,spare2,  priv$used,clientid,sessioncpu,proxy$sid,user$guid, in

stance#,process#,xid,scn,auditid,  sqlbind,sqltext,obj$edition,dbid,rls$info,cur

rent_user)  values(:1,:2,:3,SYS_EXTRACT_UTC(SYSTIMESTAMP),     :4,:5,:6,:7,:8,

   :9,:10,:11,:12,     :13,:14,:15,:16,:17,     :18,:19,:20,:21,:22,     :23,:24

,:25,:26,:27,     :28,:29,:30,:31,:32,     :33,:34,:35,:36,:37,:38)

=========================================================

1 row selected.

SQL>

Cause

As per R12: AP: Application of Patch 29499116 is Hanging While Creating Index XLA~DISTRIBUTION~LINKS~N1. (Doc ID 2649876.1) , The problem index is NOT a standard one.  Tilde character (~) is not standard character for Oracle object names.

The index may have been created incorrectly via standard means, according to similar documents found, like Note 2513648.1 – Loader Worker Failing With ORA-00911 On MSC_SYSTEM_ITEMS

During on-line patching cycle, the indexes (new or modified) are getting changed as follows:
the “_” (underscores) are replaced by “~” (tilde) characters.

Bug 29376650 – MSCSLD FAILED WITH ORA-00911: INVALID CHARACTER ORA-06512, ERROR IN MAIN 103

Fix

1. Check the DB session details associated with running worker and capture SID, Serial#
2. Using adctrl make the concerned running worker failed
3. If the workers goes to “wait” status (meaning patch is not moving forward) after failing the worker in #2, check if the SID in#1 is still ACTIVE and kill from background
4. Ensure xla_distribution_links_n1.xdf gets picked up by another worker and gets completed(Validate from worker logs).

adop prepare phase was failing with below error

Validating configuration on node: [calrapuat03].
Log: /u01/install/APPS/fs_ne/EBSapps/log/adop/9/20221201_114840/prepare/validate/calrapuat03
[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
[ERROR]: Domain might be locked by some other WLS user process
[UNEXPECTED]Error occurred running “perl /u01/install/APPS/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl -contextfile=/u01/install/APPS/fs2/inst/apps/HFTUAT2_calrapuat03/appl/admin/HFTUAT2_calrapuat03.xml -patchctxfile=/u01/install/APPS/fs1/inst/apps/HFTUAT2_calrapuat03/appl/admin/HFTUAT2_calrapuat03.xml -phase=prepare -logloc=/u01/install/APPS/fs_ne/EBSapps/log/adop/9/20221201_114840/prepare/validate/calrapuat03 -promptmsg=hide”
[UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on calrapuat03

CAUSE

If the EBS Weblogic domain is in EDIT mode, it can cause adop to fail.

Someone with administrator privileges on WLS Admin Console may have locked the Weblogic domain for making changes, and has not released the checkout.

This leaves the domain in edit mode, resulting in the issue.

SOLUTION

 To resolve the issue test the following steps in a development instance and then migrate accordingly:

1. Start the admin server on the run file system (if not started already).

2. Login in to WLS Admin Console (http(s)://hostname.domain:<adminport>/console)

3. Click on the “Activate Changes” or the “Release Configuration” button (depending on the version) in the “Change Center” region on the top left corner.

4. Logout of the WLS console.

5. Rerun the adop phase=prepare command and confirm adop completes successfully.

adop prepare phase failed with below error

[ERROR] ETCC not run in the database node <node name>

Validating system setup.
Node registry is valid.
[WARNING] SYS.DUAL table has incorrect privileges
Shut down the application tier services and then run the following commands in the order shown
sqlplus / as sysdba @ $AD_TOP/patch/115/sql/ADFIXUSER.sql
sqlplus apps/ @ $AD_TOP/sql/adutlrcmp.sql
[ERROR] ETCC not run in the database node calrdbuat03
The EBS Technology Codelevel Checker needs to be run on the database node.
It is available as Patch 17537119.
[WARNING] MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh.

The node name in table APPS.TXK_TCC_RESULTS and /etc/hosts can be different, causing an adop failure to identify the database host name correctly.

Solution:

1. Validate /etc/hosts file has entries with a correct format. IP Address hostname.domain hostname.
2. Validate entries in APPS.TXK_TCC_RESULTS and make updates if needed
3. update APPLSYS.TXK_TCC_RESULTS set node_name=’uatfin’ where node_name =’uatfin.node.com’;
4. Validate the s_dbhost value in the database context file ${CONTEXT_FILE} and ensured it has only NODE_NAME and not the domain. Validate the same in applications context file ${CONTEXT_FILE} as well.
5. Downloaded the latest version of ETCC and ran on DB node.
6. Ran autoconfig on DB node.
7. Ran autoconfig on Apps node.
8. Retry prepare phase

 

 

(I had to do this as my servers were not connecting via putty and original ppk keys. Hence I wanted to check if it is a server issue or network issue apart from the fact tracert to the servers were timing out all throughout)

First create a OCI serial console connection

1. Open your OCI Dashboard Using your Web Browser.
2. Open the Cloud Menu (Hamburger Icon –  ) and select “Compute”.
3. On the “Instances” list Select the instance you would like to connect o using “Serial connection”.
4. Scroll down and Select the “Console Connections” option.
5. Press the “Create local connection” button.
6. On the Popup window, Click on “Generate a keypair for me” and save public and private keys on your local PC.
7. In few mins, you will see an active console connection.

Now connect using SSH

8. Copy the Console connection string (according to your desired method – SSH / VNC).

9. Open your Terminal / SSH Client and Paste the copied command. I used windows command prompt

Important

Don’t forget the -i switch. it is very important, especially if your SSH keys is stored on a different location or if it is a specific key you generated for this scenario.

The correct syntax will be to add the -i after the ssh word and add the path to your key file, afterwords the rest of the command will remain on it’s original form.

Note: When using single quotes with ProxyCommand=, I got Bad stdio forwarding specification ‘%h:%p’

C:\Users\heema>ssh -i ssh-key-2022-11-29.key -o ProxyCommand=’ssh -i ssh-key-2022-11-29.key -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.phx.anyhqljtxqk2jiackf2rbglisl2h7utrq6teoox57aylau5ptfs6dzdoc4iq@instance-console.us-phoenix-1.oci.oraclecloud.com” ocid1.instance.oc1.phx.anyhqljtxqk2jiacggemi6p7xgy4auhfgbmszhuq4li3ksi52yinh25edn5qC:\Users\heema>ssh -i ssh-key-2022-11-29.key -o ProxyCommand=”ssh -i ssh-key-2022-11-29.key -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.phx.anyhqljtxqk2jiackf2rbglisl2h7utrq6teoox57aylau5ptfs6dzdoc4iq@instance-console.us-phoenix-1.oci.oraclecloud.com’ ocid1.instance.oc1.phx.anyhqljtxqk2jiacggemi6p7xgy4auhfgbmszhuq4li3ksi52yinh25edn5q
Bad stdio forwarding specification ‘%h:%p’

C:\Users\heema>

Solution was to use double quotes as below

C:\Users\heema>ssh -i ssh-key-2022-11-29.key -o ProxyCommand=”ssh -i ssh-key-2022-11-29.key -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.phx.anyhqljtxqk2jiackf2rbglisl2h7utrq6teoox57aylau5ptfs6dzdoc4iq@instance-console.us-phoenix-1.oci.oraclecloud.com” ocid1.instance.oc1.phx.anyhqljtxqk2jiacggemi6p7xgy4auhfgbmszhuq4li3ksi52yinh25edn5q

Now we are connected to the server and it will ask for username/password. Since we dont have ‘opc’ password, we will follow below steps to create a new password

With your open console connection, you want to go back to the webpage where you built the console connection and reboot your instance, by clicking on the Reboot button.
Once you click Reboot go back to the terminal window and click on it to make it the active application. You’ll see the Linux stop and reboot routine scroll past the console eventually you’ll see the GRUB menu – From the GRUB menu take the following steps:

1. As soon as you see the GRUB boot menu, use the up/down arrows to stop the process, enabling you to use the boot menu.
2. In the boot menu, highlight the top item in the menu, and press e to edit the boot entry.
3. In edit mode, scroll down until you reach the line that starts with either linuxefi and add the following to the line: init=/bin/bash
   

4. You don’t need to save the file. Reboot the instance using keyboard shortcut CTRL+X

5. After the instance has rebooted, you’ll see the Bash prompt – Now you have cosmic powers to the instance, play safe and only do what you understand. Maybe even take a snapshot before if you care about the instance.

6. Run the following command to load the Linux policies to preserve the files you’re about to modify: /usr/sbin/load_policy -i  //NOTE// If you skip this step you will not able to login to your instance via SSH or console and you’ll be sad.

7. The following command will remount the root partition into read and write mode so you can modify the password files : /bin/mount -o remount, rw /

8. The following command will reset the password of the OPC default user account: sudo passwd opc

9. After the password has been reset you’re done. Reboot the machine with the following command : sudo reboot -f

Now you’ve done all of the needful and you should be to login to your Linux instance from the console after the reboot.