After adcfgclone on Rel 12.2, managed servers were starting in “admin” mode

The Node Manager is already up.

ERROR: Unable to start up the managed server oacore_server1

ERROR: Unable to start up the managed server oacore_server3

Server specific logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server1/logs

11/02/22-09:07:49 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

Server specific logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server3/logs

11/02/22-09:07:49 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

ERROR: Unable to start up the managed server oacore_server2

Server specific logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server2/logs

11/02/22-09:07:50 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

ERROR: Unable to start up the managed server oacore_server4

Server specific logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server4/logs

11/02/22-09:07:52 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

After creating a new EBS instance using Rapidclone, managed servers such as oacore_server2/oacore_server3/oacore_server4 and forms_server1 fail to start and their status on WLS console are not “RUNNING” but “ADMIN” while oafm_server1 is “RUNNING”.

ORA-01017 is shown in the log files of oacore_server1 and forms_server1 as follows.

###<Nov 26, 2021 11:45:44 AM JST> <Error> <Deployer> <hostname.domain> <oacore_server1> <[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <> <1637894744672> <BEA-149205> <Failed to initialize the application ‘EBSDataSource’ due to error weblogic.application.ModuleException:
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:327)
at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)
at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:45)
at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:648)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:191)
at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:44)
at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:154)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:61)
at weblogic.deploy.internal.targetserver.SystemResourceDeployment.prepare(SystemResourceDeployment.java:55)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:39)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:191)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:180)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:96)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: weblogic.common.resourcepool.ResourceSystemException:
Could not connect to ‘oracle.jdbc.OracleDriver’.

The returned message is: ORA-01017: invalid username/password; logon denied

It is likely that the login or password is not valid.
It is also possible that something else is invalid in
the configuration or that the database is not available.
at weblogic.jdbc.common.internal.JDBCUtil.parseException(JDBCUtil.java:301)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:383)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:241)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1322)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1239)
at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:250)The


The customer confirmed followings.

  – The initial parameter SEC_CASE_SENSITIVE_LOGON is set to FALSE.
  – They can connect to Database using sqlplus by password both in upper and lower case.

Solution

Reset the password of EBSDataSource as follows.

(1) Login to WLS Console and proceed as follows.

      Domain Structure > Services > Data Sources > EBSDataSource > Configuration TAB > Connection Pool Sub-TAB

(2) Push “Lock & Edit” button in Change Center.
     Then set APPS schema’s password for “Password” & “Confirm Password” fields and push “Save” button.

(3) Push “Activate Chang” button in Change Center.

(4) Stop oacore_server1 and forms_server1 as follows.

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server2
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server3
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server4
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop forms_server1


(5) Start  oacore_server1 and forms_server1 as follows.

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server2
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server3
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server4
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start forms_server1


(6) Check if the status of “oacore_server1” and “forms_server1” has been changed to “RUNNING”.

Reference: Status of “oacore_server1” And “forms_sever1” Is “ADMIN” Not “RUNNING” on EBS Created by Rapidclone (Doc ID 2824612.1)

fs_clone issue on a cloned environment

Verifying existence of context files in database.

Checking if adop can continue with available nodes in the configuration.

Log: /u01/install/APPS/fs_ne/EBSapps/log/adop/7/20221108_074858/fs_clone/am501amerap0005

txkADOPEvalSrvStatus.pl returned SUCCESS

Validating configuration on admin node: [

![image][pasted-2022.11.08-15.57.05.png]

am501amerap0005].

Output: /u01/install/APPS/fs_ne/EBSapps/log/adop/7/20221108_074858/fs_clone/validate/remote_execution_result_level1.xml

[ERROR] txkADOPValidation failed on Node: “am501amerap0005”

[ERROR] txkADOPValidation failed or is incomplete on Admin node: am501amerap0005

[ERROR] Unable to continue processing on other available nodes: am501amerap0006

[UNEXPECTED]Error running “txkADOPValidation” on node(s): am501amerap0005.

Summary report for current adop session:

Node am501amerap0005:

– Fs_clone status: Failed

Node am501amerap0006:

– Fs_clone status: Not Started

For more details, run the command: adop -status -detail

[STATEMENT] Please run adopscanlog utility, using the command

“adopscanlog -latest=yes”

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 2 (Fail)

[applmgr@AM501AMERAP0005 ~]$

Cause

The global inventory on secondary node was corrupt or missing on secondary node(s) where the txkADOPValidation was failing. ADOPValidation Log files showed below error for secondary node

ERROR: Either /u01/app/oraInventory/ContentsXML/inventory.xml file doesn’t exist or doesn’t have READ permission.

Solution

Copy the contents of /etc/oraInst.loc from Primary to Secondary. Take a tar ball of global inventory from Primary to Secondary nodes. Ensure permissions are same as primary node.

Re-run the fs_clone

adop prepare phase failing with below error

[STATEMENT] ICM status cannot be determined.
[STATEMENT] [END 2022/11/01 10:48:57] Check if ICM is running
[UNEXPECTED]Error determining status of Internal Concurrent Manager
[UNEXPECTED]Error calling runConcurrentProgram subroutine.

I was running “adop phase=prepare” from a directory that contained a “*.dbc” file

Solution

Move the dbc file to a backup directory(if possible) or run adop prepare command from a different directory where “*.dbc” file is not presetn

Script to identify and clear print queues at OS level

Below script will identify all current print queues and clean up queues older than “1” day(you can change this number as per your retention policy)

[printadm@AM501AMERAP0001 ~]$ cat /tmp/Heema/printclear.sh

!/bin/bash

#

Author : Heema Satapathy

About : Clear Printer Queue

export A=$(date +%j)
mv /tmp/Heema/printqueue.txt /tmp/Heema/printqueue-date +%Y%m%d%H%M%S.txt
mv /tmp/Heema/printqueuename.txt /tmp/Heema/printqueuename-date +%Y%m%d%H%M%S.txt
find /tmp/Heema/ -name “printqueue.” -mtime +5 -exec rm {} \;
rm /tmp/Heema/printclear.out
lpstat -o | awk ‘{print $5 $6 $7}’ > /tmp/Heema/printqueue.txt
lpstat -o | awk ‘{print $1}’ > /tmp/Heema/printqueuename.txt
file1=/tmp/Heema/printqueue.txt
file2=/tmp/Heema/printqueuename.txt
export B=$(date +%Y)
Lines=$(cat $file1)
for Line in $Lines
do
echo “Queue in deletion process was from $Line”
export C=$(date +%j –date “$Line”)
export D=$(date +%Y –date “$Line”)
echo “This queue was on :$C day of $D year”
if [ “$B” = “$D” ]
then
diff=$(($A – $C))
echo “The difference is :$diff days”
echo “The current year is :$D”
if [ $diff -gt 1 ]
then
echo “The queue is older than 1 days during current year, clearing the queue….”
num=1
#E=$(sed -n ‘p’ $file2)
#echo “$E”
F=$(head -n $num $file2)
echo “$F”
cancel -a $F
num=num+1
else
echo “The queue is newer than 1 days during current year, NOT CLEARING THIS QUEUE”
fi
else
echo “There are queues from last year, PLEASE REVIEW AND DELETE THEM MANUALLY”| /usr/sbin/sendmail -t dbasupport@ctrworld.com
fi
echo “The day today is :$A th day out of $B year”
done

[printadm@AM501AMERAP0001 ~]$

FNDLOAD failes in Rel 12.2.10

$FND_TOP/bin/FNDLOAD apps/****** Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct O2C_EBTAXAZ.ldt CUSTOM_MODE=FORCE UPLOAD_MODE=REPLACE
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Attempt to connect to the database using dbcfile /u01/install/APPS/fs1/inst/apps/DEV_am501amerap0009/appl/fnd/12.0.0/secure/DEV.dbc and Applications User apps/MhmLoRt01Dmu# has failed. The dbcfile may be configured incorrectly or the Applications Username and Password may not match.

Cause

If the password is correct then another cause of this error is a missing ‘0’ ( zero) parameter . This is explained in the example below:

FNDLOAD apps/<appspwd> 0  Y UPLOAD $FND_TOP/patch/115/import/afsload.lct f1.ldt O2C_EBTAXAZ.ldt CUSTOM_MODE=FORCE UPLOAD_MODE=REPLACE =>  THIS IS CORRECT , THIS HAS A ZERO

FNDLOAD apps/<appspwd> Y UPLOAD $FND_TOP/patch/115/import/afsload.lct f1.ldt O2C_EBTAXAZ.ldt CUSTOM_MODE=FORCE UPLOAD_MODE=REPLACE     => THIS IS WRONG IT WILL FAIL , THIS IS MISSING A ZERO

Solution

I entered “0” and it worked

$FND_TOP/bin/FNDLOAD apps/******* 0 Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct O2C_EBTAXAZ.ldt CUSTOM_MODE=FORCE UPLOAD_MODE=REPLACE

Metalink Reference : FNDLOAD Errors With ‘Cannot complete applications logon. You may have entered an invalid applications password’ (Doc ID 1988658.1)