PRVF-4037 : CRS is not installed on any of the nodes

During the health check, we observed that has service status is showing as down in EM12C

Error as shown in the Incident Manager :

PRVF-4037 : CRS is not installed on any of the nodes,Verification cannot proceed,,,CRS integrity check failed,,Verification of CRS integrity was unsuccessful on all the specified nodes. ,NODE_STATUS::<Host>:EFAIL,OVERALL_STATUS::EFAIL

Logged in to the host as Grid owner :

[oracle@<host> ~]$  cluvfy comp ocr

Verifying OCR integrity

Checking OCR integrity...

ERROR:
PRVF-4037 : CRS is not installed on any of the nodes
Verification cannot proceed

OCR integrity check failed

Verification of OCR integrity was unsuccessful on all the specified nodes.
[oracle@<host> ~]$

Cause :

/etc/oraInst.loc was pointing to incorrect Inventory..

Solution :

Corrected the /etc/oraInst.loc and pointed it to expected ( Grid ) Inventory

[oracle@<host> ~]$  cluvfy comp ocr

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


ASM Running check passed. ASM is running on all specified nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Disk group for ocr location "+DA_XX" available on all the nodes


NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Verification of OCR integrity was successful.
[oracle@<host> ~]$

Create an Alias for [ sqlplus "/as sysdba" ]

We can use this OS utility to create an alias of the commands we use very frequently:

Here, for example :

-bash-3.00$ cat .bash_profile

alias sqllogin='sqlplus "/as sysdba"'

-bash-3.00$

-bash-3.00$ sqllogin

SQL*Plus: Release 11.2.0.1.0 Production on Fri Jul 5 07:24:46 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>

Hope this helps !!

Upgrading Application [ 11i CU1 to 11i CU2 ] and Database [ 9.2.0.6 to 11.2.0.1 ]

Hello All,


Recently I was involved in the upgrade [ application and database ]
Here is the walkthrough of the project.


::::::::::::::::::::: Overview :::::::::::::::::::::


: Existing System :
Application version : 11.5.10.1 [ 11i CU1 ]
Database version    : 9.2.0.6   [ 9iR2    ]




: Target System :
Application version : 11.5.10.2 [ 11i CU2 ]
Database version    : 11.2.0.1  [ 11gR2   ]




:Steps Involved :
1] Review of the System to be upgraded.
2] Clone for the iterations.
3] Phase - 1 : Upgrading Application from 11iCU1 to 11iCU2.
4] Phase - 2 : Upgrading Database from 9iR2 to 10gR2
5] Phase - 3 : Upgrading the required Family Packs 
6] Phase - 4 : Upgrading Database from 10gR2 to 11gR2.
7] Phase - 5 : Patches for the issues faced after upgrade.






Now , Lets get into the details of each step.


1] Review of the System to be upgraded >>
As we all aware , that 11i premium support ended in Nov 2010 , so we referred Metalink Note-ID 
"883202.1 for Minimum Baseline Patch Requirements for Extended Support"  to make our review easier.


To check the existing patchset level of the system :


Execute below sql :
select APP_SHORT_NAME, max(PATCH_LEVEL) from AD_PATCH_DRIVER_MINIPKS GROUP BY APP_SHORT_NAME ;


OR


Download the latest patchset.sh from the Metalink Note-ID :
Oracle Applications Current Patchset Comparison Utility - patchsets.sh [ID 139684.1]


This shell script will provide you the current patchset level and also the latest patchset available on the metalink.


This will ease our review at module level as well.


Very Important Note : Prepare a detailed sheet of the steps to be executed.




2] Clone for the iterations >>
For Cloning we used Rapid clone method .


Importantly , Always check UAT system configuration to avoid surprise performance issues after Production Upgrade.
Keep the configuration at optimum for benchmarking testing.




3] Phase - 1 : Upgrading Application from 11iCU1 to 11iCU2 >>


Phase - 1 is three segments step.


A - Applying the CU2 patch.
B - Applying the latest TXK AUTOCONFIG AND TEMPLATES ROLLUP PATCH.
C - Applying 11.5.10 INTEROP PATCH FOR 10GR2.


A] Applying the CU2 patch >>


The Read me suggests to apply latest Oracle Applications DBA Minipack 11i.AD.I :: i.e AD.I.7 plus few pre-requisites :


Next step would be to check whether our Application-Database Tier is ready to be upgraded. So, we can run TXKScript.pl on both the tiers.


The script TXKScript.pl is placed under FND_TOP on Application Tier and ORACLE_HOME on Database Tier.


Pre-requisites >>


# Run the admkappsutil.pl under AD_TOP to create the appsutil.zip file
# Copy the successfully generated zip file to DB Tier and unzip it [ unzip -o appsutil.zip ]




Next step would be to run the scripts on both the Tiers :


Sample Script :


>> Run the Technology Validation Utility using the below script on Application Tier:
$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl
        -script=$FND_TOP/patch/115/bin/txkVal11510MP_AP.pl
        -txktop=$APPLTMP
        -appspass=<apps_password>
        -outfile=$APPLTMP/txkVal11510MP.html




>>  Run the Technology Validation Utility using the below script on Database Tier:
$ADPERLPRG $ORACLE_HOME/appsutil/bin/TXKScript.pl
        -script=$ORACLE_HOME/appsutil/bin/txkVal11510MP.pl
        -txktop=$ORACLE_HOME/appsutil
        -appspass=<apps_password>
        -outfile=$ORACLE_HOME/appsutil/temp/txkVal11510MP_DB.html






IMPORTANT :: Make sure the output of the script is "ALL PASS " on both the Tiers.


Next step would be to apply the CU2 patch on the application :
# Bounce the database and listener to ensure that there are no locks.
# Use the option "nocompilejsp" to save the time of upgrade.


Note : Check for the successful completion.




B] Applying the latest TXK AUTOCONFIG AND TEMPLATES ROLLUP PATCH >>


Pre-Steps :
We have to check whether the application is ready to techstack upgrade :


Run the Technology Stack Validation script as follows:
           NOTE: Ensure that your current working directory is
           [patch unzipped location]/fnd/patch/115/bin


          ./txkprepatchcheck.pl -script=ValidateRollup -outfile=$APPLTMP/txkValidateRollup.html -appspass=<apps database password>




Once we pass the validation , we can apply the TXK patch on the application Tier.




C] Applying 11.5.10 INTEROP PATCH FOR 10GR2 >>
This patch will prepare the application tier for the database upgrade to 10gR2.


Once the Phase - 1 is completed , we are sure that our Application is upgraded from 11iCU1 to 11iCU2 and also is ready to database upgrade from 9iR2 to 10gR2.




4] Phase - 2 : Upgrading Database from 9iR2 to 10gR2 >>


A] Installation of Oracle 10gR2 Base Release [ 10.2.0.1 ]


# Create a directory on the server to install the Oracle Binaries onto the server.


# Download the stage from OTN and upload & unzip the database 10gR2 base release stage to the server.


# Install the Base Release to directory. [ with companion ]


# Now download the 10gR2 patchset 3 patch and apply it on 10g Home.


# Apply the pre-req optaches on the new home. [ Refer upgrade metalink note-id ]


# Check whether the Existing database is ready for the upgrade :


Goto 10.2.0/rdbms/admin
ls –ltr  utlu112i.sql
Sqlplus “/as sysdba”


SQL> spool <filename>.log
SQL>@utlu112i.sql


B] Upgrade the database from 9iR2 to 10gR2:


# Unset the existing 9i environment:


# Set new ORACLE_HOME and ORACLE_SID 


cd $ORACLE_HIME/bin


./dbua




Follow the screens and start the upgrade.


Make sure the upgrade finishes successfully.


Check the dba_registry for successful upgrade of the components.




Post Steps :


Re-create 9idata directory
Run changeperm.sh from $ORACLE_HOME/install
Fix Korean lexers
Run adgrants.sql
Grant create Procedure on CTXSYS
Implement Autoconfig
Gather Stats for SYS schema


Note : Set new environment and check the application.




5] Phase - 3 : Upgrading the required Family Packs >>


Review required family packs ( In our case )


A] 11i.ATG_PF.H.delta.7 (RUP 7)
B] 11I.HR_PF.K delta 5
C] 11i.FIN.PF.G
D] MINI-PACK 11I.AP.O
E] MINI-PACK 11I:PN.M


Post Steps :
Generate message files. 
Compile APPS schema(s). 
Compile flexfield data in AOL tables. 
Compile menu information. 




6] Phase - 4 : Upgrading Database from 10gR2 to 11gR2


Pre-Steps :


exec dbms_stats.gather_dictionary_stats;
exec dbms_stats.gather_fixed_objects_stats;
EXEC DBMS_STATS.GATHER_SCHEMA_STATS('SYS');
EXEC DBMS_STATS.GATHER_SCHEMA_STATS('SYSMAN'); [ if the user exist ]


A] Installation of Oracle 11gR2 Base Release [ 11.2.0.1 ]


# Create a directory on the server to install the Oracle Binaries onto the server.


# Download the stage from OTN and upload & unzip the database 11gR2 base release stage to the server.


# Install the Base Release to directory. [ with companion ]


# Apply the pre-req optaches on the new home. [ Refer upgrade metalink note-id ]


# Check whether the Existing database is ready for the upgrade :


Login as Database user 
Goto 11.2.0/rdbms/admin
ls –ltr  utlu112i.sql
Sqlplus “/as sysdba”


SQL> spool <filename>.log
SQL>@utlu112i.sql
....
SQL> spool off;


# Unset the existing 10g environment:


# Set new ORACLE_HOME and ORACLE_SID 


cd $ORACLE_HIME/bin


./dbua




Follow the screens and start the upgrade.


Make sure the upgrade finishes successfully.


Check the dba_registry for successful upgrade of the components.




Post Steps :


Re-create 9idata directory
Run adgrants.sql  
Grant create procedure privilege on CTXSYS  
Set CTXSYS parameter  
Implement Autoconfig
Gather statistics for SYS schema 


Note : Set 11g environment and check the application health.



7] Phase - 5 : Patches for the issues faced after upgrade >>


This phase is very important phase of the upgrade.


Once we release the instance to the users , they will report list of issues.


Some issues wil resolved using patches, so log a ticket and apply all the required patches.


Important Note : To avoid surprise issues after upgrade , we have to make sure that most of the issues are resolved in the iteration phase.


This will reduce the downtime of the instance after upgrade.


I have tried my best to summarize the upgrade ; I hope this will help !!




Best Of luck for the upgrade.

Metalink Note-IDs related to Concurrent Managers

1) Concurrent Processing Tables and Purge Concurrent Request and/or Manager Data Program (FNDCPPUR) [ID 104282.1]

2) Purge Concurrent Request FNDCPPUR Does Not Delete Files From File System or Slow performance [ID 822368.1]

3) Best Practices for Performance for Concurrent Managers in E-Business Suite [ID 1057802.1]

4) Using the Output Post Processor (OPP) in Oracle Applications Concurrent Processing [ID 291792.1]

5) Cannot View Concurrent Manager Log And Out Files in R12 (APP-FND-01634 ) [ID 555394.1]

6) Deadlock Error During Concurrent Request Termination [ID 1060736.1]

7) Concurrent Processing (CP) / APPS Reporting Scripts [ID 213021.1]

8) How to Activate Parallel Concurrent Processing - More Facts [ID 602899.1]

9) How to Configure Concurrent Managers Applications for Parallel Concurrent Processing [ID 157352.1]10) Performance: Purge Obsolete Workflow Runtime Data ( FNDWFPR ) Concurrent Request is Slow [ID 985332.1]

11) Concurrent Requests Errors Out With ORA-20100: Error: FND_FILE Failure [ID 460643.1]

12) In R12 Get APP-FND-01564 ORA-01116 ORA-01110 ORA-27041 Errors When Running Concurrent Requests [ID 577291.1]

13) Performance Degradation When Submitting a Concurrent Request [ID 563434.1]

14) Concurrent Mangagers Down Cause Of Output Post Process Failing To Start [ID
798362.1]

15) How To Create a Java Concurrent Program? [ID 827563.1]

16) How to Troubleshoot Concurrent Manager Start Up Issue Related to Service Manager [ID 735148.1]

17) How To Submit A Concurrent Request Set Using Fnd_Request.Submit_Request [ID 382791.1]

18) How to Solve Concurrent Request Database Performance Issues [ID 863780.1]

19) How To Get Level 12 Trace And FND Debug File For Concurrent Programs [ID 726039.1]

20) How To Print Concurrent Requests in PDF Format [ID 333504.1]

21) When Submitting Create Accounting Program It Completes With Warning "Setting the Profile Option Concurrent: OPP Response Timeout to a Higher Value May Be Necessary" [ID 553320.1]

22) Troubleshooting AutoInvoice: Concurrent Request Failures (Request Status = Error) [ID 1082238.1]

23) How To Create A Custom Concurrent Program With Host Method and Pass Parameters To The Shell Script [ID 266268.1]

24) How to Submit a Concurrent Request With CONCSUB Using a User Other Than APPS [ID 1050415.1]

25) How To Gather Statistics On Oracle Applications 11.5.10(and above) - Concurrent Process,Temp Tables, Manually [ID 419728.1]

26) Where Is 'Delete Data From Temporary Table' Concurrent Program - ICXDLTMP.SQL [ID 397118.1]

27) Error ORA-20100 .tmp Creation For FND_FILE Failed In FA Concurrent Requests [ID 392635.1]

28) Concurrent Requests Fail with FDPSTP failed due to ORA-20100: File o0002712.tmp creation for FND_FILE [ID 148262.1]

29) Concurrent Requests Complete with ORACLE Error 20100 in FDPSTP [ID 157756.1]

30) Ora-01555 Caused By SELECT Statements With Low Query Duration and no Concurrent DML [ID 361992.1]

31) ORA-20100 & ORA-6512 When Running a Concurrent Process [ID 123977.1]

32) How to Find Database Session & Process Associated with a Concurrent Program Which is Currently Running. [ID 735119.1]

33) How to Solve Concurrent Request Database Performance Issues [ID 863780.1]

34) How to move Concurrent Processing Server from one node to another node [ID 373611.1]

35) A closer examination of the Concurrent Program Purge Obsolete Workflow Runtime Data [ID 337923.1]

36) Concurrent Managers Will Only Start on One Node in PCP/RAC Environment [ID 264296.1]

37) How to Solve Concurrent Request Database Performance Issues [ID 863780.1]

38) How to Submit a Concurrent Request With CONCSUB Using a User Other Than APPS [ID 1050415.1]

39) How To Create A Custom Concurrent Program With Host Method and Pass Parameters To The Shell Script [ID 266268.1]

40) Troubleshooting AutoInvoice: Concurrent Request Failures (Request Status = Error) [ID 1082238.1]

41) When Submitting Create Accounting Program It Completes With Warning "Setting the Profile Option Concurrent: OPP Response Timeout to a Higher Value May Be Necessary" [ID 553320.1]

Hope this helps !!!!