Hi All,
Here we have Two Application nodes and user requests are controlled by hardware load balancer.
Few days back , the OS team complained us about the CPU 100% utilization by frmweb processes on both the servers.
Issue :
frmweb process are consuming 100%CPU utilization on the Server thereby increasing the load .
viz: On Oracle Applications R12 when checking the top processes on the OS level for the middle tier, you find that forms process (frmweb) almost consumes 100% of the CPU.
Cause :
The root cause of the issue is that returning rows from LOVs in core forms causes the forms process to grow up into memory depending on the number of rows returned.
When an end user login to forms and start working with LOV within core forms sometimes and according to the search criteria that the user will provide to filter the results in LOV, it may fetch huge numbers of records in which causes the frmweb process to grow very large, and in extreme cases this can even lock up the current process or even the whole machine.
So when executing a LOV query, every row is fetched into memory on the middle tier, the frmweb process can get extremely large, and the larger it gets the more likely it is to start paging.
Eventually it starts consuming excessive CPU just paging the process in and out of memory, which is probably what you can see here in this case as the amount of memory consumed when the LOV records are fetched into memory obviously depends on the amount of data in each record.
viz : The forms runaway processes used to consume the CPU %.
Solution:
IMP :: Bring down all the application services.
1- For Forms Variable "FORMS_CATCHTERM" the context vairable name is: "s_forms_catchterm" and you can update the context file located in ($INST_TOP/appl/admin/<SID_HOSTNAME.xml>)
2- For other forms variable "FORMS_RECORD_GROUP_MAX" there is no variable defined in Autoconfig for that one and have to customize the autoconfig for the forms variables to set that environment as following:
a- Go to the autoconfig Template folder:
$cd $AD_TOP/admin/template
b- Create new directory named (custom)
$ mkdir custom
c- Make sure that new directory has same file permissions as ($AD_TOP/admin/template)
d- Copy the following autoconfig template to the new custom directory:
$cp $AD_TOP/admin/template/APPLSYS_ux.env $AD_TOP/admin/template/custom/APPLSYS_ux.env
e- Edit the file copied file under custom directory and add the following 2 lines at the end of section:
####################################
# Oracle Forms environment variables
####################################
FORMS_RECORD_GROUP_MAX=10000
export FORMS_RECORD_GROUP_MAX
f- Save and exit from the file.
g- Next time autoconfig run, it will read the custom directory and check for any customizations there.
3. Run Autoconfig on the middle tier and make sure it is completed successfully.
4. Startup all services.
Monitor the CPU utilization on the server.
In our case ; we performed above mentioned Action plan on one server at a time; So no downtime is required ....... HURRAY !!! ;)
Refer : Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 [ID 745711.1]
