FMW11gR1 wls_ods1 Managed Server Throws java.lang.OutOfMemoryError After Running DIP With EBS Provisioning Profiles For Some Time

2014-09-05 11:26:00
sucheasy
原创
5549

Applies to:

Oracle Internet Directory - Version 11.1.1.1.0 to 11.1.1.6.0 [Release 11g]
Information in this document applies to any platform.

Symptoms

Oracle Identity Management (IDM) Directory Integration Platform (DIP) 11gR1 (11.1.1.x) is configured and running in a FusionMiddleware (FMW) WebLogic Server domain.

Oracle E-Business Suite (EBS) is installed elsewhere in the application architecture and is configured for integration with Oracle Internet Directory (OID) 11g.

Therefore DIP is running EBS two-way provisioning profiles.

After running for several days, the wls_ods1 Weblogic Managed Server running the DIP application throws an OutOfMemory exception.

For example the following message is seen in the wls_ods1-diagnostic.log file:

[2012-05-27T21:18:55.354+01:00] [wls_ods1] [WARNING] [DFW-40125] [oracle.dfw.incident] [tid: [ACTIVE].ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <WLS Kernel>] [ecid: bfceb245227eed4b:-4fdaca4b:1377427f05e:-8000-000000000001264c,0] incident flood controlled with Problem Key "OFM-99998 [Unknown][java.lang.OutOfMemoryError]"



This causes DIP provisioning between OID and EBS to stop and Oracle Directory Services Manager (ODSM) is not accessible.

At the time of the problem, the weblogic muxer threads are overloaded trying to handle inbound traffic.  At this point there are a huge number of DIP and ldap related threads, many of which are waiting on the Quartz scheduler threads.


If the DIP EBS provisioning profile(s) are disabled, the problem does not occur.


Other Symptoms

1. Java Heap Dump Analysis of the wls_ods1 server process shows 1000's of instances of oracle.dms.jmx.MetricMBeanInfo and oracle.dms.instrument.Noun.

Also in the thread dumps there are hundreds of these threads:

"Thread-581" daemon prio=3 tid=0x00000001049a2800 nid=0x294 runnable [0x7fffffff27e7e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331) 
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
- locked (a java.lang.Object)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755) 
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
- locked (a com.sun.net.ssl.internal.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked (a java.io.BufferedInputStream) 
at com.sun.jndi.ldap.Connection.run(Connection.java:808)
at java.lang.Thread.run(Thread.java:662


In the thread dumps there are a very large number of threads which appear to be workers for the DIP connections to the EBS server (appsrv01) :

"appsrv01_E" prio=3 tid=0x00000001035ae000 nid=0x5f in Object.wait() [0x7fffffff29f7f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
- locked (a org.quartz.simpl.SimpleThreadPool$WorkerThread


2. SQL is left running in the OID repository DB, which is causing blocking session.

select * from ODSQTZ_LOCKS where LOCK_NAME = :1 FOR UPDATE


3. DIP is not releasing JDBC sessions/connections to EBS database. Can cause subsequent connection failure because of reaching maximum sessions.

4. The following message may be seen in the wls_ods1 log:

<Mar 27, 2012 2:40:53 PM GMT> <Error> <oracle.dip.appsrv01_E> <DIP-20326> <Error while processing MODIFY operation>



Steps to reproduce


1. Start the wls_ods1 manaaged server, which has the DIP application deployed. DIP is configured to run the EBS provisioning profile(s).
2. Some time later, the managed server reports java.lang.OutOfMemoryError.



Cause


This issue is caused by Bug 13977226 - WLS_ODS1 OUTOFMEMORY (OOM) CONDITION FOR THE MANAGED SERVER.
Where DIP is running the bi-directional provisioning profiles for EBS-OID integration, DIP is failing to close the JDBC connections to the EBS database, eventually resulting in an out of memory error on the DIP (wls_ods1 managed server) machine.

This issue reproduces in all DIP 11gR1 releases up to and including 11.1.1.7.


Solution

To resolve this issue:


Download Patch 13977226 for your DIP release from My Oracle Support Patches and Updates and install following the patch README instructions.


NOTE: You may also need to clean up the tmp directory.
For example,
rm -rf $DOMAIN_HOME/servers/wls_ods1/tmp/*



文章分类
联系我们
联系人: 如易
Email: admin@sucheasy.com
QQ: 172787898
微信: sucheasy
地址: 北京市海淀区上地十街辉煌国际1号院1号楼20层