Applies to:Oracle Internet Directory - Version 220.127.116.11.0 to 18.104.22.168.0 [Release 11g]
Information in this document applies to any platform.
Oracle Identity Management (IDM) Directory Integration Platform (DIP) 11gR1 (11.1.1.x) is configured and running in a FusionMiddleware (FMW) WebLogic Server domain.
Oracle E-Business Suite (EBS) is installed elsewhere in the application architecture and is configured for integration with Oracle Internet Directory (OID) 11g.
Therefore DIP is running EBS two-way provisioning profiles.
After running for several days, the wls_ods1 Weblogic Managed Server running the DIP application throws an OutOfMemory exception.
For example the following message is seen in the wls_ods1-diagnostic.log file:
This causes DIP provisioning between OID and EBS to stop and Oracle Directory Services Manager (ODSM) is not accessible.
At the time of the problem, the weblogic muxer threads are overloaded trying to handle inbound traffic. At this point there are a huge number of DIP and ldap related threads, many of which are waiting on the Quartz scheduler threads.
If the DIP EBS provisioning profile(s) are disabled, the problem does not occur.
1. Java Heap Dump Analysis of the wls_ods1 server process shows 1000's of instances of oracle.dms.jmx.MetricMBeanInfo and oracle.dms.instrument.Noun.
Also in the thread dumps there are hundreds of these threads:
"Thread-581" daemon prio=3 tid=0x00000001049a2800 nid=0x294 runnable [0x7fffffff27e7e000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798) - locked (a java.lang.Object) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755) at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) - locked (a com.sun.net.ssl.internal.ssl.AppInputStream) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) - locked (a java.io.BufferedInputStream) at com.sun.jndi.ldap.Connection.run(Connection.java:808) at java.lang.Thread.run(Thread.java:662
In the thread dumps there are a very large number of threads which appear to be workers for the DIP connections to the EBS server (appsrv01) :
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
- locked (a org.quartz.simpl.SimpleThreadPool$WorkerThread
2. SQL is left running in the OID repository DB, which is causing blocking session.
select * from ODSQTZ_LOCKS where LOCK_NAME = :1 FOR UPDATE
3. DIP is not releasing JDBC sessions/connections to EBS database. Can cause subsequent connection failure because of reaching maximum sessions.
4. The following message may be seen in the wls_ods1 log:
<Mar 27, 2012 2:40:53 PM GMT> <Error> <oracle.dip.appsrv01_E> <DIP-20326> <Error while processing MODIFY operation>
Steps to reproduce
1. Start the wls_ods1 manaaged server, which has the DIP application deployed. DIP is configured to run the EBS provisioning profile(s). 2. Some time later, the managed server reports java.lang.OutOfMemoryError.
This issue is caused by Bug 13977226 - WLS_ODS1 OUTOFMEMORY (OOM) CONDITION FOR THE MANAGED SERVER.
Where DIP is running the bi-directional provisioning profiles for EBS-OID integration, DIP is failing to close the JDBC connections to the EBS database, eventually resulting in an out of memory error on the DIP (wls_ods1 managed server) machine.
This issue reproduces in all DIP 11gR1 releases up to and including 22.214.171.124.
To resolve this issue:
Download Patch 13977226 for your DIP release from My Oracle Support Patches and Updates and install following the patch README instructions.
rm -rf $DOMAIN_HOME/servers/wls_ods1/tmp/*