weblogic内存溢出outofmemory日志分析

Question

软件环境： aix 6.1+weblogic10.3+oracle11
系统每天同时在线人数100左右，数据库的数据量有多张频繁操作的表数据记录在千万以上，主要后台自动处理线程过多。
系统前段时间运行一直正常，只是从7月底出现OOM,最近差不多半个月出现一次。
对weblogic产生的phd文件和javacore文件分析的不太清楚。。。
以下是javacore的日志：

WARNING Java heap is almost exhausted : 0% free Java heap Please enable verbosegc trace and use IBM Pattern Modeling and Analysis Tool(http://www.alphaworks.ibm.com/tech/pmat) to analyze garbage collection activities. If heapdumps are generated at the same time, please use IBM HeapAnalyzer(http://www.alphaworks.ibm.com/tech/heapanalyzer) to analyze Java heap.

File name : C:\Users\Wcy\Desktop\20140813\ebills\heapdump\javacore.20140813.091552.10551312.0007.txt

Cause of thread dump : Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received

Date: 2014/08/13 at 09:16:49

Process ID : Not available

Operating System : AIX 6.1

Processor Architecture : ppc

Number of Processors : 12

Java version : JRE 1.6.0 IBM J9 2.4 AIX ppc-32 build jvmap3260sr9-20110624_85526

Virtual machine version : VM build 20110624_085526

Just-In-Time(JIT) compiler switch, Ahead-Of-Time (AOT) compiler switch, Compiler version : JIT enabled, AOT enabled - r9_20101028_17488ifx17

Garbage collector version : GC - 20101027_AA

Java Heap Information
Maximum Java heap size : 1024m
Initial Java heap size : 512m

Java Home Directory : /usr/java6/jre

Java DLL Directory : /usr/java6/jre/bin

System Classpath : /usr/java6/jre/lib/vm.jar;/usr/java6/jre/lib/annotation.jar;/usr/java6/jre/lib/beans.jar;/usr/java6/jre/lib/java.util.jar;/usr/java6/jre/lib/jndi.jar;/usr/java6/jre/lib/logging.jar;/usr/java6/jre/lib/security.jar;/usr/java6/jre/lib/sql.jar;/usr/java6/jre/lib/ibmorb.jar;/usr/java6/jre/lib/ibmorbapi.jar;/usr/java6/jre/lib/ibmcfw.jar;/usr/java6/jre/lib/rt.jar;/usr/java6/jre/lib/charsets.jar;/usr/java6/jre/lib/resources.jar;/usr/java6/jre/lib/ibmpkcs.jar;/usr/java6/jre/lib/ibmcertpathfw.jar;/usr/java6/jre/lib/ibmjgssfw.jar;/usr/java6/jre/lib/ibmjssefw.jar;/usr/java6/jre/lib/ibmsaslfw.jar;/usr/java6/jre/lib/ibmjcefw.jar;/usr/java6/jre/lib/ibmjgssprovider.jar;/usr/java6/jre/lib/ibmjsseprovider2.jar;/usr/java6/jre/lib/ibmcertpathprovider.jar;/usr/java6/jre/lib/ibmxmlcrypto.jar;/usr/java6/jre/lib/management-agent.jar;/usr/java6/jre/lib/xml.jar;/usr/java6/jre/lib/jlm.jar;/usr/java6/jre/lib/javascript.jar;

User Arguments : -Xjcl:jclscar_24 -Dcom.ibm.oti.vm.bootstrap.library.path=/usr/java6/jre/lib/ppc -Dsun.boot.library.path=/usr/java6/jre/lib/ppc -Djava.library.path=/usr/java6/jre/lib/ppc:/usr/java6/jre/lib/ppc:/usr/java6/jre/lib/ppc/j9vm:/usr/java6/jre/lib/ppc/j9vm:/usr/java6/jre/lib/ppc:/usr/java6/jre/../lib/ppc::/home/ebills/bea/wlserver_10.3/server/native/aix/ppc:/usr/lib:/usr/lib -Djava.home=/usr/java6/jre -Djava.ext.dirs=/usr/java6/jre/lib/ext -Duser.dir=/weblogic/ebills/bea/user_projects/domains/nbdomain _j2se_j9=71168 0xF0A89414 -Djava.runtime.version=pap3260sr9fp2-20110627_03 (SR9 FP2) -Xdump -Djava.class.path=:/home/ebills/bea/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/ebills/bea/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java6/lib/tools.jar:/home/ebills/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/home/ebills/bea/wlserver_10.3/server/lib/weblogic.jar:/home/ebills/bea/modules/features/weblogic.server.modules_10.3.0.0.jar:/home/ebills/bea/wlserver_10.3/server/lib/webservices.jar:/home/ebills/bea/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/home/ebills/bea/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar::/home/ebills/bea/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar:/home/ebills/bea/wlserver_10.3/server/lib/xqrl.jar:: -Xms512m -Xmx1024m -da -Dplatform.home=/home/ebills/bea/wlserver_10.3 -Dwls.home=/home/ebills/bea/wlserver_10.3/server -Dweblogic.home=/home/ebills/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole= -Dclient.encoding.override=GBK -Dfile.encoding=GBK -Duser.language=zh -Duser.region=CN -Ddefault.client.encoding=GBK -Dweblogic.threadpool.MinPoolSize=200 -Dweblogic.threadpool.MaxPoolSize=500 -Djava.awt.headless=true -Dweblogic.ext.dirs=/home/ebills/bea/patch_wls1030/profiles/default/sysext_manifest_classpath:/home/ebills/bea/patch_cie660/profiles/default/sysext_manifest_classpath -Dweblogic.management.username=weblogic -Dweblogic.management.password=weblogic -Dweblogic.Name=AdminServer -Djava.security.policy=/home/ebills/bea/wlserver_10.3/server/lib/weblogic.policy -Dsun.java.command=weblogic.Server -Dsun.java.launcher=SUN_STANDARD _port_library 0xF0A89C18 _org.apache.harmony.vmi.portlib 0x3013DE18

User Limit Analysis

User Limit Analysis
Type
Soft Limit
Hard Limit
RLIMIT_AS
unlimited
unlimited
RLIMIT_CORE
unlimited
unlimited
RLIMIT_CPU
unlimited
unlimited
RLIMIT_DATA
2,147,483,645 bytes
unlimited
RLIMIT_FSIZE
unlimited
unlimited
RLIMIT_NOFILE
1,024
1,024
RLIMIT_RSS
unlimited
unlimited
RLIMIT_STACK
2,147,483,646 bytes
2,147,483,646 bytes

Environment Variables Analysis

Environment Variable
Environment Variable
Value
_
/usr/java6/bin/java
POST_CLASSPATH
:/home/ebills/bea/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar:/home/ebills/bea/wlserver_10.3/server/lib/xqrl.jar
CLUSTER_PROPERTIES
-Dweblogic.management.discover=true
JAVA_VENDOR
IBM
LANG
Zh_CN
PRODUCTION_MODE
true
DOMAIN_HOME
/weblogic/ebills/bea/user_projects/domains/nbdomain
LOGIN
ebills
CLASSPATHSEP
:
DATABASE_CLASSPATH
/home/ebills/bea/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar
CIE660_PATCH_LIBPATH
/home/ebills/bea/patch_cie660/profiles/default/native
WLS1030_PATCH_EXT_DIR
/home/ebills/bea/patch_wls1030/profiles/default/sysext_manifest_classpath
POINTBASE_HOME
/home/ebills/bea/wlserver_10.3/common/eval/pointbase
SSH_TTY
/dev/pts/0
debugFlag
false
MEM_MAX_PERM_SIZE
-XX:MaxPermSize=1024m
SUN_JAVA_HOME

CLCMD_PASSTHRU
1
PATCH_CLASSPATH
/home/ebills/bea/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/ebills/bea/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar
PATH
/home/ebills/bea/wlserver_10.3/server/bin:/home/ebills/bea/modules/org.apache.ant_1.6.5/bin:/usr/java6/jre/bin:/usr/java6/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/home/ebills/bin:/usr/bin/X11:/sbin:/usr/java6/bin:/usr/java6/jre/bin:.
FEATURES_DIR
/home/ebills/bea/modules/features
CIE660_PATCH_PATH
/home/ebills/bea/patch_cie660/profiles/default/native
verboseLoggingFlag
false
ANT_CONTRIB
/home/ebills/bea/modules/net.sf.antcontrib_1.0.0.0_1-0b2
PATCH_PATH
/home/ebills/bea/patch_wls1030/profiles/default/native:/home/ebills/bea/patch_cie660/profiles/default/native
BEA_JAVA_HOME

JAVA_VM

ARDIR
/home/ebills/bea/wlserver_10.3/server/lib
LC__FASTMSG
true
POINTBASE_CLASSPATH
:/home/ebills/bea/wlserver_10.3/common/eval/pointbase/lib/pbembedded57.jar:/home/ebills/bea/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar
SSH_AUTH_SOCK
/tmp/ssh-IbX9437420/agent.9437420
CIE660_PATCH_CLASSPATH
/home/ebills/bea/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar
JAVA_PROFILE

JAVA_DEBUG

CLASSPATH
:/home/ebills/bea/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/ebills/bea/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java6/lib/tools.jar:/home/ebills/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/home/ebills/bea/wlserver_10.3/server/lib/weblogic.jar:/home/ebills/bea/modules/features/weblogic.server.modules_10.3.0.0.jar:/home/ebills/bea/wlserver_10.3/server/lib/webservices.jar:/home/ebills/bea/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/home/ebills/bea/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar::/home/ebills/bea/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar:/home/ebills/bea/wlserver_10.3/server/lib/xqrl.jar::
LOGNAME
ebills
SAMPLES_HOME
/home/ebills/bea/wlserver_10.3/samples
MAIL
/usr/spool/mail/ebills
enableHotswapFlag

POINTBASE_TOOLS
/home/ebills/bea/wlserver_10.3/common/eval/pointbase/lib/pbtools57.jar
LOCPATH
/usr/lib/nls/loc
MODULES_DIR
/home/ebills/bea/modules
PATCH_LIBPATH
/home/ebills/bea/patch_wls1030/profiles/default/native:/home/ebills/bea/patch_cie660/profiles/default/native
PATHSEP
:
WLS1030_PATCH_PATH
/home/ebills/bea/patch_wls1030/profiles/default/native
WLS1030_PATCH_CLASSPATH
/home/ebills/bea/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar
iterativeDevFlag
false
doExitFlag
false
USER
ebills
SERVER_NAME
AdminServer
AUTHSTATE
compat
SERVER_CLASS
weblogic.Server
PRE_CLASSPATH

JAVA_PROPERTIES
-Dplatform.home=/home/ebills/bea/wlserver_10.3 -Dwls.home=/home/ebills/bea/wlserver_10.3/server -Dweblogic.home=/home/ebills/bea/wlserver_10.3/server -Dweblogic.management.discover=true
BEA_HOME
/home/ebills/bea
MEM_DEV_ARGS

SHELL
/usr/bin/ksh
ODMDIR
/etc/objrepos
OMNIORB_CONFIG
/home/gjyw/ebills/config/ecorba.cfg
JAVA_HOME
/usr/java6
testConsoleFlag
false
WLS_HOME
/home/ebills/bea/wlserver_10.3/server
WLS1030_PATCH_LIBPATH
/home/ebills/bea/patch_wls1030/profiles/default/native
CIE660_PATCH_EXT_DIR
/home/ebills/bea/patch_cie660/profiles/default/sysext_manifest_classpath
JAVA_OPTIONS
-da -Dplatform.home=/home/ebills/bea/wlserver_10.3 -Dwls.home=/home/ebills/bea/wlserver_10.3/server -Dweblogic.home=/home/ebills/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole= -Dclient.encoding.override=GBK -Dfile.encoding=GBK -Duser.language=zh -Duser.region=CN -Ddefault.client.encoding=GBK -Dweblogic.threadpool.MinPoolSize=200 -Dweblogic.threadpool.MaxPoolSize=500 -Djava.awt.headless=true -Dweblogic.ext.dirs=/home/ebills/bea/patch_wls1030/profiles/default/sysext_manifest_classpath:/home/ebills/bea/patch_cie660/profiles/default/sysext_manifest_classpath -Dweblogic.management.username=weblogic -Dweblogic.management.password=weblogic
HOME
/home/ebills
MEM_ARGS
-Xms512m -Xmx1024m
WEBLOGIC_EXTENSION_DIRS
/home/ebills/bea/patch_wls1030/profiles/default/sysext_manifest_classpath:/home/ebills/bea/patch_cie660/profiles/default/sysext_manifest_classpath
ANT_HOME
/home/ebills/bea/modules/org.apache.ant_1.6.5
FILEDIR
/home/gjyw/ebills/acct
SSH_CONNECTION
10.100.67.22 3456 10.100.133.2 22
SSH_CLIENT
10.100.67.22 3456 22
WEBLOGIC_CLASSPATH
/home/ebills/bea/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/ebills/bea/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java6/lib/tools.jar:/home/ebills/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/home/ebills/bea/wlserver_10.3/server/lib/weblogic.jar:/home/ebills/bea/modules/features/weblogic.server.modules_10.3.0.0.jar:/home/ebills/bea/wlserver_10.3/server/lib/webservices.jar:/home/ebills/bea/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/home/ebills/bea/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar
TERM
vt100
MAILMSG
[YOU HAVE NEW MAIL]
POINTBASE_FLAG
false
LONG_DOMAIN_HOME
/weblogic/ebills/bea/user_projects/domains/nbdomain
PWD
/weblogic/ebills/bea/user_projects/domains/nbdomain
TZ
Asia/Shanghai
MEM_PERM_SIZE
-XX:PermSize=512m
BZJFILEDIR
/weblogic/ebills/gjyw/ebills/ebillsData/DepositAcct
WL_HOME
/home/ebills/bea/wlserver_10.3
DEBUG_PORT
8453
A__z
! LOGNAME
IBM_JVM_AIXTHREAD_SCOPE_NEW_VALUE
S
AIXTHREAD_SCOPE
S
IBM_JVM_CHANGED_ENVVARS_10551312
AIXTHREAD_SCOPE,NULLPTR,CORE_MMAP,LDR_CNTRL
IBM_JVM_NULLPTR_NEW_VALUE
NOSEGV
NULLPTR
NOSEGV
IBM_JVM_CORE_MMAP_NEW_VALUE
yes
CORE_MMAP
yes
IBM_JVM_LDR_CNTRL_NEW_VALUE
MAXDATA=0XA0000000@DSA
LDR_CNTRL
MAXDATA=0XA0000000@DSA
NLSPATH
/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat
LIBPATH
/usr/java6/jre/lib/ppc:/usr/java6/jre/lib/ppc/j9vm:/usr/java6/jre/lib/ppc/j9vm:/usr/java6/jre/lib/ppc:/usr/java6/jre/../lib/ppc::/home/ebills/bea/wlserver_10.3/server/native/aix/ppc:/usr/lib:/usr/java6/jre/lib/ppc/headless
IBM_JAVA_COMMAND_LINE
/usr/java6/bin/java -Xms512m -Xmx1024m -da -Dplatform.home=/home/ebills/bea/wlserver_10.3 -Dwls.home=/home/ebills/bea/wlserver_10.3/server -Dweblogic.home=/home/ebills/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole= -Dclient.encoding.override=GBK -Dfile.encoding=GBK -Duser.language=zh -Duser.region=CN -Ddefault.client.encoding=GBK -Dweblogic.threadpool.MinPoolSize=200 -Dweblogic.threadpool.MaxPoolSize=500 -Djava.awt.headless=true -Dweblogic.ext.dirs=/home/ebills/bea/patch_wls1030/profiles/default/sysext_manifest_classpath:/home/ebills/bea/patch_cie660/profiles/default/sysext_manifest_classpath -Dweblogic.management.username=weblogic -Dweblogic.management.password=weblogic -Dweblogic.Name=AdminServer -Djava.security.policy=/home/ebills/bea/wlserver_10.3/server/lib/weblogic.policy weblogic.Server

Free Java heap size: 11.89 KB

Allocated Java heap size: 1 GB

Memory Segment Analysis

Memory Segment Analysis
Memory Type

of Segments

Used Memory(bytes)
Used Memory(%)
Free Memory(bytes)
Free Memory(%)
Total Memory(bytes)
Internal
431
0
0
28,332,996
100
28,332,996
Object(reserved)
1
1,073,741,824
100
0
0
1,073,741,824
Class
17,984
174,683,076
96.44
6,441,464
3.56
181,124,540
JIT Code Cache
7
58,720,256
100
0
0
58,720,256
JIT Data Cache
4
26,811,512
79.9
6,742,920
20.1
33,554,432
Overall
18,427
1,333,956,668
96.98
41,517,380
3.02
1,375,474,048

Current Thread :

Thread Name
[ACTIVE] ExecuteThread: '119' for queue: 'weblogic.kernel.Default (self-tuning)'
State
Runnable
Java Stack
at java/lang/String. (String.java:350(Compiled Code))
at java/lang/Throwable.printStackTrace(Throwable.java:369(Compiled Code))
at java/lang/Throwable.printStackTrace(Throwable.java:212(Compiled Code))
at weblogic/utils/StackTraceUtilsClient.throwable2StackTrace(StackTraceUtilsClient.java:25(Compiled Code))
at weblogic/jdbc/common/internal/ConnectionEnv.setup(ConnectionEnv.java:308(Compiled Code))
at weblogic/common/resourcepool/ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:303(Compiled Code))
at weblogic/jdbc/common/internal/ConnectionPool.reserve(ConnectionPool.java:427(Compiled Code))
at weblogic/jdbc/common/internal/ConnectionPool.reserve(ConnectionPool.java:316(Compiled Code))
at weblogic/jdbc/common/internal/ConnectionPoolManager.reserve(ConnectionPoolManager.java:85(Compiled Code))
at weblogic/jdbc/common/internal/ConnectionPoolManager.reserve(ConnectionPoolManager.java:61(Compiled Code))
at weblogic/jdbc/jta/DataSource.getXAConnectionFromPool(DataSource.java:1450(Compiled Code))
at weblogic/jdbc/jta/DataSource.refreshXAConnAndEnlist(DataSource.java:1272(Compiled Code))
at weblogic/jdbc/jta/DataSource.getConnection(DataSource.java:425(Compiled Code))
at weblogic/jdbc/jta/DataSource.connect(DataSource.java:382(Compiled Code))
at weblogic/jdbc/common/internal/RmiDataSource.getConnection(RmiDataSource.java:336(Compiled Code))
at com/amerisia/ebills/commons/util/ServiceLocator.getConnection(ServiceLocator.java:66(Compiled Code))
at com/amerisia/ebills/commons/util/BaseDAO.makeConnection(BaseDAO.java:72(Compiled Code))
at com/amerisia/ebills/commons/util/BaseDAO.loadAllRowBySql(BaseDAO.java:623(Compiled Code))
at com/amerisia/ebills/chat/ejbs/ChatDAO.getLeaveWord(ChatDAO.java:119(Compiled Code))
at com/amerisia/ebills/chat/ejbs/ChatManagerEJB.getLeaveWord(ChatManagerEJB.java:165(Compiled Code))
at com/amerisia/ebills/chat/ejbs/ChatManager_oxcdb1_ELOImpl.getLeaveWord(ChatManager_oxcdb1_ELOImpl.java:358(Compiled Code))
at com/amerisia/ebills/chat/facade/ChatNoteManagerEJB.getLeaveWord(ChatNoteManagerEJB.java:110(Compiled Code))
at com/amerisia/ebills/chat/facade/ChatNoteManager_hfunq5_EOImpl.getLeaveWord(ChatNoteManager_hfunq5_EOImpl.java:339(Compiled Code))
at com/amerisia/ebills/chat/facade/ChatNoteManager_hfunq5_EOImpl_WLSkel.invoke(Bytecode PC:240(Compiled Code))
at weblogic/rmi/internal/ServerRequest.sendReceive(ServerRequest.java:174(Compiled Code))
at weblogic/rmi/cluster/ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345(Compiled Code))
at weblogic/rmi/cluster/ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259(Compiled Code))
at com/amerisia/ebills/chat/facade/ChatNoteManager_hfunq5_EOImpl_1030_WLStub.getLeaveWord(Bytecode PC:37(Compiled Code))
at com/amerisia/ebills/chat/action/ChatAction.getMsgsForLeft(ChatAction.java:466(Compiled Code))
at sun/reflect/GeneratedMethodAccessor234.invoke(Bytecode PC:64(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:589(Compiled Code))
at com/amerisia/ebills/commons/action/EbillsBaseDispatchAction.dispatchMethod(EbillsBaseDispatchAction.java:1072(Compiled Code))
at org/apache/struts/actions/DispatchAction.execute(DispatchAction.java:194(Compiled Code))
at org/apache/struts/action/RequestProcessor.processActionPerform(RequestProcessor.java(Compiled Code))
at org/apache/struts/action/RequestProcessor.process(RequestProcessor.java:203(Compiled Code))
at org/apache/struts/action/ActionServlet.process(ActionServlet.java:1196(Compiled Code))
at org/apache/struts/action/ActionServlet.doGet(ActionServlet.java:414(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:707(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820(Compiled Code))
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227(Compiled Code))
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:121(Compiled Code))
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:292(Compiled Code))
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26(Compiled Code))
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42(Compiled Code))
at com/amerisia/ebills/commons/web/EncodingFilter.doFilter(EncodingFilter.java(Compiled Code))
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42(Compiled Code))
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3486(Compiled Code))
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321(Compiled Code))
at weblogic/security/service/SecurityManager.runAs(Bytecode PC:18(Compiled Code))
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2120(Compiled Code))
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2086(Compiled Code))
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1406(Compiled Code))
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201(Compiled Code))
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173(Compiled Code))
Native Stack
(0xD2647A48 [libj9prt24.so+0x9a48])
(0xD2BA2C38 [libj9dmp24.so+0x10c38])
(0xD263FBAC [libj9prt24.so+0x1bac])
(0xD2BA0AB4 [libj9dmp24.so+0xeab4])
(0xD2B9F0CC [libj9dmp24.so+0xd0cc])
(0xD263FBAC [libj9prt24.so+0x1bac])
(0xD2B9ED10 [libj9dmp24.so+0xcd10])
(0xD2BA5280 [libj9dmp24.so+0x13280])
(0xD2B944DC [libj9dmp24.so+0x24dc])
(0xD2B983A0 [libj9dmp24.so+0x63a0])
(0xD263FBAC [libj9prt24.so+0x1bac])
(0xD2B98344 [libj9dmp24.so+0x6344])
(0xD2B98130 [libj9dmp24.so+0x6130])
(0xD2BB3A94 [libj9dmp24.so+0x21a94])
(0xD2BB3ED0 [libj9dmp24.so+0x21ed0])
(0xD0ED2204 [libj9hookable24.so+0x204])
(0xD4007228 [libj9vm24.so+0x9228])
(0xD4007C8C [libj9vm24.so+0x9c8c])
(0xD5016A8C [libj9jit24.so+0x56aa8c])
(0xD4009910 [libj9vm24.so+0xb910])
(0xD263FBAC [libj9prt24.so+0x1bac])
(0xD4009830 [libj9vm24.so+0xb830])
(0xD2197CC0 [libj9thr24.so+0x1cc0])
_pthread_body+0xec (0xD04EFC50 [libpthreads.a+0x3c50])

Number of loaded classes in Java heap : 34,648

Number of classloaders in Java heap : 9,321

Recommended -Xmxcl setting (only for IBM Java 5.0, up to and including Service Refresh 4 (build date:February 1st ,2007)) : 12,117 or greater

note: oNLY FOR jAVA 5.0 sERVICE rEFRESH 4 (BUILD DATE:fEBRUARY 1ST, 2007) AND OLDER. wHEN YOU USE DELEGATED CLASS LOADERS, THE jvm CAN CREATE A LARGE NUMBER OF cLASSlOADER OBJECTS. oN ibm jAVA 5.0 sERVICE rEFRESH 4 AND OLDER, THE NUMBER OF CLASS LOADERS THAT ARE PERMITTED IS LIMITED TO 8192 BY DEFAULT AND AN oUToFmEMORYeRROR EXCEPTION IS THROWN WHEN THIS LIMIT IS EXCEEDED. uSE THE -xMXCL PARAMETER TO INCREASE THE NUMBER OF CLASS LOADERS ALLOWED TO AVOID THIS PROBLEM, FOR EXAMPLE TO 25000, BY SETTING -xMXCL25000, UNTIL THE PROBLEM IS RESOLVED. pLEASE EXAMINE THE CURRENT THREAD STACK TRACE TO CHECK WHETHER A CLASS LOADER IS BEING LOADED IF THERE IS AN oUToFmEMORYeRROR. fOR EXAMPLE, THE FOLLOWING STACK TRACE INDICATES THAT A CLASS LOADER IS BEING LOADED: AT COM/IBM/OTI/VM/vm.INITIALIZEcLASSlOADER(nATIVE mETHOD) AT JAVA/LANG/cLASSlOADER. (cLASSlOADER.JAVA:120)

cOMMAND LINE : /USR/JAVA6/BIN/JAVA -xMS512M -xMX1024M -DA -dPLATFORM.HOME=/HOME/EBILLS/BEA/WLSERVER_10.3 -dWLS.HOME=/HOME/EBILLS/BEA/WLSERVER_10.3/SERVER -dWEBLOGIC.HOME=/HOME/EBILLS/BEA/WLSERVER_10.3/SERVER -dWEBLOGIC.MANAGEMENT.DISCOVER=TRUE -dWLW.ITERATIVEdEV=FALSE -dWLW.TESTcONSOLE=FALSE -dWLW.LOGeRRORStOcONSOLE= -dCLIENT.ENCODING.OVERRIDE=gbk -dFILE.ENCODING=gbk -dUSER.LANGUAGE=ZH -dUSER.REGION=cn -dDEFAULT.CLIENT.ENCODING=gbk -dWEBLOGIC.THREADPOOL.mINpOOLsIZE=200 -dWEBLOGIC.THREADPOOL.mAXpOOLsIZE=500 -dJAVA.AWT.HEADLESS=TRUE -dWEBLOGIC.EXT.DIRS=/HOME/EBILLS/BEA/PATCH_WLS1030/PROFILES/DEFAULT/SYSEXT_MANIFEST_CLASSPATH:/HOME/EBILLS/BEA/PATCH_CIE660/PROFILES/DEFAULT/SYSEXT_MANIFEST_CLASSPATH -dWEBLOGIC.MANAGEMENT.USERNAME=WEBLOGIC -dWEBLOGIC.MANAGEMENT.PASSWORD=WEBLOGIC -dWEBLOGIC.nAME=aDMINsERVER -dJAVA.SECURITY.POLICY=/HOME/EBILLS/BEA/WLSERVER_10.3/SERVER/LIB/WEBLOGIC.POLICY WEBLOGIC.sERVER

tHREAD sTATUS aNALYSIS

sTATUS
nUMBER OF tHREADS : 287
Percentage
Deadlock
0
0 (%)
Runnable
6
2 (%)
Waiting on condition
263
92 (%)
Waiting on monitor
0
0 (%)
Suspended
0
0 (%)
Object.wait()
0
0 (%)
Blocked
16
6 (%)
Parked
2
1 (%)
Thread Method Analysis
Method Name
Number of Threads : 287
Percentage
java/lang/Object.wait(Native Method)
167
58 (%)
java/lang/Thread.sleep(Native Method)
22
8 (%)
NO JAVA STACK
13
5 (%)
weblogic/socket/PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93(Compiled Code))
10
3 (%)
weblogic/timers/internal/TimerManagerImpl.complete(TimerManagerImpl.java:654(Compiled Code))
6
2 (%)
java/lang/Throwable.fillInStackTrace(Native Method)
5
2 (%)
java/net/PlainSocketImpl.socketAccept(Native Method)
5
2 (%)
weblogic/servlet/internal/session/SessionContext$SessionInvalidator.timerExpired(SessionContext.java:788(Compiled Code))
2
1 (%)
sun/misc/Unsafe.park(Native Method)
2
1 (%)
java/util/AbstractList.iterator(AbstractList.java:583(Compiled Code))
2
1 (%)
java/io/ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3036(Compiled Code))
2
1 (%)
java/lang/StringBuilder.toString(StringBuilder.java:803(Compiled Code))
2
1 (%)
sun/reflect/UTF8.encode(UTF8.java:33(Compiled Code))
2
1 (%)
java/nio/CharBuffer.wrap(CharBuffer.java:361(Compiled Code))
2
1 (%)
java/util/Hashtable.newEntry(Hashtable.java:91(Compiled Code))
2
1 (%)
java/util/regex/Pattern.newSingle(Pattern.java:2962(Compiled Code))
1
0 (%)
java/util/Vector. (Vector.java:76(Compiled Code))
1
0 (%)
java/util/TreeMap.putImpl(TreeMap.java:4544(Compiled Code))
1
0 (%)
java/util/Hashtable.put(Hashtable.java:769(Compiled Code))
1
0 (%)
java/util/Hashtable.clone(Hashtable.java:327(Compiled Code))
1
0 (%)
java/util/AbstractList$FullListIterator. (AbstractList.java:94(Compiled Code))
1
0 (%)
java/util/AbstractCollection.toArray(AbstractCollection.java:352(Compiled Code))
1
0 (%)
java/security/SecureRandom.nextBytes(SecureRandom.java:292(Compiled Code))
1
0 (%)
java/net/SocketInputStream.read(SocketInputStream.java:179(Compiled Code))
1
0 (%)
java/net/Inet6AddressImpl.lookupAllHostAddr(Native Method)
1
0 (%)
java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
1
0 (%)
java/lang/reflect/Array.newArrayImpl(Native Method)
1
0 (%)
java/lang/Throwable.printStackTrace(Throwable.java:363(Compiled Code))
1
0 (%)
java/lang/Throwable.printStackTrace(Throwable.java:338(Compiled Code))
1
0 (%)
java/lang/StringBuilder.ensureCapacityImpl(StringBuilder.java:339(Compiled Code))
1
0 (%)
java/lang/String.toUpperCase(String.java:1300(Compiled Code))
1
0 (%)
java/lang/String.toUpperCase(String.java:1129(Compiled Code))
1
0 (%)
java/lang/String. (String.java:350(Compiled Code))
1
0 (%)
java/io/ObjectStreamClass.lookup(ObjectStreamClass.java:291(Compiled Code))
1
0 (%)
java/io/ObjectStreamClass.lookup(ObjectStreamClass.java:287(Compiled Code))
1
0 (%)
java/io/ObjectInputStream.readString(ObjectInputStream.java:1629(Compiled Code))
1
0 (%)
com/ibm/tools/attach/javaSE/IPC.waitSemaphore(Native Method)
1
0 (%)
com/ibm/misc/SignalDispatcher.waitForSignal(Native Method)
1
0 (%)
com/ibm/lang/management/OperatingSystemNotificationThread.processNotificationLoop(Native Method)
1
0 (%)
com/ibm/lang/management/MemoryNotificationThread.processNotificationLoop(Native Method)
1
0 (%)
com/bea/security/xacml/combinator/standard/StandardRuleCombinerLibrary$3$1.evaluate(StandardRuleCombinerLibrary.java:200(Compiled Code))
1
0 (%)
com/amerisia/ebills/schedule/data/FileSource.findAll(FileSource.java:213(Compiled Code))
1
0 (%)
com/amerisia/ebills/commons/util/BaseDAO.loadAllRowBySql(BaseDAO.java:644(Compiled Code))
1
0 (%)
weblogic/utils/io/ChunkedDataOutputStream.makeChunkedDataInputStream(ChunkedDataOutputStream.java:376(Compiled Code))
1
0 (%)
weblogic/utils/http/QueryParams.getCurrent(QueryParams.java:21(Compiled Code))
1
0 (%)
weblogic/utils/collections/SecondChanceCacheMap.put(SecondChanceCacheMap.java:72(Compiled Code))
1
0 (%)
weblogic/socket/PosixSocketMuxer.poll(Native Method)
1
0 (%)
weblogic/servlet/internal/MuxableSocketHTTP. (MuxableSocketHTTP.java:112(Compiled Code))
1
0 (%)
weblogic/security/service/SecurityManager.runAs(Bytecode PC:18(Compiled Code))
1
0 (%)
weblogic/security/service/JNDIResource.initialize(JNDIResource.java:146(Compiled Code))
1
0 (%)
weblogic/jndi/internal/BasicNamingNode.getPrefix(BasicNamingNode.java:984(Compiled Code))
1
0 (%)
sun/util/calendar/ZoneInfo.getOffsets(ZoneInfo.java:237(Compiled Code))
1
0 (%)
sun/reflect/GeneratedMethodAccessor120.invoke(Bytecode PC:0(Compiled Code))
1
0 (%)
org/apache/struts/util/RequestUtils.populate(RequestUtils.java:399(Compiled Code))
1
0 (%)
oracle/net/ano/AnoComm.o(Bytecode PC:1(Compiled Code))
1
0 (%)
oracle/jdbc/driver/T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:801(Compiled Code))
1
0 (%)
oracle/jdbc/driver/T4CMAREngine.unmarshalKEYVAL(T4CMAREngine.java:1859(Compiled Code))
1
0 (%)
oracle/jdbc/driver/DBConversion.stringToDriverCharBytes(DBConversion.java:443(Compiled Code))
1
0 (%)
Thread Aggregation Analysis
Thread Type
Number of Threads : 287
Percentage
Thread
276
96 (%)