2 wj3319 wj3319 于 2016.01.20 20:33 提问

在系统访问高峰期出现无法获取数据库连接的异常 50C

最近遇到一个棘手的问题,请教各位能否查明报出异常的原因:
1:正常情况下系统运行没有问题,但是系统访问高峰情况下会出现,日志如下,这段日志出现一段时间之后,应用就会假死,持续1分钟左右之后恢复。
org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException:
Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException
: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/PnrDS
The error may exist in com/huifu/muser/common/dal/dao/MerUsrMapMapper.xml
The error may involve com.huifu.muser.common.dal.dao.MerUsrMapMapper.queryMerUsrMap
The error occurred while executing a query
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceE
xception: IJ000453: Unable to get managed connection for java:jboss/datasources/PnrDS
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:365)
at com.sun.proxy.$Proxy157.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:195)
at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:124)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:90)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:40)
at com.sun.proxy.$Proxy158.queryMerUsrMap(Unknown Source)
at com.huifu.muser.service.core.cash.impl.CashReqServiceImpl.checkMerUsrMap(CashReqServiceImpl.java:267)
at com.huifu.muser.biz.cash.impl.CashReqManager2Impl.cashConfirmReq(CashReqManager2Impl.java:211)
at com.huifu.muser.webapp.controller.cash.CashController.cashConfirm2(CashController.java:200)
at sun.reflect.GeneratedMethodAccessor2127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:426)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:414)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)

2:我的应用框架是 spring3.0.5+mybatis3.1.1+JBoss7.1+oracle11+jdk1.6
3:负载均衡 netscaler,两台虚拟机,四核,内存8G,jvm启动参数配置如下:
-XX:+UseCompressedOops -XX:+TieredCompilation -Xms4096m -Xmx4096m -Xmn1024m -XX:PermSize=256M -XX:MaxPermSize=512M -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:SurvivorRatio=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xnoclassgc -XX:+CMSClassUnloadingEnabled
4:我的数据库链接池是配置在jboss的配置文件standalone-ha.xml中,连接池配置为18-300,高峰期能达到300,即应用确实与数据库建立300链接,但是活跃链接在15左右,配置如下
图片说明

9个回答

devmiao
devmiao   Ds   Rxr 2016.01.20 20:37

Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection;
看下你的jdbc的最大连接数是多少?

wj3319
wj3319 18-300,文中有提到。
接近 2 年之前 回复
devmiao
devmiao   Ds   Rxr 2016.01.20 20:38
liao00001
liao00001   2016.01.20 22:07

jdbc最大连接数多少?提供片段读取写入的代码,看一下你重用连接的代码。

wj3319
wj3319 jdbc最大连接是18-300,文中有提到。
接近 2 年之前 回复
oyljerry
oyljerry   Ds   Rxr 2016.01.20 22:47

查看一下数据库的日志 里面一般有描述当前数据库连接的问题等 看是连接数太多还是数据处理问题

catclaw2000
catclaw2000   2016.01.20 23:28

" 连接池配置为18-300,高峰期能达到300 "

说明高峰期瞬间连接数确实可能超过300,那你分别把配置文件中max-pool-size调大一点,比如到400或500,然后确认Oracle最大进程数够用;再试试就可以了。

wj3319
wj3319 高峰期可以达到300,应用端可数据库服务端连接是达到300,但是观察数据库的连接情况,活跃连接数在15左右,即有可能是没有及时释放造成的,而不是真的达到300了
接近 2 年之前 回复
rui888
rui888   Ds   Rxr 2016.01.21 09:16
qq_16414307
qq_16414307   2016.01.21 14:49

直接原因就是 CannotGetJdbcConnectionException

当数据库连接到了设定最大值时,再想获得连接,就出错了

一方面,要检查是否有连接用完没关闭的情况

如果没问题,那么就要加大最大连接数了

你可以监控一下数据库本身,如果忙得时候很慢,会导致很多连接无法及时完成而无法释放,此时加大连接数只能更糟糕
这时你只能优化数据的软硬件和应用本身了

a549871449
a549871449   2016.06.13 11:51

楼主 我也遇到这样的问题 图片说明图片说明 我的解决思路是:1 首先排查代码 是否 connection 没释放 2 模拟一个http请求直接访问服务器 但是效果都不明显 楼主有啥好建议吗?求交流

a549871449
a549871449   2016.06.13 11:51

11:05:27,825 INFO Transaction [xwadmin]-queryPspGuarTaskList rollback current transaction...
11:05:27,826 INFO MVC [xwadmin]-queryPspGuarTaskList --------END--------VIEW=[psp/PspGuarTask/queryPspGuarTaskList.jsp]
11:05:27,828 INFO JMX Emmit notify from [CMISMvcServlet] as: Response too low in execute /cmis-main/queryPspGuarTaskList.do used time as:30060
11:05:44,904 ERROR stderr com.ecc.emp.jdbc.GetConnectionFailedException
11:05:44,906 ERROR MVC [113111633]-queryPubWbAllTaskList Failed to execute operation com.yucheng.cmis.pub.op.core.pubwbitem.QueryPubWbAllTaskListOp ! : com.ecc.emp.jdbc.GetConnectionFailedException: Failed to get connection from :com.ecc.emp.jdbc.JNDIDataSource@73220fd7 caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/WFDataSource cause: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/WFDataSource
at com.ecc.emp.jdbc.ConnectionManager.getConnection(ConnectionManager.java:57) [emp2.2-101210.jar:]
at com.yucheng.cmis.operation.CMISOperation.getConnection(CMISOperation.java:528) [classes:]
at com.yucheng.cmis.operation.CMISOperation.getConnection(CMISOperation.java:497) [classes:]
at com.yucheng.cmis.pub.op.core.pubwbitem.QueryPubWbAllTaskListOp.doExecute(QueryPubWbAllTaskListOp.java:24) [classes:]
at com.yucheng.cmis.operation.CMISOperation.execute(CMISOperation.java:182) [classes:]
at com.yucheng.cmis.ad.mvc.ADRequestController.executeEMPLogic(ADRequestController.java:113) [classes:]
at com.ecc.emp.web.servlet.mvc.EMPRequestController.executeTheFlow(EMPRequestController.java:487) [emp2.2-101210.jar:]
at com.ecc.emp.web.servlet.mvc.EMPRequestController.doRequest(EMPRequestController.java:307) [emp2.2-101210.jar:]
at com.ecc.emp.web.servlet.EMPRequestServlet.service(EMPRequestServlet.java:427) [emp2.2-101210.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jasig.cas.client.util.AssertionThreadLocalFilter.doFilter(AssertionThreadLocalFilter.java:54) [cas-client-core-extension-3.2.1.jar:3.2.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jasig.cas.client.util.HttpServletRequestWrapperFilter.doFilter(HttpServletRequestWrapperFilter.java:75) [cas-client-core-extension-3.2.1.jar:3.2.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jasig.cas.client.authentication.AuthenticationExtensionFilter.doFilter(AuthenticationExtensionFilter.java:106) [cas-client-core-extension-3.2.1.jar:3.2.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:76) [cas-client-core-extension-3.2.1.jar:3.2.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at com.ecc.emp.web.jsptags.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:44) [emp2.2-101210.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at com.yucheng.cmis.base.CMISZipFilter.doFilter(CMISZipFilter.java:39) [classes:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at com.ecc.emp.log.filter.Log4jFilter.doFilter(Log4jFilter.java:33) [classes:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final-RECOMPILE.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/WFDataSource
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:137)
at com.ecc.emp.jdbc.JNDIDataSource.getConnection(JNDIDataSource.java:97) [classes:]
at com.ecc.emp.jdbc.ConnectionManager.getConnection(ConnectionManager.java:44) [emp2.2-101210.jar:]
... 43 more
Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/WFDataSource
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:129)
... 45 more
Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:377)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)
... 48 more

11:05:44,966 INFO Transaction [113111633]-queryPubWbAllTaskList rollback current transaction...
11:05:44,967 INFO MVC [113111633]-queryPubWbAllTaskList --------END--------VIEW=[core/PubWbItem/queryPubWbAllTaskList.jsp]
11:05:44,969 INFO JMX Emmit notify from [CMISMvcServlet] as: Response too low in execute /cmis-main/queryPubWbAllTaskList.do used time as:30069
11:05:45,662 INFO stdout chenmingping
11:05:45,662 INFO MVC Accept request: /cmis-main/userSignOn.do
11:05:45,664 INFO Transaction [not session created]-userSignOn Start new transaction...
11:05:45,665 INFO [Transaction] (http--

Csdn user default icon
上传中...
上传图片
插入图片
准确详细的回答,更有利于被提问者采纳,从而获得C币。复制、灌水、广告等回答会被删除,是时候展现真正的技术了!