独家记忆shine 2017-11-16 06:05 采纳率: 42.9%
浏览 9650
已采纳

我们的生产环境dubbo服务出现了重大问题:求各位大神帮忙解决下。具体异常信息日志如下

问题描述:
部署在测试环境上的dubbo服务,有时候会自动down掉,别人调用不了,但是我们人为没有动过它。这种出现概率平均1-2天发生1次,不知道什么原因引起的。目前还解决不了。如果到线上还这样,会影响公司的交易量的,很危险。
看了下日志报错是这样的:
1:客户端调用方的日志:
com.alibaba.dubbo.rpc.RpcException: Forbid consumer 192.168.18.86 access service com.niiwoo.rce.service.RCEComputeService from registry 192.168.18.86:2181 use dubbo version 2.8.4, Please check registry access list (whitelist/blacklist).
at com.alibaba.dubbo.registry.integration.RegistryDirectory.doList(RegistryDirectory.java:579)
at com.alibaba.dubbo.rpc.cluster.directory.AbstractDirectory.list(AbstractDirectory.java:73)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.list(AbstractClusterInvoker.java:260)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:219)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy0.updateVariable(proxy0.java)
at com.tianchengsys.rdp.dubbo.consumer.service.imp.VariableServiceImpl.updateVariable(VariableServiceImpl.java:30)
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl.updateVariable(McsVariableServiceImpl.java:108)
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl$$FastClassBySpringCGLIB$$fbc6f996.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl$$EnhancerBySpringCGLIB$$fef2ce11.updateVariable()
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl$$FastClassBySpringCGLIB$$fbc6f996.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl$$EnhancerBySpringCGLIB$$9282b9c7.updateVariable()
at com.tianchengsys.rdp.web.controller.mcs2.McsVariableController.edit(McsVariableController.java:289)
at com.tianchengsys.rdp.web.controller.mcs2.McsVariableController$$FastClassBySpringCGLIB$$e5da7779.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.apache.shiro.spring.security.interceptor.AopAllianceAnnotationsAuthorizingMethodInterceptor$1.proceed(AopAllianceAnnotationsAuthorizingMethodInterceptor.java:82)
at org.apache.shiro.authz.aop.AuthorizingMethodInterceptor.invoke(AuthorizingMethodInterceptor.java:39)
at org.apache.shiro.spring.security.interceptor.AopAllianceAnnotationsAuthorizingMethodInterceptor.invoke(AopAllianceAnnotationsAuthorizingMethodInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)

2:dubbo服务相关日志
2017-11-11 07:26:35,994 [main-SendThread(192.168.18.86:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1096) - Client session timed out, have not heard from server in 40281ms for sessionid 0x45f95de2d891148, closing socket connection and attempting reconnect
2017-11-11 07:26:52,703 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:975) - Opening socket connection to server 192.168.18.90/192.168.18.90:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-11 07:27:03,752 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:852) - Socket connection established to 192.168.18.90/192.168.18.90:2181, initiating session
2017-11-11 07:29:21,928 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1096) - Client session timed out, have not heard from server in 149223ms for sessionid 0x45f95de2d891148, closing socket connection and attempting reconnect
2017-11-11 07:29:55,024 [main-SendThread(192.168.18.89:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:975) - Opening socket connection to server 192.168.18.89/192.168.18.89:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-11 07:29:55,025 [main-SendThread(192.168.18.89:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:852) - Socket connection established to 192.168.18.89/192.168.18.89:2181, initiating session
2017-11-11 07:29:55,448 [main-SendThread(192.168.18.89:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1094) - Unable to reconnect to ZooKeeper service, session 0x45f95de2d891148 has expired, closing socket connection
2017-11-11 07:30:31,361 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:975) - Opening socket connection to server 192.168.18.90/192.168.18.90:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-11 07:30:31,362 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:852) - Socket connection established to 192.168.18.90/192.168.18.90:2181, initiating session
2017-11-11 07:30:31,777 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1235) - Session establishment complete on server 192.168.18.90/192.168.18.90:2181, sessionid = 0x35e93bd6a5cbc61, negotiated timeout = 60000
2017-11-11 07:31:41,632 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1096) - Client session timed out, have not heard from server in 41620ms for sessionid 0x35e93bd6a5cbc61, closing socket connection and attempting reconnect
2017-11-11 07:31:58,063 [main-SendThread(192.168.18.86:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:975) - Opening socket connection to server 192.168.18.86/192.168.18.86:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-11 07:32:05,662 [main-SendThread(192.168.18.86:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:852) - Socket connection established to 192.168.18.86/192.168.18.86:2181, initiating session

  • 写回答

9条回答 默认 最新

  • 如影随形小浩儿 2017-11-17 02:32
    关注

    没有遇到过这种问题,看日志很可能是网络超时的原因。

    还有一种可能,调用服务时,连接一直阻塞,然后连接大量增加的情况下,直接让服务宕掉,这个是由于dubbo没有熔断机制的原因造成的
    建议你监测一下你的服务那边的内存情况以及线程情况。

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(8条)

报告相同问题?

问题事件

  • 已采纳回答 10月11日

悬赏问题

  • ¥15 自动转发微信群信息到另外一个微信群
  • ¥15 outlook无法配置成功
  • ¥30 这是哪个作者做的宝宝起名网站
  • ¥60 版本过低apk如何修改可以兼容新的安卓系统
  • ¥25 由IPR导致的DRIVER_POWER_STATE_FAILURE蓝屏
  • ¥50 有数据,怎么建立模型求影响全要素生产率的因素
  • ¥50 有数据,怎么用matlab求全要素生产率
  • ¥15 TI的insta-spin例程
  • ¥15 完成下列问题完成下列问题
  • ¥15 C#算法问题, 不知道怎么处理这个数据的转换