独家记忆shine
2017-11-16 06:05
采纳率: 64.7%
浏览 9.6k
已采纳

我们的生产环境dubbo服务出现了重大问题:求各位大神帮忙解决下。具体异常信息日志如下

问题描述:
部署在测试环境上的dubbo服务,有时候会自动down掉,别人调用不了,但是我们人为没有动过它。这种出现概率平均1-2天发生1次,不知道什么原因引起的。目前还解决不了。如果到线上还这样,会影响公司的交易量的,很危险。
看了下日志报错是这样的:
1:客户端调用方的日志:
com.alibaba.dubbo.rpc.RpcException: Forbid consumer 192.168.18.86 access service com.niiwoo.rce.service.RCEComputeService from registry 192.168.18.86:2181 use dubbo version 2.8.4, Please check registry access list (whitelist/blacklist).
at com.alibaba.dubbo.registry.integration.RegistryDirectory.doList(RegistryDirectory.java:579)
at com.alibaba.dubbo.rpc.cluster.directory.AbstractDirectory.list(AbstractDirectory.java:73)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.list(AbstractClusterInvoker.java:260)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:219)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy0.updateVariable(proxy0.java)
at com.tianchengsys.rdp.dubbo.consumer.service.imp.VariableServiceImpl.updateVariable(VariableServiceImpl.java:30)
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl.updateVariable(McsVariableServiceImpl.java:108)
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl$$FastClassBySpringCGLIB$$fbc6f996.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl$$EnhancerBySpringCGLIB$$fef2ce11.updateVariable()
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl$$FastClassBySpringCGLIB$$fbc6f996.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
at com.tianchengsys.rdp.service.impl.McsVariableServiceImpl$$EnhancerBySpringCGLIB$$9282b9c7.updateVariable()
at com.tianchengsys.rdp.web.controller.mcs2.McsVariableController.edit(McsVariableController.java:289)
at com.tianchengsys.rdp.web.controller.mcs2.McsVariableController$$FastClassBySpringCGLIB$$e5da7779.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.apache.shiro.spring.security.interceptor.AopAllianceAnnotationsAuthorizingMethodInterceptor$1.proceed(AopAllianceAnnotationsAuthorizingMethodInterceptor.java:82)
at org.apache.shiro.authz.aop.AuthorizingMethodInterceptor.invoke(AuthorizingMethodInterceptor.java:39)
at org.apache.shiro.spring.security.interceptor.AopAllianceAnnotationsAuthorizingMethodInterceptor.invoke(AopAllianceAnnotationsAuthorizingMethodInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)

2:dubbo服务相关日志
2017-11-11 07:26:35,994 [main-SendThread(192.168.18.86:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1096) - Client session timed out, have not heard from server in 40281ms for sessionid 0x45f95de2d891148, closing socket connection and attempting reconnect
2017-11-11 07:26:52,703 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:975) - Opening socket connection to server 192.168.18.90/192.168.18.90:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-11 07:27:03,752 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:852) - Socket connection established to 192.168.18.90/192.168.18.90:2181, initiating session
2017-11-11 07:29:21,928 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1096) - Client session timed out, have not heard from server in 149223ms for sessionid 0x45f95de2d891148, closing socket connection and attempting reconnect
2017-11-11 07:29:55,024 [main-SendThread(192.168.18.89:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:975) - Opening socket connection to server 192.168.18.89/192.168.18.89:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-11 07:29:55,025 [main-SendThread(192.168.18.89:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:852) - Socket connection established to 192.168.18.89/192.168.18.89:2181, initiating session
2017-11-11 07:29:55,448 [main-SendThread(192.168.18.89:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1094) - Unable to reconnect to ZooKeeper service, session 0x45f95de2d891148 has expired, closing socket connection
2017-11-11 07:30:31,361 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:975) - Opening socket connection to server 192.168.18.90/192.168.18.90:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-11 07:30:31,362 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:852) - Socket connection established to 192.168.18.90/192.168.18.90:2181, initiating session
2017-11-11 07:30:31,777 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1235) - Session establishment complete on server 192.168.18.90/192.168.18.90:2181, sessionid = 0x35e93bd6a5cbc61, negotiated timeout = 60000
2017-11-11 07:31:41,632 [main-SendThread(192.168.18.90:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:1096) - Client session timed out, have not heard from server in 41620ms for sessionid 0x35e93bd6a5cbc61, closing socket connection and attempting reconnect
2017-11-11 07:31:58,063 [main-SendThread(192.168.18.86:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:975) - Opening socket connection to server 192.168.18.86/192.168.18.86:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-11 07:32:05,662 [main-SendThread(192.168.18.86:2181)] INFO org.apache.zookeeper.ClientCnxn$SendThread (ClientCnxn.java:852) - Socket connection established to 192.168.18.86/192.168.18.86:2181, initiating session

  • 写回答
  • 好问题 提建议
  • 关注问题
  • 收藏
  • 邀请回答

9条回答 默认 最新

  • 如影随形小浩儿 2017-11-17 02:32
    已采纳

    没有遇到过这种问题,看日志很可能是网络超时的原因。

    还有一种可能,调用服务时,连接一直阻塞,然后连接大量增加的情况下,直接让服务宕掉,这个是由于dubbo没有熔断机制的原因造成的
    建议你监测一下你的服务那边的内存情况以及线程情况。

    已采纳该答案
    评论
    解决 无用
    打赏 举报
  • Drizzly_ 2017-11-16 06:10

    RPC调用有问题, 网络连接也超时,确定一下你的网络状态,与你配置的ip与端口信息。及防火墙配置

    评论
    解决 无用
    打赏 举报
  • the_fool_ 2017-11-16 06:29

    网络超时,可能与外部网络有关系。配置文件配置成超时后无限重连

    评论
    解决 无用
    打赏 举报
  • the_fool_ 2017-11-16 06:42

    日志里session认证失败,做个联通测试,监测一天吧

    评论
    解决 无用
    打赏 举报
  • 独家记忆shine 2017-11-17 01:18

    我们模拟了下网络io问题,不断对大数据服务进行压力访问,机器上网络读写达到了 76m/s;然后再不断发送请求到dubbo服务上,
    发现时没有问题的

    评论
    解决 无用
    打赏 举报
  • 独家记忆shine 2017-11-17 01:21

    检测系统我们这边是没有的

    评论
    解决 无用
    打赏 举报
  • TomZXY 2017-11-17 03:34

    你的dubbo服务down掉没有错误日志吗?能具体说清楚一下吗,是进程直dead了吗

    评论
    解决 无用
    打赏 举报
  • Dream.Bird 2018-02-28 03:21

    去年在项目开发中遇到了这个问题 结果是web层跟dataservice层的dubbo版本不一致导致的 当时都快疯掉了

    评论
    解决 无用
    打赏 举报
  • Dream.Bird 2018-02-28 03:22

    就是两边dubbo版本不一样

    评论
    解决 无用
    打赏 举报

相关推荐 更多相似问题