ActiveMQ线上故障

  • A+
所属分类:Linux基础

为了保证生产环境安全稳定运行,对中间件部署集群,从而达到高可用和负载均衡。业务用的activemq消息队列,版本使用的是最新稳定版5.13,配合zookeeper和Dubbo对外提供服务,集群方式采用zookeeper+activemq。一天业务监控收到告警,不能正常提供服务。经过排查,发现activemq有大量报警信息。

2016-03-09 21:10:16,329 | WARN | [/activemq/leveldb-stores/00000000157] 
[ZooKeeperTreeTracker@4e51a33] \
[Thread[main-EventThread,5,main]] Error in treeWatcher (ignored) | 
org.linkedin.zookeeper.tracker.ZooKeeperTreeTracker | main-EventThread
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
= ConnectionLoss for \
/activemq/leveldb-stores/00000000157at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)[zookeeper-3.4.5.jar:3.4.5-1392090]
at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)[zookeeper-3.4.5.jar:3.4.5-1392090]
at 
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)[zookeeper-3.4.5.jar:3.4.5-1392090]
at 
org.linkedin.zookeeper.client.ZooKeeperImpl.getData(ZooKeeperImpl.java:191)[org.linkedin.zookeeper-impl-1.4.0.jar:]
at 
org.linkedin.zookeeper.client.AbstractZooKeeper.getData(AbstractZooKeeper.java:213)[org.linkedin.zookeeper-impl-1.4.0.jar:]
at 
org.linkedin.zookeeper.client.AbstractZKClient.getZKByteData(AbstractZKClient.java:213)[org.linkedin.zookeeper-impl-

谷歌查找原因,看到很多人在就在讨论这个问题,最后在官网看到别人递交的bug。说是预计在5.14会修复,但是这bug提了很长时间了,目前最新版执行5.13,显然没办法升级版本。更换其他集群方式,通过多实例部署,程序解决负载均衡。

https://issues.apache.org/jira/browse/AMQ-5082

YaLei

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: