Introducing Hazelcast ... a new way to cluster Openfire!

I haven’t used the MUC components, but it sounds like a glitch with the cached presence information that is shared across the cluster for multi-user chat. Have you noticed any error messages in your log files that might help isolate the issue?

I wonder if your case would work using the older Coherence-based clustering plugin … might be worth a try if you have the time/inclination to set it up. That would also help clarify whether the issue is with the MUC component or the Hazelcast plugin.

Hello

Can I used Connection manager before Openfire with Hazelcast???

Now, I tested it. But it’s failed. Client can’t login.

Thanks

Nick

Not a developer here, just wanting to test this out. Where can I get the hazelcast.jar file from?

I’m on 3.7.2 Beta and looking to put this in place behind Microsoft NLB.

I found the hazelcast folder in svn, and I’m unfamiliar with the process, but do the contents of this folder need to be built to hazelcast.jar? I don’t have the tools/environment to do this, or knowledge, if that is the case.

I tried modifying a separate .jar file by removing its contents and placing in it all items from the hazelcast folder, but the plugin would not load (error messages in log).

Can anyone host or send my way the hazelcast.jar file?

The required jars for the plugin are already in the plugin’s lib directory. As you are not a developer, I would suggest you use a nightly build instead of trying to build it yourself.

Just select the appropriate build for your platform and drill down into the build artifacts.

Thanks for the direction, I am using the method mentioned above to specify nodes instead of multicast.

I’m getting this in the logs:

2012.11.22 11:03:37 org.jivesoftware.openfire.plugin.BroadcastPlugin - Hazelcast Instance is not active!

java.lang.IllegalStateException: Hazelcast Instance is not active!

at com.hazelcast.impl.FactoryImpl.initialChecks(FactoryImpl.java:711)

at com.hazelcast.impl.MProxyImpl.beforeCall(MProxyImpl.java:102)

at com.hazelcast.impl.MProxyImpl.get(MProxyImpl.java:112)

at com.jivesoftware.util.cache.ClusteredCache.get(ClusteredCache.java:86)

at org.jivesoftware.util.cache.CacheWrapper.get(CacheWrapper.java:121)

at org.jivesoftware.openfire.spi.RoutingTableImpl.removeComponentRoute(RoutingTabl eImpl.java:821)

at org.jivesoftware.openfire.component.InternalComponentManager.removeComponent(In ternalComponentManager.java:238)

at org.jivesoftware.openfire.component.InternalComponentManager.removeComponent(In ternalComponentManager.java:211)

at org.jivesoftware.openfire.plugin.BroadcastPlugin.destroyPlugin(BroadcastPlugin. java:120)

at org.jivesoftware.openfire.container.PluginManager.shutdown(PluginManager.java:1 46)

at org.jivesoftware.openfire.XMPPServer.shutdownServer(XMPPServer.java:948)

at org.jivesoftware.openfire.XMPPServer.access$700(XMPPServer.java:145)

at org.jivesoftware.openfire.XMPPServer$ShutdownHookThread.run(XMPPServer.java:896 )

An update to my issue above… I removed all other plugins and this error went away however I was still getting another error, which turned out to simply be a configuration file mis-type!

Now both instances show up as they should. On to testing, awesome!

Hello

My Openfire Version 3.7.1. I fonund this package for 3.7.1. I installed it.

It’s OK. I can login to server.

I tested MUC with two openfire server and cluster package. It’s OK. Can work. no Problems.

But If I added to three server. I get the error message. It’s not work. Server always tell me “No response from server”

It’s my error log.

org.jivesoftware.openfire.IQRouter - Could not route packet

java.lang.ClassCastException: com.hazelcast.impl.MemberImpl cannot be cast to java.lang.Comparable

at java.util.TreeMap.put(Unknown Source)

at java.util.TreeSet.add(Unknown Source)

at com.jivesoftware.util.cache.CoherenceClusteredCacheFactory.doClusterTask(Cohere nceClusteredCacheFactory.java:221)

at org.jivesoftware.util.cache.CacheFactory.doClusterTask(CacheFactory.java:499)

at org.jivesoftware.openfire.component.InternalComponentManager.process(InternalCo mponentManager.java:512)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToComponent(RoutingTableImp l.java:352)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:237)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:324)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:121)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)

at org.jivesoftware.openfire.component.InternalComponentManager.sendPacket(Interna lComponentManager.java:281)

at org.jivesoftware.openfire.plugin.SearchPlugin.processPacket(SearchPlugin.java:2 43)

at org.jivesoftware.openfire.component.InternalComponentManager.checkDiscoSupport( InternalComponentManager.java:458)

at org.jivesoftware.openfire.component.InternalComponentManager.addComponent(Inter nalComponentManager.java:171)

at org.jivesoftware.openfire.plugin.SearchPlugin.initializePlugin(SearchPlugin.jav a:164)

at org.jivesoftware.openfire.container.PluginManager.loadPlugin(PluginManager.java :483)

at org.jivesoftware.openfire.container.PluginManager.access$300(PluginManager.java :80)

at org.jivesoftware.openfire.container.PluginManager$PluginMonitor.run(PluginManag er.java:1067)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)

at java.util.concurrent.FutureTask.runAndReset(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101 (Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodi c(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknow n Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

How to slove this problem??

Thanks for your help.

This issue has been fixed in the latest nightly builds for 3.7.2. Unfortuately, the fix was applied after the custom plugin build (back-ported for 3.7.1) was created. There are three possible solutions:

  1. Update to the latest nightly build for 3.7.2 including this fix (among others)
  2. Limit your cluster size to two nodes with the existing 3.7.1 build
  3. Apply the simple fix to the 3.7.1 code base and rebuild (here’s a link)

Hope that helps!

Hello

Thanks for your reply.

I try re-compile openfire 3.7.1 with clustering.

But I got this error.

I’m not R&D. I don’t know why it. Can you give me a suggestion?? Thanks a lot.

[javac] Compiling 27 source files to /opt/openfire_src/work/plugins-dev/hazelcast/target/classes

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/openfire/sess ion/RemoteSession.java:172: cannot find symbol

[javac] symbol : method getClusterNodeInfo(byte[])

[javac] location: class org.jivesoftware.util.cache.CacheFactory

[javac] ClusterNodeInfo info = CacheFactory.getClusterNodeInfo(nodeID);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/openfire/sess ion/RemoteSession.java:189: cannot find symbol

[javac] symbol : method getClusterNodeInfo(byte[])

[javac] location: class org.jivesoftware.util.cache.CacheFactory

[javac] ClusterNodeInfo info = CacheFactory.getClusterNodeInfo(nodeID);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Ca cheListener.java:67: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] NodeID nodeID = NodeID.getInstance(StringUtils.getBytes(event.getMember().getUuid()));

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:417: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:442: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:475: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:613: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] nodePresences.put(NodeID.getInstance(StringUtils.getBytes(event.getMember().get Uuid())),

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:616: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] ClusterManager.fireJoinedCluster(StringUtils.getBytes(event.getMember().getUuid ()), true);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:623: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterExternalizableUtil.java:43: com.jivesoftware.util.cache.ClusterExternalizableUtil is not abstract and does not override abstract method readSerializableMap(java.io.DataInput,java.util.Map<java.lang.String,? extends java.io.Serializable>,java.lang.ClassLoader) in org.jivesoftware.util.cache.ExternalizableUtilStrategy

[javac] public class ClusterExternalizableUtil implements ExternalizableUtilStrategy {

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:229: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] return StringUtils.getBytes(oldest.getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:238: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] return StringUtils.getBytes(cluster.getLocalMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:253: members is already defined in doClusterTask(org.jivesoftware.util.cache.ClusterTask)

[javac] Set members = new HashSet();

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:286: cannot find symbol

[javac] symbol : method getString(byte[])

[javac] location: class org.jivesoftware.util.StringUtils

[javac] logger.warn(“Requested node " + StringUtils.getString(nodeID) + " not found in cluster”);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:351: cannot find symbol

[javac] symbol : method getString(byte[])

[javac] location: class org.jivesoftware.util.StringUtils

[javac] logger.warn(“Requested node " + StringUtils.getString(nodeID) + " not found in cluster”);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:369: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] if (Arrays.equals(StringUtils.getBytes(member.getUuid()), nodeID)) {

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cluster/ HazelcastClusterNodeInfo.java:48: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] nodeID = NodeID.getInstance(StringUtils.getBytes(member.getUuid()));

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cluster/ HazelcastClusterNodeInfo.java:50: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] seniorMember = ClusterManager.getSeniorClusterMember().equals(StringUtils.getBytes(member.getU uid()));

[javac] ^

[javac] Note: Some input files use or override a deprecated API.

[javac] Note: Recompile with -Xlint:deprecation for details.

[javac] Note: Some input files use unchecked or unsafe operations.

[javac] Note: Recompile with -Xlint:unchecked for details.

[javac] 18 errors

[trycatch] Caught exception: Compile failed; see the compiler error output for details.

[echo] Error building plugin: hazelcast. Exception:

[echo] /opt/openfire_src/build/build.xml:1310: Compile failed; see the compiler error output for details.

Like others, I’m having similar issues with the tcp-ip settings in hazelcast working properly. I have added both of my openfire servers to the config as so:

jabbertest.switchtower.com:5701

jabbertest2.switchtower.com:5701

I’ve verified the server is listening on the 5701 port:

netstat -lan | grep 5701

tcp 0 0 0.0.0.0:5701 0.0.0.0:* LISTEN

Doing a tcpdump between the servers, I see no communication with the servers on that port. There are no iptables rules and the HW firewalls these servers sit behind are configured to allow full IP communciation.

Check the nohup.out file in the log directory (console output) to determine whether your Openfire installation is finding your customized Hazelcast configuration file. You should also be able to see in this log file exactly which servers have joined the cluster.

Hi @Tom Evans.

Good to see you!

I has composing Openfire HA cluster on Amazon AWS using hazelcast cluster plugin.

Unfortunately! Amazon Service do not support direct connection between their regions.

So, I use the hazelcast cluster plugin on openfire. It seemed to be like which did not have a problem.

But, Increasing of hazelcast cluster node has occured big problem.

First, this service is very unstable between cluster node.

Second, this service has occured latency between clients message about 2~3 second. It tested just two clients.

So, I was find replication property on hazelcast document about “wan-replication”. and I was configured your hazelcast configuration xml file.

But, this do not operate. What is problem on my configuration?

like this:

com.hazelcast.impl.wan.WanNoDelayReplication</replication-imp l>

10.0.0.50:5701

com.hazelcast.impl.wan.WanNoDelayReplication</replication-imp l>

10.1.0.50:5701

The issuses of after deployed cluster with Hazelcast Clustering Plugin of openfire .

I have three server nodes : A (10.0.1.113),B (10.0.1.176),C (10.0.1.158).

And all of them share a same Mysql database. Initially, I installed Openfire to B & C server.After that I installed Openfire to A. Such an installation made domain of all servers same as domain of A server.All server domains are 10.0.1.113 now. I used Strophe.js accessing A node successfully via BOSH After deployed cluster with Hazelcast Clustering Plugin. But might be owing to domains of B and C node are same as A’s , lead to I can not access B and C via BOSH with Strophe.js but the clients software like Spark can access all of them and works fine. Accessing B and C via BOSH made errors below:

Now my issue is that I need to access the cluster by using BOSH.Is it enough access only A node or I need to be able to access all nodes via BOSH? If I wanna access all nodes via BOSH in the meantime,how should I deploy cluster of openfire.

Here is configuration of Hazelcast

224.2.2.3

54327

10.0.1.113

10.0.1.176

10.0.1.158

10.0.1.113

I appreciate any help ! I hope you could forgive my awful english and understand my issue description.

What is your domain name? Have you created a DNS record for your domain name pointing to all three IP addresses?

No,I have not created.I merely use an IP among these three nodes such as 10.0.1.113.And I test the cluster in Local Area Nework.

Thanks for reply!

I don’t think it will work. The domain name should be a valid DNS name that resolves via the network to the next IP address by cycling through the 3 IP addresses

1 Like

I think having A records (IPs) in DNS foe each server is necessary as a fallback mechanism. It’s possible (and recommended) to have SRV records instead.

kentzen, maybe this will help you: http://wiki.xmpp.org/web/SRV_Records

1 Like

Yeah.Dele and Alex,Thanks you a lot .It works now.

Here is my steps:

My server is Redhat6 32 bit.

1.Modify all nodes /etc/hosts file .Plus code as following:

127.0.0.1 openfire #Any valid DNS name you want.

2.Change ‘setup’ option to false within ‘/openfire/conf/openfire.xml’,this step means that you could reset configuration for openfire.And type a DNS name you just bound with IP ‘127.0.0.1’ into ‘domain’ option.

3.Check ‘xmpp.domain’ field in ‘ofProperty’ table of openfire database,see if its value was set to ‘admin@openfire’ (or admin@).Change it if it isn’t.

4.Restart openfire.

(note : Do these steps to all nodes,and domain name of each node is the same like ‘openfire’ )

Via steps above .It should work fine! Please contact me if you confuse!

I doubt hardcoding IPs left and right will get you too far, but if it works for now… enjoy.

1 Like

Lol .Thanks.