Introducing Hazelcast ... a new way to cluster Openfire!

I can see the both node in the admin console of OF. I can join MUC from other servers but I can’t see anyone, as in my roster: I can see only contacts from my own server.

Maybe all connections goes to one server due to DNS setting ?

I think the “master” don’t share the connections with the other node when I recieved one.

No other plugin except admin. The configure is rather simple too, only 2 multicasted nodes with default configuration. And the cluster can be started correctly, but one or two nodes would hang after compress test.

I also settled a cluster with 4 nodes (all nodes started successfully) but without compressing test, all 4 nodes hanged after one or two days.

Stupid question: are all users in the same group? I mean, I had that once, users didn’t see each other, only to discover they weren’t in each other’s roster to begin with.

I’m not sure to understand.

How user can be in the same group if they are on different servers ?

Well, there’s a single DB regardless. It’s like users on Yahoo messenger: they may be online, but until you add them to your own roster, you can’t see them.

What you need to do is add each user to the other’s roster first.

Well, it’s not necessary. For exemple, I can see online people in MUC if I don’t enabled the clustering plugin.

In my roster, I can see users from google, jabber.org, etc. are online when the plugin is disabled

Ok, so users are in the same group. If I understood correctly, the same users stop seeing each other once you enable clustering and they connect to different servers. I’m out of ideas, this worked fine when I tried it.

Yes, if the plugin is enabled, users of my server can see the presence of the others users in their roster of my server, but not the one of users on other servers.

Edit: There is another issue, PubSub doesn’t work when the plugin is enabled.

I’ve encountered an issue in using MUC which I think could either be a misconfiguration on my part, or possibly a bug?

I’ve got 2 nodes setup with the clustering plugin running on OF 3.7.1, and talking to each other over direct TCP rather than multicast. There’s only 2 users on node 2 so far, with all the other 2000 users connecting to node 1.

In a MUC room, when either of us on node 2 join the room, we only get a partial list of current members of the room. Those of us on node 2 can still see anything that anyone on node 1 says in the room, but until they leave and rejoin, they don’t appear to be in the room. If those of us on node 2 leave/rejoin the room, then the list of current members in the room shrinks back down again.

Does this sound like a configuration issue? Or is it a problem with the members of rooms not being synced across the nodes?

I haven’t used the MUC components, but it sounds like a glitch with the cached presence information that is shared across the cluster for multi-user chat. Have you noticed any error messages in your log files that might help isolate the issue?

I wonder if your case would work using the older Coherence-based clustering plugin … might be worth a try if you have the time/inclination to set it up. That would also help clarify whether the issue is with the MUC component or the Hazelcast plugin.

Hello

Can I used Connection manager before Openfire with Hazelcast???

Now, I tested it. But it’s failed. Client can’t login.

Thanks

Nick

Not a developer here, just wanting to test this out. Where can I get the hazelcast.jar file from?

I’m on 3.7.2 Beta and looking to put this in place behind Microsoft NLB.

I found the hazelcast folder in svn, and I’m unfamiliar with the process, but do the contents of this folder need to be built to hazelcast.jar? I don’t have the tools/environment to do this, or knowledge, if that is the case.

I tried modifying a separate .jar file by removing its contents and placing in it all items from the hazelcast folder, but the plugin would not load (error messages in log).

Can anyone host or send my way the hazelcast.jar file?

The required jars for the plugin are already in the plugin’s lib directory. As you are not a developer, I would suggest you use a nightly build instead of trying to build it yourself.

Just select the appropriate build for your platform and drill down into the build artifacts.

Thanks for the direction, I am using the method mentioned above to specify nodes instead of multicast.

I’m getting this in the logs:

2012.11.22 11:03:37 org.jivesoftware.openfire.plugin.BroadcastPlugin - Hazelcast Instance is not active!

java.lang.IllegalStateException: Hazelcast Instance is not active!

at com.hazelcast.impl.FactoryImpl.initialChecks(FactoryImpl.java:711)

at com.hazelcast.impl.MProxyImpl.beforeCall(MProxyImpl.java:102)

at com.hazelcast.impl.MProxyImpl.get(MProxyImpl.java:112)

at com.jivesoftware.util.cache.ClusteredCache.get(ClusteredCache.java:86)

at org.jivesoftware.util.cache.CacheWrapper.get(CacheWrapper.java:121)

at org.jivesoftware.openfire.spi.RoutingTableImpl.removeComponentRoute(RoutingTabl eImpl.java:821)

at org.jivesoftware.openfire.component.InternalComponentManager.removeComponent(In ternalComponentManager.java:238)

at org.jivesoftware.openfire.component.InternalComponentManager.removeComponent(In ternalComponentManager.java:211)

at org.jivesoftware.openfire.plugin.BroadcastPlugin.destroyPlugin(BroadcastPlugin. java:120)

at org.jivesoftware.openfire.container.PluginManager.shutdown(PluginManager.java:1 46)

at org.jivesoftware.openfire.XMPPServer.shutdownServer(XMPPServer.java:948)

at org.jivesoftware.openfire.XMPPServer.access$700(XMPPServer.java:145)

at org.jivesoftware.openfire.XMPPServer$ShutdownHookThread.run(XMPPServer.java:896 )

An update to my issue above… I removed all other plugins and this error went away however I was still getting another error, which turned out to simply be a configuration file mis-type!

Now both instances show up as they should. On to testing, awesome!

Hello

My Openfire Version 3.7.1. I fonund this package for 3.7.1. I installed it.

It’s OK. I can login to server.

I tested MUC with two openfire server and cluster package. It’s OK. Can work. no Problems.

But If I added to three server. I get the error message. It’s not work. Server always tell me “No response from server”

It’s my error log.

org.jivesoftware.openfire.IQRouter - Could not route packet

java.lang.ClassCastException: com.hazelcast.impl.MemberImpl cannot be cast to java.lang.Comparable

at java.util.TreeMap.put(Unknown Source)

at java.util.TreeSet.add(Unknown Source)

at com.jivesoftware.util.cache.CoherenceClusteredCacheFactory.doClusterTask(Cohere nceClusteredCacheFactory.java:221)

at org.jivesoftware.util.cache.CacheFactory.doClusterTask(CacheFactory.java:499)

at org.jivesoftware.openfire.component.InternalComponentManager.process(InternalCo mponentManager.java:512)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToComponent(RoutingTableImp l.java:352)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:237)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:324)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:121)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)

at org.jivesoftware.openfire.component.InternalComponentManager.sendPacket(Interna lComponentManager.java:281)

at org.jivesoftware.openfire.plugin.SearchPlugin.processPacket(SearchPlugin.java:2 43)

at org.jivesoftware.openfire.component.InternalComponentManager.checkDiscoSupport( InternalComponentManager.java:458)

at org.jivesoftware.openfire.component.InternalComponentManager.addComponent(Inter nalComponentManager.java:171)

at org.jivesoftware.openfire.plugin.SearchPlugin.initializePlugin(SearchPlugin.jav a:164)

at org.jivesoftware.openfire.container.PluginManager.loadPlugin(PluginManager.java :483)

at org.jivesoftware.openfire.container.PluginManager.access$300(PluginManager.java :80)

at org.jivesoftware.openfire.container.PluginManager$PluginMonitor.run(PluginManag er.java:1067)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)

at java.util.concurrent.FutureTask.runAndReset(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101 (Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodi c(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknow n Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

How to slove this problem??

Thanks for your help.

This issue has been fixed in the latest nightly builds for 3.7.2. Unfortuately, the fix was applied after the custom plugin build (back-ported for 3.7.1) was created. There are three possible solutions:

  1. Update to the latest nightly build for 3.7.2 including this fix (among others)
  2. Limit your cluster size to two nodes with the existing 3.7.1 build
  3. Apply the simple fix to the 3.7.1 code base and rebuild (here’s a link)

Hope that helps!

Hello

Thanks for your reply.

I try re-compile openfire 3.7.1 with clustering.

But I got this error.

I’m not R&D. I don’t know why it. Can you give me a suggestion?? Thanks a lot.

[javac] Compiling 27 source files to /opt/openfire_src/work/plugins-dev/hazelcast/target/classes

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/openfire/sess ion/RemoteSession.java:172: cannot find symbol

[javac] symbol : method getClusterNodeInfo(byte[])

[javac] location: class org.jivesoftware.util.cache.CacheFactory

[javac] ClusterNodeInfo info = CacheFactory.getClusterNodeInfo(nodeID);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/openfire/sess ion/RemoteSession.java:189: cannot find symbol

[javac] symbol : method getClusterNodeInfo(byte[])

[javac] location: class org.jivesoftware.util.cache.CacheFactory

[javac] ClusterNodeInfo info = CacheFactory.getClusterNodeInfo(nodeID);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Ca cheListener.java:67: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] NodeID nodeID = NodeID.getInstance(StringUtils.getBytes(event.getMember().getUuid()));

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:417: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:442: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:475: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:613: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] nodePresences.put(NodeID.getInstance(StringUtils.getBytes(event.getMember().get Uuid())),

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:616: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] ClusterManager.fireJoinedCluster(StringUtils.getBytes(event.getMember().getUuid ()), true);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:623: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterExternalizableUtil.java:43: com.jivesoftware.util.cache.ClusterExternalizableUtil is not abstract and does not override abstract method readSerializableMap(java.io.DataInput,java.util.Map<java.lang.String,? extends java.io.Serializable>,java.lang.ClassLoader) in org.jivesoftware.util.cache.ExternalizableUtilStrategy

[javac] public class ClusterExternalizableUtil implements ExternalizableUtilStrategy {

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:229: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] return StringUtils.getBytes(oldest.getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:238: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] return StringUtils.getBytes(cluster.getLocalMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:253: members is already defined in doClusterTask(org.jivesoftware.util.cache.ClusterTask)

[javac] Set members = new HashSet();

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:286: cannot find symbol

[javac] symbol : method getString(byte[])

[javac] location: class org.jivesoftware.util.StringUtils

[javac] logger.warn(“Requested node " + StringUtils.getString(nodeID) + " not found in cluster”);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:351: cannot find symbol

[javac] symbol : method getString(byte[])

[javac] location: class org.jivesoftware.util.StringUtils

[javac] logger.warn(“Requested node " + StringUtils.getString(nodeID) + " not found in cluster”);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:369: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] if (Arrays.equals(StringUtils.getBytes(member.getUuid()), nodeID)) {

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cluster/ HazelcastClusterNodeInfo.java:48: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] nodeID = NodeID.getInstance(StringUtils.getBytes(member.getUuid()));

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cluster/ HazelcastClusterNodeInfo.java:50: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] seniorMember = ClusterManager.getSeniorClusterMember().equals(StringUtils.getBytes(member.getU uid()));

[javac] ^

[javac] Note: Some input files use or override a deprecated API.

[javac] Note: Recompile with -Xlint:deprecation for details.

[javac] Note: Some input files use unchecked or unsafe operations.

[javac] Note: Recompile with -Xlint:unchecked for details.

[javac] 18 errors

[trycatch] Caught exception: Compile failed; see the compiler error output for details.

[echo] Error building plugin: hazelcast. Exception:

[echo] /opt/openfire_src/build/build.xml:1310: Compile failed; see the compiler error output for details.

Like others, I’m having similar issues with the tcp-ip settings in hazelcast working properly. I have added both of my openfire servers to the config as so:

jabbertest.switchtower.com:5701

jabbertest2.switchtower.com:5701

I’ve verified the server is listening on the 5701 port:

netstat -lan | grep 5701

tcp 0 0 0.0.0.0:5701 0.0.0.0:* LISTEN

Doing a tcpdump between the servers, I see no communication with the servers on that port. There are no iptables rules and the HW firewalls these servers sit behind are configured to allow full IP communciation.

Check the nohup.out file in the log directory (console output) to determine whether your Openfire installation is finding your customized Hazelcast configuration file. You should also be able to see in this log file exactly which servers have joined the cluster.