Introducing Hazelcast ... a new way to cluster Openfire!

The performance tests are a great idea. I actually have a small testing project I was using when I initially refactored pubsub. I was mainly aiming at the memory usage, but it will also work quite well for testing throughput as well.

Even without testing though, I can’t imagine that caching will make any difference in a use case where there are only a couple of items published per second, or less. Now when we get into 10s or 100s per second, that is a completely different story. In any case, some metrics would be valuable, I will have to see what I can do.

The error case should be easy enough, as the db call becomes part of the flow of processing the publish anyway, thus it’s failure would bubble up and can be reported as an **internal-server-error **to the publisher. The db call is already part of the processing chain if the item being published actually triggers a flush if it exceeds the cache limit.

Without the cache, the timer would not be required, so it would simply not be turned on. With a cache, having an extremely short timer would cause the same problem as having no cache, overly frequent db access. In a constantly busy system, the timer is actually not needed as we would be constantly flushing due to the size restriction. I think the only use case where the timer would be needed is when we have bursts of traffic, where the burst would tirgger flushing due to size, but when the burst is over and there are still items in the cache, then the timer would take care of persisting them (and any subsequent low volume publishing between bursts). This is assuming that no cache is the best option in a low volume system.

I have attached to this post a slighlty modified version of hazelcast that is backwards compatible with Openfire 3.6.4 and Openfire 3.7.1. I had to rename the plugin and some classes.

Unzip and copy the clustering.jar file to your plugins folder.
hazelcast-364-1.0.0.zip (1680594 Bytes)

I’ve just taken the plugin for a spin and it seems to be in good working condition. Tried session replication, presence changes, MUC, pubsub, all worked across nodes.

Now, the question is, is the version of the plugin that works with Openfire 3.7.1 available from an official source?

What qualifies as an official source?

It is the same source code. I just renamed the plugin and some classes to appease Opennfire 3.7.1

@@Dele Olajide ; @@Alex : Please tell me what did you do to make this plugin work. I installed it on my 2 Openfire servers and it doesn’t show other cluster members than current host. Thanks in advance !

The default cluster configuration uses multicast (UDP) to discover member nodes, which can be problematic in certain deployments. If you are unable to use multicast due to your network configuration, you can configure the unicast (TCP) settings as documented in the Hazelcast plugin’s readme file:


*The Hazelcast plugin uses the XML configuration builder to initialize the cluster from the XML configuration file (hazelcast-cache-config.xml). By default the cluster members will attempt to discover each other via multicast at the following location: *

  • IP Address: 224.2.2.3
  • Port: 54327
  • Note that these values can be overridden in the plugin’s /classes/hazelcast-cache-config.xml file (via the multicast-group and multicast-port elements). Many other initialization and discovery options exist, as documented in the Hazelcast configuration docs noted above. For example, to set up a two-node cluster using well-known DNS name/port values, try the following alternative: *
*...
<join>
   <multicast enabled="false"/>
   <tcp-ip enabled="true">
     <hostname>of-node-a.example.com:5701</hostname>
     <hostname>of-node-b.example.com:5701</hostname>
   </tcp-ip>
   <aws enabled="false"/>
</join>
...
*

Hope that helps!

2 Likes

Thank you very much for your reply ! I did the configuration you suggested in *hazelcast-cache-config.xml *file on both servers. Unfortunately it still doesn’t work for me. This is weird tough because there is no firewall in between and I can connect using telnet to port tcp 5701 and ping hostnames from both servers.

L.E.: finnaly got the plugin working by enabling multicast in local network.

I was thinking about something we could check out and integrate into a build process. I have added the JAR for now, hopefully we won’t have to look at the source code…

I have a problem with that: the configuration file must be inside the plugin JAR file. Thus, when nodes are added, removed, we have to edit the file inside the JAR. Openfire exposes a property to specify a different configuration file, but this seems to be useless, because the plugin class loader will not look outside the JAR in the first place.

Am I missing something here? Is it possible to use a file outside the JAR?

I installed the plugin this morning for my domain. When I enable the plugin, users from my server don’t see online contacts of the other servers. Besides, they can’t join MUC from other servers. Finaly, PubSub doesn’t work, we can’t fetch our network to see the messages but we can see publication from other servers.

I am wondering how the traffic is distribued. For now, all connections go to one server, the first I installed, never to the second one.

It costs more than two times more memory when I enable the plugin.

I was in a similar situation, but was able to fix it by doing two things:

  1. enable clustering from OF console (it’s disabled by default, even if you have the plugin installed)

  2. make sure all servers are using the same DB (cluster), as some session info is saved there (e.g. offline status)

There are a few options for configuring Hazelcast:

  1. The default configuration for the Hazelcast plugin allows cluster members to join and leave the cluster dynamically, using multicast messages (UDP) to announce these membership changes. However, as described above this approach may no be ideal for every deployment.
  2. After the Hazelcast plugin is installed, a custom configuration file can be copied into the plugin’s /classes/ directory. Note that the “hazelcast.config.xml.filename” system property should be set to the name of this file.
  3. The Hazelcast clustering plugin uses a custom class loader that searches the combined classpaths of all installed plugins for classes and other resources. As such, a simple plugin that includes a custom cluster configuration could be deployed. This is in fact the approach we use in our environment. Our plugin further customizes its plugin class loader to add an external directory to the classpath where we manage our various configurations (e.g. test vs. prod).

Hope that helps!

I tried #2, but the folder is erased and recreated on each restart. Copying a configuration file inside the JAR isn’t very promising either. I should probably take a closer look at #3, but it still looks like the configuration file must be placed inside a JAR (or not, if the plugin can add a folder to the classpath). Maybe it makes more sense to patch Openfire to include the conf folder to the plugins classpaths? It would certainly be much cleaner than placing configuration files inside archives.

Edit: It seems Hazelcast already offers an alternative: set the configuration file by setting the hazelcast.config system property. Unfortunately, this is bypassed by the clustering plugin, which goes directly to ClasspathXmlConfig.

The plugin is enable on both OF nodes and they use the same DB : the list of users is the same on both servers.

Other ideas ?

Edit: it’s better without French in the message ^^

Not sure what ‘plugin is enabled’ means, but I was talking about going into OF’s console (the one that runs on ports 9094/9095) and making sure clustering is enabled. Simply copying the JAR under plugins folder won’t do it.

Ne t’inquiete pas, je compred un peu

I meant I uploaded the jar file then enabled the clustering in the tab “Clustering” (By default, OF admin panel runs on ports 9090 and 9091).

Check nohup.out. You should find something like this:

INFO: [10.46.37.118]:5701 [OF37-cluster]

Members [1] {

Member [AA.BB.CC.DD]:5701 this

}

If nodes can see each other, there should be more than one member in the group.

I don’t find this file.

Strange, if you start openfire with service openfire start, this gets created under /opt/openfire/logs

Alternatively, the info we’re looking for should be on stdout. Check whether you have this redirected.

I installed the .deb provided by the OF team.

I’ve got the logs in /var/log/openfire : debug.info, erro.log, info.log and warn.log. But none of this files have an entry like that.

Where is stdout ?