Openfire + XIFF - compression

Hi there,

This post is aimed primarily at the developers of Openfire and XIFF but if anyone wants to contribute, then the more the merrier! Unfortunately this post is quite verbose, but I have a lot of information to get across. I’m also happy to provide sample code to anyone who needs it (it’s not pretty, but it illustrates that compressed connections are possible).

I have been investigating getting Openfire and XIFF to work together with compression enabled. I read many posts describing how it wasn’t quite working yet but that there was hope for the future. So, I decided to look into exactly why it wasn’t working, and my findings are below.

The first thing to say is that I have it working, pretty much. There are a few hiccups, but I can get a bare bones Flash XIFF client to connect to Openfire with no problems. These hiccups - rosters not working, connections failing intermittently - are stopping me from releasing this work internally, however, and so I am sharing this with the community in order to hopefully progress. The best end result for everyone would be for XIFF and Openfire to work together properly with compression enabled - a lot of people seem to want this!

What I’ve discovered so far:

  1. Actionscript’s built-in ByteArray.compress() and ByteArray.uncompress() don’t seem to do what we need. This was my first avenue of investigation, and I found that a library called as3zlib does the job (pretty much spot on) for us, when using the correct settings. as3zlib is a port of JZlib, which, as you probably know, is the library used for compression within Openfire, so it was a nice surprise to find that it had already been ported to Actionscript. You can find it here:

http://code.google.com/p/as3zlib/

One weird thing: I had to manually modify the last two bytes of the deflated data to FF FF, but apart from this, the binary data produced by as3zlib is identical to JZlib. Perhaps my usage isn’t quite right, but the manual modification works for now. Inflation of data seems to work perfectly (I see some errors but the inflated data is always correct - again, this could be my usage).

  1. Compressing the data correctly in Flash is not enough. Openfire will happily accept the first compressed message, inflate it, and handle it as normal, but successive messages were being ignored. The second message is never inflated properly. However, when I instead swapped the order of the first two messages I was sending to Openfire, it happily handled the message that it was earlier unable to deal with. So I knew it wasn’t the message itself that was causing the problem. Some delving into Openfire, then Mina, then JZlib led me to discover that Mina stores a Zlib deflater and a Zlib inflater in the IoSession, so I wondered whether the inflater was ‘single use’ and was meant to be destroyed after use. I changed the code to do this, and, lo and behold, the second compressed message that was sent to Openfire was now being handled correctly.

However, this threw up a different problem - the second compressed message from Openfire to XIFF was failing - it wasn’t getting compressed properly. So I applied the same tweak to the deflater and it started working.

For reference, here’s my tweak (which I don’t expect to be part of the actual solution!): in Connectionhandler.java, add the following to the bottom of the messageReceived method (please bear in mind that I know this is messy - this is purely illustrative of my theory that there is a strange issue within Openfire or Mina):

      NIOConnection nioConnection = (NIOConnection)session.getAttribute(CONNECTION);

      if(nioConnection.isFlashClient()) {
          Zlib inflater = (Zlib) session.getAttribute(INFLATER);
         
          if(inflater != null) {
              inflater.cleanUp();
          }
          Zlib newInflater = new Zlib(Zlib.COMPRESSION_MAX, Zlib.MODE_INFLATER);
          session.setAttribute(INFLATER, newInflater);
      }

And add the following to the bottom of the messageSent method:

      NIOConnection connection = (NIOConnection)session.getAttribute(CONNECTION);
     
      if(connection.isFlashClient()) {
          Zlib deflater = (Zlib) session.getAttribute(DEFLATER);
         
          if(deflater != null) {
              deflater.cleanUp();
          }
          Zlib newDeflater = new Zlib(Zlib.COMPRESSION_MAX, Zlib.MODE_DEFLATER);
          session.setAttribute(DEFLATER, newDeflater);
      }

Then add (to the top of the same class) the following (copied from CompressionFilter.java in Mina):

   /**
   * A session attribute that stores the {@link Zlib} object used for compression.
   */
  private static final String DEFLATER = CompressionFilter.class.getName() + ".Deflater";
  /**
   * A session attribute that stores the {@link Zlib} object used for decompression.
   */
  private static final String INFLATER = CompressionFilter.class.getName() + ".Inflater";

At this stage I thought I’d solved it, but we’re not quite there yet. I am seeing intermittent problems. Every now and again a message will fail, and my fix - by its nature - will stop working. I don’t know enough about the innards of Openfire and Mina, but I think that someone who does should be able to take my example and see what’s wrong.

Obviously it’s weird that you can connect using compression via Spark and everything works perfectly, so I can only imagine that either I am not doing something in my Flash client, or Openfire treats Flash clients differently (I know it already does treat Flash clients slightly differently, but I don’t believe the inflate/deflate behaviour should be different for Flash clients).

So anyway, there we are. Hopefully there is enough demand and keenness to get this working! I’ll be around, and I’ll be more than happy to elaborate on things, answer any questions or provide source code - just let me know.

Hope someone can help!

Cheers,

Mani

1 Like

A shot in the dark, but do you have a policy server for cross domain browsing answering? Not certain it’s needed or relevant in your case, but typically Flash will not be allowed to connect via socket unless it has this permission. http://www.adobe.com/devnet/flashplayer/articles/socket_policy_files.html

Thanks slicer32, I don’t think this is relevant, though. Without compression turned on, everything is working fine, and even with compression turned on, things work for a short while, so I really don’t think this is relevant…

Cheers,

Mani

Forgot to mention: I’ve been using Openfire 3.6.4 and XIFF 3.0.0.

Mani

Update: we now have this working in XIFF with zero modifications to Openfire or its dependencies. The key info:

You cannot use the built in ByteArray.compress and ByteArray.uncompress. They do seem to create valid zlib compressed data and uncompress it, however they only work standalone. This is the key: the inflater and deflater in Openfire (actually, in Mina) are reused - they’re not single-use. So instead of calling inflateInit(), inflate(), then inflateEnd() every time, the same inflater is used, so inflateInit() is called once at the start, then inflate() is called repeatedly on separate chunks of XML. Basically it just seems to work. What I was trying in Flash always hinged around creating a new inflater/deflater each time XIFF received data, which just doesn’t work after the first packet.

After utilising zlib in XIFF (via as3zlib) and reusing the inflater and deflater, a few tweaks later everything seems to be working perfectly.

Clearly this introduces a new dependency for XIFF - namely as3zlib - however I think it’s worth it.

Hopefully this can be looked at for inclusion into trunk? I am happy to work with you guys to get this up and running on your systems - just let me know.

Cheers,

Mani

Have you used this with the current trunk version of XIFF and would you be able to provide a patch against it?

I would not mind of having as3zlib as dependancy for those who wish to use compression, perhaps the best way would to disable compression by default and explain what to do in order to get it working, which would require them to download as3zlib.

I was searching for info about the current state of this in XIFF – did you perhaps put some patch available somewhere?

Cheers,

Any chance of getting your changes to the trunk?

I would like to take a look and could commit them possibly.

Hey Juga,

I think Mike did commit to trunk ages ago, shortly after we completed the work. Could you take a look? If its not there, we’d love to share what we have.

It may be worth thinking about making this an optional library as it does increase the file size of the Swc a bit, but see what you think.

Keep me posted!

Thanks,

Mani

Sorry, but nothing related in the trunk.

Only the negotiation for enabling the compression which I added about the same time when you started working on this…

Last night I did similar trial with as3zlib and will commit some if it today.

Please post the code you had under work.


Just to clarify, I only work on XIFF.

I’ll take a look on Monday when I get back to the office. We have been using compression in production now for about a year so it’s definitely stable :slight_smile:

Mani

XIFF trunk now contains as far as I got with it.

The uncompressed incoming data is incomplete… Perhaps the example used in

http://code.google.com/p/as3zlib/source/browse/trunk/src/test_deflate_inflate.as

is not complete…