Message loss while timeout

When a client loses the connection to the server without properly closing it, for example because of a typical cyclic DSL-disconnect or if the network-cable is pulled out, the server doesn’t seem to recognize this and every message sent to this client is simply dropped until the client’s timeout period is over and the status of the client is automatically set to offline.

I’m experiencing this problem at least since Openfire 3.3, but I guess it was always the way it is now. There are comments on the board dating back to 2005 about this. In Openfire 3.7 on Debian Lenny (x86_64) with Sun Java JRE 1.6.0_22, there is still the same problem. I think this is a horrible and often hidden bug and an unacceptable experience for users to drop their messages without any notice.

Please fix this soon!

I think what you ask is a protocol support for XEP-0184: Message Delivery Receipts. Plain XMPP does not support this out-of the box. Both, the client and server must support this XEP.

We would also like to see this in a future version of OF (and also Smack API).

No, not exactly. I want a bug to be fixed that loses messages. If XEP-0184 notifications would be implemented the bug would have gotten more attention. I’m simply asking for the server to recognize if messages sent to a client via TCP aren’t delivered and in that case, disconnect the client and store the messages as offline messages, which are delivered if the client reconnects. Currently the server doesn’t check this and seems to drop every message as delivered which was pushed into the TCP stream.

I tend to agree with you.

aef wrote:

… to recognize if messages sent to a client via TCP aren’t delivered …

This might be possible if using TCP/IP ACK or the TCP sequence number to identify each byte of data that is sent. This is currently not done in OF and also not XMPP community-wide. Most (if not all) messages are delivered in asynchronous mode (see public void deliverRawText(String text) in NIOConnection.java).

I notice that XMPP Ping is now implemented.

As I said earlier I agree with you. After checking the OF 3.7.1 Open Issues, I see that many bugs still exist!

The next release of 3.7.1 should look at bug fixing and not at adding new features! The following bugs may be related to your problem:

  • Openfire shouldn’t close idle client connections unconditionally
  • xmpp.client.idle should disconnect an idle client
  • Investigate xmpp ping behavior
  • ldap.connect.timeout not working with SSL connection
  • Contacts are seeing offline when their server is down
  • review and fix outgoing connection
  • Last logouts are not recorded when server is shut down
  • Insconsistent session status when Openfire goes back online after an offline period while Clearspace is offline
  • MINA’s SessionCreated or SessionOpened?

aef wrote:

No, not exactly. I want a bug to be fixed that loses messages. If XEP-0184 notifications would be implemented the bug would have gotten more attention. I’m simply asking for the server to recognize if messages sent to a client via TCP aren’t delivered and in that case, disconnect the client and store the messages as offline messages, which are delivered if the client reconnects. Currently the server doesn’t check this and seems to drop every message as delivered which was pushed into the TCP stream.

To me XEP-0184 looks exactly what you are asking. Because application layer xmpp can’t check TCP status. It needs some mechanism to check if message was delivered, so XEP-0184 looks fitting. XMPP Ping probably is not enough in this case. I have filed this, but can’t say when and who can implement this:

OF-434

SPARK-1238

Your request is valid, however we should note that OF 3.7.0 was pretty much an effort lead and done in major parts by Guus alone. He is most helpful to new developmers. If you are providing development ressources, the project would move ahead. Doing so would greatly help the overall project. To my experience, it takes a week to have a seasoned developer at speed with OF/Spark development.

We (aef and I) are looking at using OF for a production environment. When I look at the OF 3.7.1 requirements list I am afraid that it is not aiming for a short release cycle, i.e. bug fixing. Openfire is a great software but some bigger XMPP projects are looking for alternatives to OF like OSW (see http://groups.google.com/group/onesocialweb/msg/44b170b3c218824d).

In my opinion there should be a OF 3.7.1 maintenance release that just looks at blockers and major bugs. Client disconnects seems to be an issue with OF 3.7.0 as multiple community messages indicate.

What is the long term vision in terms of the OF project planning, roadmap and future versions (major.minor[.maintenance[.build]])?

A roadmap requires a somewhat stable and continously working community. Guus is doing large parts of the technical work. The more people are actually writing code, the better the roadmap gets.

If OF is a great software in you opinion: Make it even better by writing code and participate in the discussion (for example by drafting a road map).

Walter, Guus,

You are right, but I cannot write stable code. I can help with testing. Ideally would be an overall Igniterealtime roadmap that includes Openfire, Spark, Tinder, Smack API and XIFF API.

Below is my proposal for OF 3.7.1. The idea is stolen from the ignite realtime web site:

The next version aims to make Openfire more stable. Future versions will look at requirements of the XSF roadmap which has identified several high-priority initiatives which include:

  • Make XMPP file transfer more reliable by transitioning to Jingle (see XEP-0234) with SOCKS5 bytestreams (XEP-0260) and in-band bytestreams (XEP-0261).
  • Revise the specification for Multi-User Chat, incorporating the results of implementation experience, interoperability testing, and input from the XSF’s technical review team.
  • Combine Multi-User Chat with Jingle to enable multi-user media session management.
  • Define a method for distributing multi-user chatrooms across multiple servers.
  • Transition from the vcard-temp format to the vCard4 standard developed at the IETF (seeXEP-0292).
  • Work with the microblogging community and other social networking developers to define consistent methods for integrating “social” and “mobile” features into XMPP, based onPubSub and Personal Eventing Protocol.
  • Solidify XMPP extensions for whiteboarding and collaborative document editing based on the Shared XML Editing specification.
  • Continue to improve the resistance of XMPP to spam, phishing, abuse, and denial of service attacks.

Version 3.7.1, Q3 2011 - Q4 2011

Future

  • OF-271: fix “duplicate key violates unique constraint “ofpresence_pk””
  • OF-270: Duplicate entry ‘user xxx’ for key 1 – Exposed during load testing
  • OF-341: Openfire shouldn’t close idle client connections unconditionally
  • OF-439: Memory leak in PEP service
  • OF-362: After login, not all contacts that are online appear online to the logged in user
  • OF-142: Change caching strategy for shared groups
  • OF-141: rosters are lost
  • OF-122: Massive errors in rooms permissions and options
  • OF-104: Color in groupchat text can hang Openfire
  • OF-102: Deleting user does not clear out ofUserFlag
  • OF-97: Openfire should not allow illegal users to be added to a user’s roster
  • OF-92: UTF-8 message decoding within XMLLightweightParser
  • OF-435: Openfire doesn’t return disco information about the MUC rooms
  • OF-415: Group disappears from the Group Summary view after editing its details
  • OF-410: BOSH too picky about content-type with requests
  • OF-405: Openfire fails to verify chained certificates
  • OF-398: MacOSX installer fails
  • OF-396: RPM install overwrites modified /etc/sysconfig/openfire
  • OF-392: pubsub publish - fix potential loop when an error condition occurs
  • OF-378: Fastpath Service is not showing up in Admin Console
  • OF-376: Illegal JIDs prevent MUC service to start
  • OF-356: Newlines in the stream tag are not handled properly
  • OF-338: The property xmpp.pubsub.multiple-subscriptions does not appear to work when set to false
  • OF-335: Pidgin disconnects while accessing Openfire using BOSH(http-bind)
  • OF-331: CA certificates are incorrectly detected as self-signed certificates
  • OF-329: Using Privacy List fail (at least in Pidgin)
  • OF-310: EntityCapabilityManager should not assume that every client supports version 1.4 of XEP-0115
  • OF-308: EntityCapabilityManager requeries clients even though they are known to return an error
  • OF-305: unicode passwords block successful logins
  • OF-283: NPE with http-bind
  • OF-275: Remove jre-dist.tar.gz from source
  • OF-251: Clicking stop and start immediately locks the embedded database
  • OF-248: Admin console listeners cannot be disabled from the admin console
  • OF-234: The user filters and group filters is limited to 256 characters
  • OF-232: On startup, conference room list on admin console sometimes blank
  • OF-201: Bad JID with /40 in it not parsed well
  • OF-198: RosterEventListener events triggering in an odd fashion
  • OF-193: Last logouts are not recorded when server is shut down
  • OF-181: NPE when admin deletes logged in owner from members-only room
  • OF-161: xmpp.client.idle should disconnect an idle client
  • OF-153: mysql Ver 14.7 Distrib 4.1.22 upgrade errors for OF3.4.5 to 3.6.0a
  • OF-148: Arabic characters in username doesn’t work
  • OF-119: Ldap issue (search filter and ‘@’ encoding) [patch]
  • OF-118: Handle unregistration of clearspace-conference when chat plugin not installed
  • OF-117: Email test page doesn’t work with LDAP
  • OF-114: Clearing cache can lock up MUC
  • OF-113: Openfires admin gui allows to enter long group names while the database supports only 50 characters
  • OF-111: Inconsistent session status when Openfire goes back online after an offline period while Clearspace is offline
  • OF-110: Huge list of offline messages may consume all server memory
  • OF-107: LdapProvider, Manager, and LdapVCard Provider now escape and unescape usernames properly
  • OF-105: fix duplicate roster items in database
  • OF-101: escapeHTMLTags needed for webchat
  • OF-100: PEP subscriptions are not created for shared group members
  • OF-98: Bug in xmlns for grantowner and grantadmin / not in sync with JEP-0045
  • OF-96: Ability to delete a Clearspace user via Openfire although this feature is not present in Clearspace 2.0.5
  • OF-95: Plugin removal should be case insensitive
  • OF-94: NPE with system-emailtest.jsp
  • OF-93: fix custom database user integration bug
  • OF-51: Update dwr to 1.1.4
  • OF-345: MINA’s SessionCreated or SessionOpened?
  • OF-238: Improve performance when loading huge number of rooms history
  • OF-115: fix “the s2s out of order issue” (MUC messages in wrong)
  • OF-274: improve DNS cache
  • OF-306: review and fix outgoing connection
  • OF-134: Update Jetty to latest version
  • File sharing in rooms
  • Multi-domains support
  • vCard4 XEP-292 support
  • Jingle Bytestream XEP-260 and 261 support

*Plan as of March 11, 2011. Subject to change; we make no promises on future releases. :slight_smile:

Your vote counts! We prioritize work on new features and improvements based on community feedback. The top ten issues (by votes) are listed below. You can also view the fulllist.

To cast your own votes, visit the issue tracker, login (or create an account), then click to vote for an issue while viewing the issue report.

Issue

Votes

OF-168: Add support for nested groups from LDAP

69

OF-142: Change caching strategy for shared groups

56

OF-299: Add support for removing inactive accounts

30

OF-302: Automatically share LDAP groups

26

OF-257: Send existing/new password of a user to his email from the admin console

25

OF-164: Add support for not showing a shared group to its group members

22

OF-151: Individual/Group Broadcast

16

OF-174: Add some improvements to the “User Permissions” page

13

OF-115: fix “the s2s out of order issue” (MUC messages in wrong)

13

OF-250: Allow to configure the groups of a user from the user profile

12

OF-16: password of admin is not encrypted

11

OF-141: rosters are lost

10

A reply from Hung Pham on a ticket SMACK-331:

I don’t think XEP-0184 completely solve the issue, at least not solving it if i set offline message policy to “Always Store”. Provide this scenario:

I have 2 users chatting, call them A and B

A sends B a message while B’s connection just got lost!

The message got dropped

But A does not know that the message got dropped, it’ll just assume that the message got delivered to the server and server will eventually send it to B

In that case: B will lose the message forever

To fix this, I can manually check for every message after a few minutes for: B’s presence / message delivery receipt? But I think it’s a bit overkill

Can’t we simply do: if message failed to deliver and the offline message policy is “Always Store”, then we store it to “ofoffline” for delayed delivery?

In reply for Hung. I agree with Florian and don’t see the case here. Example:

A sends a message to B with a message receipt packet (given his client already supports this by using Smack implementation of XEP-0184 or some other library)

Message is delivered to Openfire first and it prepares to push it to user B

User B temporary loses connection

Openfire don’t know yet that user B is unavailable, it still “sees” user B as online and sends the message

Message is lost in the void

User B gets back online but doesn’t receive anything

User’s A client still haven’t received deluvery receipt from user’s B client and shows “message not delivered” status (this is up to the client developer how to desing this, maybe show “delivered”, “not delivered” near every message or some other way).

User A waits for a few minutes (depends on a user ) and tries again and this time user’s B client sends a receipt and User A sees the status “delivered”.

Offline storage is not involved in this scenario as Openfire has no way to notice a user connection lost immediately. Usually it can take minutes even. You can control this by setting xmpp.idle.client in the system properties and set it to as low as you wish (value in miliseconds).

I have a requirement to show Whatsapp like tick marks.

  1. One tick to show that the message has reached the server. HOW WOULD I GET TO KNOW THIS?

  2. Two ticks when the message reaches the other end. WHICH IS BETTER. XEP-0184 or SENDING A TEXT ACKNOWLEDGEMENT WITH MESSAGE ID?

Could someone help me out in determining the response for the sent messages. I would basically need to know when the message reaches the server and also when it reaches the client. Help would be appreciated.