As i’ve said it should be in Raw Packets Received. But there should be actual message text in the packets. If it is not there, then the message is not actually received by Spark.
Ok, I’m back on Spark and I’ll post back once it happens again.
To start the debugger, select “advanced” on the login screen (where you fill out your username/password), then make sure that “start debugger on startup” is checked. It will give you a new screen with lots of debugging information after you log in.
You’ve already mentioned that the message shows up in the message archive on the Openfire side, but it’s unclear to me if you looked at the Openfire log files. Perhaps a warning or error is logged there, that indicates why a packet is not processed properly.
The easiest way to look at the Openfire logs is by looking at the file “all.log” in the directory “logs” of the installation folder of Openfire. You can also view the logs via the Openfire admin console, but for a reason that I never quite understood, the log files there is separated in four distinct tabs (each for a different severity).
Here is everything logged by Openfire around the time of one of these lost/missed messages:
2016.08.30 10:28:41 WARN [pool-29-thread-4]: org.jivesoftware.openfire.net.SocketUtil - Unable to create a socket connection to XMPP domain 'Domain' using remote host: Domain:5269. Cause: Connection refused: connect (a full stacktrace is logged on debug level)
2016.08.30 10:28:41 WARN [pool-29-thread-4]: org.jivesoftware.openfire.net.SocketUtil - Unable to create a socket connection to XMPP domain 'Domain': Unable to connect to any of its remote hosts.
2016.08.30 10:28:41 INFO [pool-29-thread-4]: org.jivesoftware.openfire.session.LocalOutgoingServerSession[Create outgoing session for: OpenFireServer to Domain] - Unable to create new session: Cannot create a plain socket connection with any applicable remote host.
2016.08.30 10:28:41 WARN [pool-29-thread-4]: org.jivesoftware.openfire.session.LocalOutgoingServerSession[Authenticate local domain: 'OpenFireServer' to remote domain: 'Domain'] - Unable to authenticate: Fail to create new session.
2016.08.30 10:30:48 WARN [pool-29-thread-5]: org.jivesoftware.openfire.net.SocketUtil - Unable to create a socket connection to XMPP domain 'Domain' using remote host: Domain:5269. Cause: Connection refused: connect (a full stacktrace is logged on debug level)
2016.08.30 10:30:48 WARN [pool-29-thread-5]: org.jivesoftware.openfire.net.SocketUtil - Unable to create a socket connection to XMPP domain 'Domain': Unable to connect to any of its remote hosts.
2016.08.30 10:30:48 INFO [pool-29-thread-5]: org.jivesoftware.openfire.session.LocalOutgoingServerSession[Create outgoing session for: OpenFireServer to Domain] - Unable to create new session: Cannot create a plain socket connection with any applicable remote host.
2016.08.30 10:30:48 WARN [pool-29-thread-5]: org.jivesoftware.openfire.session.LocalOutgoingServerSession[Authenticate local domain: 'OpenFireServer' to remote domain: 'Domain'] - Unable to authenticate: Fail to create new session.
2016.08.30 10:33:49 WARN [pool-29-thread-1]: org.jivesoftware.openfire.net.SocketUtil - Unable to create a socket connection to XMPP domain 'Domain' using remote host: Domain:5269. Cause: Connection refused: connect (a full stacktrace is logged on debug level)
2016.08.30 10:33:49 WARN [pool-29-thread-1]: org.jivesoftware.openfire.net.SocketUtil - Unable to create a socket connection to XMPP domain 'Domain': Unable to connect to any of its remote hosts.
2016.08.30 10:33:49 INFO [pool-29-thread-1]: org.jivesoftware.openfire.session.LocalOutgoingServerSession[Create outgoing session for: OpenFireServer to Domain] - Unable to create new session: Cannot create a plain socket connection with any applicable remote host.
2016.08.30 10:33:49 WARN [pool-29-thread-1]: org.jivesoftware.openfire.session.LocalOutgoingServerSession[Authenticate local domain: 'OpenFireServer' to remote domain: 'Domain'] - Unable to authenticate: Fail to create new session.
Guus, correct me if I’m wrong, but that looks like a DNS and S2S issue. my guess is the client is using a cname or something as “server” address, instead of the xmpp domain, which is prob causing the issue
Here are all of the raw packets received today when I received a message from a user, Jill, that didn’t display although the chat window opened and flashed. You can see in the data below that the message itself did come in to spark, it just wasn’t displayed.
<message to="gretchen@OpenfireServer" id="8h9tm-613" type="chat" from="jill@OpenfireServer/Spark 2.6.3"><thread>eSYGso</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="8h9tm-614" type="chat" from="jill@OpenfireServer/Spark 2.6.3"><thread>eSYGso</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="8h9tm-615" type="chat" from="jill@OpenfireServer/Spark 2.6.3"><thread>eSYGso</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="8h9tm-616" type="chat" from="jill@OpenfireServer/Spark 2.6.3"><thread>eSYGso</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="8h9tm-617" type="chat" from="jill@OpenfireServer/Spark 2.6.3"><body>i am having some printer issues - could you pls take a look?</body><thread>eSYGso</thread><x xmlns="jabber:x:event"><offline/><composing/></x><active xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="8h9tm-619" type="chat" from="jill@OpenfireServer/Spark 2.6.3"><thread>eSYGso</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="8h9tm-620" type="chat" from="jill@OpenfireServer/Spark 2.6.3"><thread>eSYGso</thread><gone xmlns="http://jabber.org/protocol/chatstates"/></message>
<iq type="get" id="89-27224" from="OpenfireServer" to="gretchen@OpenfireServer/Spark 2.6.3"><ping xmlns="urn:xmpp:ping"/></iq>
Here’s another where the message didn’t show in my Spark window, but it opened and flashed.
<iq to="gretchen@OpenfireServer/Spark 2.6.3" id="jy7qe-24829" type="get" from="glenda@OpenfireServer/Spark 2.6.3"><query xmlns="jabber:iq:last"/></iq>
<message to="gretchen@OpenfireServer" id="jy7qe-24831" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<presence id="n5D6o-1128" from="rob@OpenfireServer/Spark 2.6.3" to="gretchen@OpenfireServer"><status>Away</status><priority>0</priority><show>away</show><c xmlns="http://jabber.org/protocol/caps" hash="sha-1" node="http://www.igniterealtime.org/projects/smack" ver="TJuVIXqTCVfJSthaPu4MtTbaf9A="/></presence>
<message to="gretchen@OpenfireServer" id="jy7qe-24833" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24834" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24835" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24836" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24837" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24838" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24839" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24840" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24841" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24842" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24843" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24844" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenfireServer" id="jy7qe-24845" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><body>Hi Gretchen, i am working at home, i laserfiche kicked me off yesterday and now today again. Do you know if we are having problems?</body><thread>Ydl73u</thread><x xmlns="jabber:x:event"><offline/><composing/></x><active xmlns="http://jabber.org/protocol/chatstates"/></message>
<presence id="qXk1H-10" from="crystal@OpenfireServer/Spark 2.6.3" to="gretchen@OpenfireServer"><status>Online</status><priority>1</priority><c xmlns="http://jabber.org/protocol/caps" hash="sha-1" node="http://www.igniterealtime.org/projects/smack" ver="TJuVIXqTCVfJSthaPu4MtTbaf9A="/></presence>
<message to="gretchen@OpenfireServer" id="jy7qe-24848" type="chat" from="glenda@OpenfireServer/Spark 2.6.3"><thread>Ydl73u</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
Although I doubt that it relates to the issue at hand, each of these log messages are all relate to pretty much the same thing: Your Openfire (which is running the XMPP domain “OpenFireServer”), is trying, and failing, to create a connection to another server (might be yours, might be someone else’s, might not exist at all) that is running the XMPP domain “Domain”.
A common cause for your Openfire to try and connect to another server is to allow users from both networks to talk to each-other. Assuming that your username is “gretchen” (which would make your JID “gretchen@OpenFireServer”), it is very possible that you want to chat with your friend named “oliver” that’s on another XMPP network, named “Domain” (Oliver’s JID would be “oliver@Domain”). You could have done so by adding “oliver@Domain” to your contact list, after which your domain will attempt to make contact with Oliver’s domain. These attempts are what we see failing in the log file that you provided.
Note that it is possible that you tried to add a contact to your contact list, but used an incorrect JID to do so. For instance, Oliver could have an account on the same XMPP domain as yours (eg: “oliver@OpenFireServer”), but you accidentally used “oliver@Domain” instead. Even if the XMPP domain “Domain” does not exist, or is currently not reachable, your domain will attempt to contact it.
The names of both XMPP domains are suspicious: “OpenFireServer” and “Domain” look odd. Did you search/replace these values before you posted the log files here? If not, your setup might need to be looked at - although you can probably make things work with these names, odd values like this can lead to confusion, and thus, problems.
Ah, this is very helpful. This confirms that it is indeed an issue in Spark, not Openfire.
I was somewhat able to reproduce the problem. Spark is very extensible in nature. All kinds of plugins can add functionality. I’ve found that Spark would stop processing messages completely, if one of the ‘add-ons’ would throw an unexpected problem. I’ve addressed this issue here: https://issues.igniterealtime.org/browse/SPARK-1798 and provided a fix here: SPARK-1798: Gracefully handle exceptions from event listeners. by guusdk · Pull Request #215 · igniterealtime/Spark · Gi…
With the fix for this issue, the root cause of the problem won’t be fixed (I don’t know what it is), but the effect would now be isolated to that ‘add-on’. It would no longer prevent further processing of the message.
There’s a good chance that this will ‘fix’ your problem (not being able to see an incoming message). As a side-effect, the root cause should now be logged in the error logs of Spark. That will allow us to address that too.
1 Like
I did a search and replace to replace the name of the server that Open Fire is installed on to OpenFireServer, but Domain is just how it is in the logs. We have only one domain and our openfire server shouldn’t be trying to contact another server or domain. I’ll have to look into that to figure out what is going on.
Ok, thank you. But I’m sorry, I don’t understand the fix. Is there a new version of Spark with that change rolled into it?
A link will be ready in few minutes
Awesome, thank you!
Here’s the build including the patch http://download.igniterealtime.org/spark/dailybuilds/spark_2_8_0_893.exe
I’ve got the latest build installed now. Here are the raw packets received from a message that again didn’t show in my chat window, but did make my chat window pop up and flash. The second message that she sent me, that you see at the end, did appear in the chat window:
<iq type="get" id="273-41011" from="OpenFireServer" to="gretchen@OpenFireServer/Spark 2.6.3"><ping xmlns="urn:xmpp:ping"/></iq>
<iq to="gretchen@OpenFireServer/Spark 2.6.3" id="yHC31-25939" type="get" from="taline@OpenFireServer/Spark 2.6.3"><query xmlns="jabber:iq:last"/></iq>
<iq to="gretchen@OpenFireServer/Spark 2.6.3" id="yHC31-25941" type="get" from="taline@OpenFireServer/Spark 2.6.3"><query xmlns="jabber:iq:last"/></iq>
<message to="gretchen@OpenFireServer" id="yHC31-25943" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25944" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25945" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<presence id="VEU93-10" from="angel@OpenFireServer/Spark" to="gretchen@OpenFireServer"><status>Online</status><priority>1</priority><c xmlns="http://jabber.org/protocol/caps" hash="sha-1" node="http://www.igniterealtime.org/projects/smack" ver="TJuVIXqTCVfJSthaPu4MtTbaf9A="/></presence>
<message to="gretchen@OpenFireServer" id="yHC31-25947" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25948" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25949" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25950" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25951" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25952" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25953" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25954" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25955" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25956" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25957" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25958" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25959" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25960" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25961" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25962" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25963" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><body>hi - can we try changing the battery to Michael's keyboard? We'll have to test it out though before giving it back lol</body><thread>Kb9PBC</thread><x xmlns="jabber:x:event"><offline/><composing/></x><active xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25965" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25966" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25967" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25968" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25969" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25970" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25971" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><composing xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25972" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><body>Unless you and Kar have changed it recently</body><thread>Kb9PBC</thread><x xmlns="jabber:x:event"><offline/><composing/></x><active xmlns="http://jabber.org/protocol/chatstates"/></message>
<message to="gretchen@OpenFireServer" id="yHC31-25974" type="chat" from="taline@OpenFireServer/Spark 2.6.3"><thread>Kb9PBC</thread><paused xmlns="http://jabber.org/protocol/chatstates"/></message>
Bah, I had hoped that your problem would have been resolved with the new build. Apparently not.
This new build did improve logging. Does this version write anything related to this issue in the Spark log files?
One last thing that you could try is to start Spark from the command line / dosshell. There’s a chance that an error is printed to the console, instead of the error log. Perhaps that could give us a new insights.
I was hoping this would fix it too! But I suppose having the problem happen fairly consistently to me, and not very often to any of my users, is the best possible troubleshooting scenario.
Nothing relevant in the spark log files. I’ll give running spark from the command line a shot.
Nothing printed to the console when it happens.