Openfire crashing frequently

I am using openfire 4.1.1 (with bosh - http-bind) on a 4 core, 16 GB ubuntu server.

Total users on openfire are about 3k, while active users at a time being around 1k.

Recently, we have been facing issues where the openfire crashing frequently (2-3 times a day) and a simple restart fixes it and makes it working.

At the time of the issue, we have the following entries in the log files:

warn.log :

org.eclipse.jetty.server.ServerConnector -

java.io.IOException: Too many open files

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)

at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java: 500)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635 )

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:748)

error.log

org.jivesoftware.openfire.audit.spi.AuditorImpl - Internal server error

java.lang.NullPointerException

at org.jivesoftware.openfire.audit.spi.AuditorImpl.ensureMaxDays(AuditorImpl.java: 299)

at org.jivesoftware.openfire.audit.spi.AuditorImpl.access$100(AuditorImpl.java:60)

at org.jivesoftware.openfire.audit.spi.AuditorImpl$SaveQueuedPacketsTask.run(Audit orImpl.java:382)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)

The file descriptors limit on this machine is 1 million.

Also, I have set “xmpp.audit.active” to false, then why it is going to org.jivesoftware.openfire.audit.spi.AuditorImpl piece of code.

Server details :

Java Version: 1.8.0_131 Oracle Corporation – OpenJDK 64-Bit Server VM

Appserver: jetty/9.2.z-SNAPSHOT

If anyone is aware of this or has any ideas, please help!

The error " Too many open files" is likely to blame. You need to increase the system limits on the number of open files a user’s process can have. Stack Overflow topics like this one linux - How to increase Neo4j’s maximum file open limit (ulimit) in Ubuntu? - Stack Overflow have the solution.

I have already increased the open file limit to 1000000. At the time when openfire crashes, they are not used fully but still openfire crashes!
Any other help would be appreciated.

There may be other file limit in the OS. If all else fails write a sort program that opens 100,000 files, and see if it fails on a similarly configured Ubuntu box.

If you have root install and use ‘lsof’. It shows whether sockets or files are in use and whether there are some unknown states.

There may be a JVM issue keeping references to abandoned sockets.