Openfire server : Exceeding the maximum number of online users

Hi,

After surfing the web and reading the openfire scalability, I found that I could have more than 50,000 users online and chatting at the same time.

My problem is that I am not able to reach more than ~9,000 users online, and when this maximum is exceeded the server disconnect some clients in order to connect the new one.

Note that my openfire server is installed on a windows server with the below specifications.

So my question is as follows… Can I really reach more than 50,000 active users ? and If the answer is yes, how can I achieve it, and what are the specifications of my server ?

Server Specifications :

Microsoft Windows Server 2008 R2 Standard

Intel® Xeon® CPU L5640 @ 2.27GHz, 2266 Mhz, 4 Core(s), 4 Logical

4GB Ram

Thanks in advance.

Disclaimer: I’ve not scaled my openfire that large, so take this just as general advice. We’ll need someone to chime in who’s scaled large.

Please post some logs from openfire (error, warning, and info logs). These will help people determine the actual problem (out of memory, out of threads, etc?)

I doubt 4GB of ram is going to cut it. You are running Windows Server, which will eat a lot of ram just sitting around idling, usually 1-2GB. My guess is openfire is probably being limited to somewhere around 800MB of JVM memory, and you probably are running out of avaiable memory for openfire to use. While I’d like to advocate using linux here as the linux os will consume much less ram by itself, and may perform better, if you are not familiar with linux already, this won’t be the time/project to learn. Instead, you probably will need to throw more ram at the problem, and make sure your openfire sees and uses more.

Log into your openfire admin page and look for where it says “Java Memory”. It will have a green bar and will say something like “xx.xx MB of xxx.xx MB (xx.x%) used”. When you start dropping users, is this maxed out?

There are ways to try to get openfire to use more ram/memory right now without adding more physical ram to your server, but this may not be ideal (or stable), and it’s much better if we determine the actual problem before continuing. Lets see your logs and go from there.

Hi,

Thanks for your reply.

Go to this link in order to see my openfire logs : https://www.dropbox.com/sh/pioengwzgxkc7wf/TUui9ZtvFI

And In my openfire admin page, the “Java Memory” bar is varying from 66% to 92.7% ( from green to red ).

kindly find below my tests and the results :

  • creating 30,000 users : succeded with no leak in memory.

  • logging in from almost 5,000 users : everything is going well.

  • when excedding this number approximatly, some old users ( from the first 5,000 users which are already connected) appears in the openfire admin page as

0
xxxxxx
Smack
Authenticated

Offline
Offline
0

and when those randomly users start to have and offline presence with an authenticated status, Java memory bar reachs 92.7% and start to varies between 66% and 92.7% continuously.

Any idea about this issue ?

Best Regards

You didn’t specify how much total memory is allocated for java (92.7% of what? 256MB, 512MB, 1GB?), but looks to me like you are definitely running out of memory. the jumping from 92.7% to 66% and back and forth is likely the JVM performing rigerous Garbage Collection in an attempt to free memory and not crash. With such a large install base as you have described, I stand by my orignal statement that 4GB of physical host ram is not enough. To put it in perspective, my laptop has 8GB of ram and I usually consume more than 4GB just with my daily activities. My workstation at the office has 8GB of ram and I usually consume about 6GB’s after opening all my programs and what-not.

You have two options (I would recommend a combination of the two actually).

**1) Upgrade your host’s ram. **I would put in as much as you can, reasonably. It’s going to be difficult to speculate on exactly how much ram is enough, it depends on a lot of things. But I would at the very least double your ram, if not more.

**2) Tell java to allocate more memory to the JVM. **Typically you do this by modifying the boot parameters for openfire to specify how much memory the JVM starts out with, and now much it’s allowed to baloon up to.

Here’s a link to a discussion about this: http://community.igniterealtime.org/message/158384#158384

Here’s a guide: http://community.igniterealtime.org/docs/DOC-1033

Given you only have 4GB of physical host ram, I don’t think you will have much headroom to increase the java memory settings without causing other problems. However if you are not “in production” with this server yet and still have acceptable room for testing, you can try first increasing how much memory is allocated for java and see what happens. For example, if openfire is only allocated 256MB’s of java memory right now, and your server has 1GB of free ram, you can try increasing openfire’s java memory to 512MB’s or more and see what happens.

If that does not work, or you have no more free ram available to allocate to java, I would upgrade your host’s ram, then I would tell openfire to use more of it.

Hello again,

Thanks for your reply.

After installing openfire on a Linux server, and by changing some Linux configuration (* hard nofile 131070, and* soft nofile 131070) ``I was able to connect 63,965 user online on the same server.

and when connecting more users to the server, the same old result appears ( some users are kicked out from the openfire to keep a place for the new users to connect )

Note that the Java Memory bar reached 93.3% and start to varies between 69% and 96.3% continuously :

Java Memory



856.56 MB of 917.75 MB (93.3%) used

Do you think that this is the maximum number of connection ?

Is this happening because Linux can’t support > 65536 open TCP connections ?

Thanks in advance.

I am interested in this too.

congrats, 9K users on windows, and about 64k on linux running from the same hardware, nice!

it appears your JVM has about 1GB of memory allocated to it. It looks like you got a lot more concurrent users by switching to linux and reducing your overhead, however you should note you still have overhead. 1GB of ram seems to be your limiting issue again. When it starts to bounce back and forth in usage, this is likely the JVM’s GC (Garbage Collection) kicking in, attempting to free memory.

Regarding the max number of users, I’m not aware of any such hard-coded value. It should be, to the best of my knowledge, only limited by your server’s available resources (memory, max number of threads, etc). In this case, you appear to be starving your server out of memory, causing it to eventually “limit” the number of concurrent users by dropping users (and eventually weirder behavior will set in).

I still do not think you have allocated enough ram/memory to openfire to handle such a large installbase.

I would take the steps to allocate more memory to openfire as seen in the guides I posted above, and/or install more physical ram in your server then allocate more of that to openfire.