Server hardware discussion. Any server Owner Must Read !

[Arrxzon]

i was wondering if anyone can help me with my cpu it is always at 90-120% usage with 20 players.
Intel Xeon E5-1620v2 4c/8t 3,7GHz+/3,9 GHz+ from ovh.ie

I really need to support alot more players as the demand on my server is pretty high
we could reach probably 50 active players if the cpu would allow it.

so i would appreciate any tips on how to reduce this crazy usage,
it isn't anyone loading chunks/big laggy setups because it's been like this since the start when everyone started fresh.
Thanks in advance

 

[DZCreeper]

i was wondering if anyone can help me with my cpu it is always at 90-120% usage with 20 players.
Intel Xeon E5-1620v2 4c/8t 3,7GHz+/3,9 GHz+ from ovh.ie

I really need to support alot more players as the demand on my server is pretty high
we could reach probably 50 active players if the cpu would allow it.

so i would appreciate any tips on how to reduce this crazy usage,
it isn't anyone loading chunks/big laggy setups because it's been like this since the start when everyone started fresh.
Thanks in advance

CPU percentage is measured in such a way that 100% = 1 core being fully utilized. You basically have 3 options:

1. Run a less intensive modpack. Not ideal but it would help. You will run into the same problem once you reach a new, larger number of players.

2. Rent a server with a better processor. Your E-5 series processor is designed for more typical servers that can spread the load across multiple cores. An E-3 series Xeon chip would perform better in this scenario, less cores, higher speeds per core.

3. Start another server. You have 4 cores and your only utilizing 1 or 2 right now. You could run another server on a different port and split the load.

 

[Connor Gavitt]

Since minecraft uses only one core max I assume he is using 1.2 logical cores so about 60% of a single core which is perfectly ok and I doubt your even having TPS drops because of the CPU. Though @DZCreeper is right about the E3 and E5 processor though yours isn't designed for high usage on a single core, it's meant to be spread out.

i was wondering if anyone can help me with my cpu it is always at 90-120% usage with 20 players.
Intel Xeon E5-1620v2 4c/8t 3,7GHz+/3,9 GHz+ from ovh.ie

I really need to support alot more players as the demand on my server is pretty high
we could reach probably 50 active players if the cpu would allow it.

so i would appreciate any tips on how to reduce this crazy usage,
it isn't anyone loading chunks/big laggy setups because it's been like this since the start when everyone started fresh.
Thanks in advance

If you are having lag issues with only 60% of a core being used id suggest opening a thread in server tech support and provide your forgelogs and server.log and modpack so we can diagnose the issues. Minecraft can and does lag without the CPU being maxed quite often and chances are you are having in game issues not actual CPU problems as the 1620v2 is not a bad CPU even for minecraft and you are just barely using over half of its full power for that server.

 

[Arrxzon]

snip

snip

damn we previously were running on an i3 but didn't have the ram to support the players
ovh provided the ram nicely but not the cpu i guess

even with the i3 we had the same issue, but that could be related to chunk generation we have since limited that with worldborder

yeah your right even do it says 100% usage our tps would still be pretty stable just spikes of lag every now and then
i just thought it should handle alot better than it is with more than 30 players (only if it could utilize it's full potential)

 

[DZCreeper]

Since minecraft uses only one core max I assume he is using 1.2 logical cores so about 60% of a single core which is perfectly ok and I doubt your even having TPS drops because of the CPU.

That's not really how hyper-threading works. Its not a full second core, it lets some parts of the CPU perform 2 functions at once. Assuming he is using a Linux OS, CPU usage is measured as 100% per core. 120% usage tells me 1 real core is maxed out along with hyper-threading. If TPS is at 20 though your performance is okay and you should look for your bottleneck elsewhere.

 

[Connor Gavitt]

That's not really how hyper-threading works. Its not a full second core, it lets some parts of the CPU perform 2 functions at once. Assuming he is using a Linux OS, CPU usage is measured as 100% per core. 120% usage tells me 1 real core is maxed out along with hyper-threading. If TPS is at 20 though your performance is okay and you should look for your bottleneck elsewhere.

Then how is he getting 120% usage if 100% is a core? Minecraft isn't multithreaded.

 

[DZCreeper]

Then how is he getting 120% usage if 100% is a core? Minecraft isn't multithreaded.

Because Minecraft is not entirely single threaded, some things like garbage collection are multi threaded and with MCPC+ networking and chunk loading is as well. The most intensive part, the world ticking is single threaded but that appears to be changing around the release of 1.8.

 

[obscurehero]

Most of the world is single-threaded. I've just written something up on the brand-new i7-4790 which is cheaper than the previously fastest E3-1270v3 here.

My players ended up having to make a TON of open block tanks and looping computercraft scripts before it even started to make a dent in the performance. I gave a few full creative and time to try to build the most laggy contraptions ever and it wasn't until openblock tanks that they finally were able to start making a dent in the Ticks-Per-Second (TPS).

I would NOT get an E5 for any modded minecraft server. You just can't get the per-core performance that you can get with an E3 or this i7.

 

[Harvest88]

The problem with the i7s that they are cheaper for a reason, they are desktop grade.. No ECC nor it is designed to run under 24/7 operation.

 

[Viperidae]

It's usually better to build your own dedicated server IF AND ONLY IF your bandwidth can suffice, pretty much every non satalite internet provider is low latency so your only worry then would be to mitigate DOS attacks your self as minecraft is prone to it.

For your server if you build it I suggest ONLY a xeon E3 1240 1245 1270 1280 or 1230 processor hopefully at least v2 as those have the best performance with minecraft and that 1280v3 is the best on the market and pretty cheap and can easily host 8 400mod servers with 30 people+ on each no prob. Also I do suggest a SSD as ALL 1.6+ packs will lag severely without high read and writes.

8-16GB of RAM will suffice unless you are gonna be doing some cray stuff or running a lot of servers.

Why I suggest only a E3 processor is because a E5 isn't gears for single core applications as much and i7 processors are simply not made to be run 24/7 with high loads, part of the reason it's classified as a "desktop" processor according to http://ark.intel.com

Also don't judge ANYTHING by clock speed. Use CPUbenchmark.net and for minecraft use the single thread list.


Anyway if you don't want to build your own server which is ok there are options. But before you buy anything online send in a support ticket asking exact CPU specifications even if they already tell you in their site as they should to test support response speed and tolerance assuming there listed it already somewhere on the site. If the response takes more then 3 hours don't even think of using them, 3 hours is worse then OVH. That's pretty sad.

For your dedicated server you may rent use the specs I reccomended for your own personal serve above, they apply here too.

Some good hosts you may like that are reputable and reliable are OVH.com or OVH.co.uk, servermainia, hetzner, fragnet( support is meh), pheonexNap, and soyoustart (they will have game servers out soon at PVH prices so they are DEFINATLY A OPTION, but these are just the few I have liked in my experiences. Also be sure your server has KVM/IPMI or something similar in case the actual server crashes so you can start it manually instantly rather then waiting an hour for support to do it.[DOUBLEPOST=1400626526][/DOUBLEPOST]
Forgot to tag you lol, there you go, tagged @Viperidae

First of all, thank you for your help!

I took a look and OVH was the one I liked the best. Do you think the VPS Classic 4 will suit my needs? http://www.ovh.co.uk/vps/vps-classic.xml

 

[Connor Gavitt]

First of all, thank you for your help!

I took a look and OVH was the one I liked the best. Do you think the VPS Classic 4 will suit my needs? http://www.ovh.co.uk/vps/vps-classic.xml

I would use OVH for larger servers however their VPSs use AMD processors which aren't that great for minecraft, intel currently has better single threaded performance on a affordable scale.

If you want a VPS make sure it uses a E3 processors like ramnode, and I think creeperhost +many others.

Also check out OVH sister company soyoustart you can get a E3 1245v2 32GB RAM and dual 120GB SSD for $70 a month

 

[systemv]

Overlooking the nature of how CPU utilization is measured vs. CPU use is measured.

I take it this a Linux (Unix) system then?

Look at what the output is from vmstat:

yogs@play:~$ vmstat 1 100
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b  swpd  free  buff  cache  si  so    bi    bo  in  cs us sy id wa
2  0  16464 237248 216488 23636708    0    0    5    43    0    0  4  0 96  0
1  0  16464 236992 216488 23636804    0    0    0    68 6609 13909 10  1 90  0
0  0  16464 236704 216488 23636960    0    0    0  332 7538 15229 16  1 83  0
0  0  16464 236400 216488 23636968    0    0    0  544 7166 14901 13  0 87  0
0  0  16464 236192 216488 23637064    0    0    0  3556 6941 14970  8  1 90  1
0  0  16464 236224 216488 23637156    0    0    0    84 7023 15884 10  1 89  0
0  0  16464 235456 216488 23637204    0    0    0    76 7267 14732 10  1 89  0
0  0  16464 235980 216488 23637300    0    0    0    28 6238 13587  7  1 92  0
0  0  16464 236112 216488 23637376    0    0    0    60 6499 13885  8  0 91  0
0  0  16464 235980 216488 23637476    0    0    0    92 6610 13737  9  1 90  0
0  0  16464 235860 216488 23637572    0    0    0    20 6660 14243  9  0 91  0
1  0  16464 235828 216488 23637636    0    0    0    88 6371 13591  8  0 91  0
0  0  16464 235860 216488 23637732    0    0    0    60 6564 13785  8  1 91  0
0  0  16464 236120 216488 23637828    0    0    0    76 6588 14223 10  0 90  0

Here's a few seconds of output from my E3 box running two MC servers simultaneously - A MagicFarm2 server and a YogscastComplete (230+ mods) server.

I'll quote the man page for vmstat:

FIELD DESCRIPTION FOR VM MODE
Procs
r: The number of processes waiting for run time.
b: The number of processes in uninterruptible sleep.

The last four columns:

CPU
These are percentages of total CPU time.
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.



My CPUs are mostly Idle. They are barely notching a dent in the CPU throughput. Utilization is probably very low and CPU use is low.


CPU utilization is a ratio of the number of clock ticks (not Minecraft ticks) when loaded with the task, vs. unloaded (not running the task) compared to the number of ticks in the IDLE thread (lowest possible priority thread) of the system. What's important is the CPU utilization because it tells you how much share of the CPU is devoted to your task per second (any unit of time you wish).

CPU % usage is a mashup of time spent in each process per second (any unit of time you wish). But these are not the same things.

It is not possible to say categorically that the "lag" you notice is caused by the CPU throughput issue. It could be your network, it could be your fixed storage controller, is your system busy swapping more than it should? it could be your server is making use of other processes (like MySQL for instance with a load that forces the MC server to be busy servicing that connection to MySQL), etc...

Come back to the forum with data from "vmstat 1 200".

It's tempting to use CPU% alone and even the output of "top" but it's good to get a clearer picture of what the system's idle task is up to, and how many processes are waiting to run, and waiting blocked (sleeping).

Edit: make double sure your BIOS has configured your cores to run hot and the numbers you see (highlighted **) are what you expect them to be. It is possible to tailor the BIOS to run cooler to the chagrin of a harried MC admin.

yogs@play:~/mcma/Minecraft$ cat /proc/cpuinfo | grep Hz
model name      : Intel(R) Xeon(R) CPU E3-1280 v3 @ 3.60GHz
**cpu MHz        : 3592.001
model name      : Intel(R) Xeon(R) CPU E3-1280 v3 @ 3.60GHz
**cpu MHz        : 3592.001
model name      : Intel(R) Xeon(R) CPU E3-1280 v3 @ 3.60GHz
**cpu MHz        : 3592.001
model name      : Intel(R) Xeon(R) CPU E3-1280 v3 @ 3.60GHz
**cpu MHz        : 3592.001
model name      : Intel(R) Xeon(R) CPU E3-1280 v3 @ 3.60GHz
**cpu MHz        : 3592.001
model name      : Intel(R) Xeon(R) CPU E3-1280 v3 @ 3.60GHz
**cpu MHz        : 3592.001
model name      : Intel(R) Xeon(R) CPU E3-1280 v3 @ 3.60GHz
**cpu MHz        : 3592.001
model name      : Intel(R) Xeon(R) CPU E3-1280 v3 @ 3.60GHz
**cpu MHz        : 3592.001

 

[systemv]

The problem with the i7s that they are cheaper for a reason, they are desktop grade.. No ECC nor it is designed to run under 24/7 operation.

i7- are a great value. And they can run as long as the juice is hot.

i7-4770K in particular. Meow.

i7-4960X. Hubba hubba

By your analogy, MC is "desktop grade" application (single core). If it was multi-threaded you'd have a point, and i7's would start to wilt to the E3/E5 advantages.

 

[Connor Gavitt]

Very nice post @systemv would you mind telling us what host your using that offers that 1280v3?

 

[systemv]

Very nice post @systemv would you mind telling us what host your using that offers that 1280v3?

Thanks. I left out some things that didn't occur to me until after I pressed "post". I'll try to collect them later into something coherent.

As far as the hosting, I did a little research on the silicon I wanted and the commercial hosting company (Allgamer.net) built the system for me to spec. I signed a contract to offset their upfront costs to build the box, but they were willing to do this to give me the opportunity to grow my userbase. I rent the whole box (no VPS) and have full latitude to administer it.

I've used them for a year+ already and despite a little extra cost, their capability to react personally to any issues far outweighs the expense. If you know your way around Unix, I'd advise any Admin/owner of a MC server to consider a company (not necessarily Allgamer.net in every case) that gives you the warm-fuzzy you're looking for.

 

[systemv]

I would use OVH for larger servers however their VPSs use AMD processors which aren't that great for minecraft, intel currently has better single threaded performance on a affordable scale.

If you want a VPS make sure it uses a E3 processors like ramnode, and I think creeperhost +many others.

Also check out OVH sister company soyoustart you can get a E3 1245v2 32GB RAM and dual 120GB SSD for $70 a month

I agree re: AMD. It's unfortunate. I like the company, but they don't have the silicon now in the field that makes it cost effective to run MC on their chips. Intel has all the cards on this one.

 

[obscurehero]

@systemv very good points. However, let's make sure people understand you have one of the fastest per-core CPUs on the market. I would hope you have a lot of idle/sleep time.

People underestimate how important disk IO and networking are for smooth operation.

What was your quote? Reliablesite offers i7-4790s for less than $100/m. PM me for more info.

 

[systemv]

For those who have multiple cores, MC does take advantage of them in some respect. @obscurehero is absolutely right about fasted per-core CPU, for the sake of MC itself.

top -H

Then turn on these flags as follows:
press f, then j, then ENTER
then press 1
press u, enter the username that runs your MC server, press ENTER

You'll see something similar to this. All those lines with "java" are just threads running from the java application (your MC jar).
MC itself is single threaded, but there are engines (MCPC+, or Bukkit, etc..) that can use multiple threads.

Notice how the threads are spread out among the cores (P column). Core 5 is where my MC is running, but the other cores are busy as well handling threads where implemented by my server (MCPC+).

To get an idea of just how many threads:

Find the PID of your java server. Something along these lines, multiple ways to do this:

ps aux | grep java

yogs@play:/proc/30818$ ps aux | grep java
game  30818 30.4  8.9 17247204 2942292 pts/3 Sl+ 04:10 255:50 /usr/bin/java -Dfile.encoding=utf-8 -Djline.terminal=jline.UnsupportedTerminal -server -XX:PermSize=128m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseStringCache -Xmx28000M -jar ftbserver.jar nogui -d yyyy-MM-dd HH:mm:ss -nojline

This PID = 30818, then go to your system:

yogs@play:~$ cat /proc/30818/status


Name:  java
State:  S (sleeping)
Tgid:  30818
Pid:    30818
PPid:  14572
[edited for brevity]
Threads:        73
[edited for brevity]
Cpus_allowed:  ff
Cpus_allowed_list:      0-7
Mems_allowed:  00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        1

73 threads at this moment.

Back to top:

What top says (removed from the paste the other two servers running on this box, probably why Cpu0, 1, etc.. riding as high as Cpu5)

The core (P column) and CPU% column will reflect best information at that moment per thread.

You can also dig around /proc/PID/ to get the same data.

top - 18:04:34 up 102 days,  5:38,  3 users,  load average: 0.63, 0.72, 0.74
Tasks: 461 total,  2 running, 457 sleeping,  0 stopped,  2 zombie
Cpu0  : 10.8%us,  1.7%sy,  0.0%ni, 87.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  8.7%us,  1.0%sy,  0.0%ni, 90.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 12.3%us,  0.7%sy,  0.0%ni, 87.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  4.7%us,  1.0%sy,  0.0%ni, 94.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  7.9%us,  0.0%sy,  0.0%ni, 92.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  : 16.4%us,  0.7%sy,  0.0%ni, 82.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  1.3%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  1.0%us,  0.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32907964k total, 32676612k used,  231352k free,  221300k buffers
Swap: 15657980k total,    20340k used, 15637640k free, 24446480k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND
30873 game      20  0 16.4g 2.8g  20m R  37  8.9 174:04.96 1 java
18898 game      20  0 16.4g 2.8g  20m S    2  8.9  0:03.18 3 java
30936 game      20  0 16.4g 2.8g  20m S    1  8.9  23:42.06 2 java
30823 game      20  0 16.4g 2.8g  20m S    1  8.9  2:50.54 0 java
30824 game      20  0 16.4g 2.8g  20m S    1  8.9  2:50.14 6 java
30825 game      20  0 16.4g 2.8g  20m S    1  8.9  2:50.72 5 java
30827 game      20  0 16.4g 2.8g  20m S    1  8.9  2:51.67 3 java
30829 game      20  0 16.4g 2.8g  20m S    1  8.9  2:50.67 1 java
11410 game      20  0 16.4g 2.8g  20m S    1  8.9  1:50.22 2 java
18658 game      20  0 16.4g 2.8g  20m S    1  8.9  0:12.41 6 java
30822 game      20  0 16.4g 2.8g  20m S    1  8.9  2:50.97 7 java
30826 game      20  0 16.4g 2.8g  20m S    1  8.9  2:50.81 4 java
30828 game      20  0 16.4g 2.8g  20m S    1  8.9  2:51.19 2 java
18417 game      20  0 16.4g 2.8g  20m S    1  8.9  0:23.01 7 java
18698 game      20  0 16.4g 2.8g  20m S    1  8.9  0:12.50 4 java
18697 game      20  0 16.4g 2.8g  20m S    0  8.9  0:00.45 7 java
18815 game      20  0 16.4g 2.8g  20m S    0  8.9  0:07.41 3 java
13924 game      20  0 73448 1792  988 S    0  0.0  0:00.04 7 sshd
13925 game      20  0 26464 7760 1760 S    0  0.0  0:00.18 1 bash
14033 game      20  0  7204  612  512 S    0  0.0  0:00.01 1 tail
14571 game      20  0 27168 1452  912 S    0  0.0  0:25.12 1 screen

 

[RSNET-Radic]

What was your quote? Reliablesite offers i7-4790s for less than $100/m. PM me for more info.

Thanks for the mention! I can answer any questions about hardware or about the i7 4790 as well.

While Minecraft does use the other cores, it's mainly for plugins, the main loop (where all the load is) uses a single thread and needs the fastest per-core processor (unless you Bungee). The Intel Haswell based CPUs are the best to-date at that.

 

[systemv]

Haswell info here.

Some other data points that I've jotted down in my mind about hosting on dedicated hardware.

1. Renting the whole box is good. VPS isn't viable for the best performance.
2. Intel over AMD, without question. The design of the microarchitecture is the key and Intel has spared no room for error in research to make their microarchitecture better for performance.
3. If you really got the inkling to design your own system, then renting a 20U space for colocating will cost you about $250/month. For huge server populations, you may consider this for the reason that you can put the best platform (designed to spec) in a rack fitted to fat pipes. I've sought out quotes for this route, and if I had a spare $2k to plunk down for the system, I'd probably do it.
4. Like fish tanks, bigger is better. So for memory get the most you can afford. All of these options so far max to 32Gb RAM. It's difficult to search for the boards and silicon that can support more than 32 GB RAM. Xeon E5 spring to mind. I don't think the E3 can, but I could be wrong. The Sandybridge i7's can, but that's a generation back from Haswell.
5. Broadwell is coming.
6. Where is your system physically in relation to the big meet-me-rooms: MAE-West would be my pick for geographical location since I am on the west coast. The Westin Building in Seattle is holds the meet-me rooms in that area. But over the last 15 years or so there's been some movement in the best geographical location to choose. The knowledge of the location of these exchanges are important in choosing your provider.

The term "die shrink" has a lot of meanings. It is a reduction of the die size. This can (by intent) reduce timing requirements between areas of the floorplan (LRU, Cache, etc..) This doesn't impact the design of the microarchitecture directly. L2 Cache? Cache Coherency, Decoders per core, etc.. Those aren't signal level design decisions.

Many of the choices made to select the host of your server aren't going to amount to much, in the end. More cores (check), good network location (check), and astute management of the resources on your hardware (knowing exactly what you're running and the impact on the system is a nice barometer of how well it's managed.)

Being on the motherboard and silicon design team of a major game console, I've come to realize that the selection of silicon and architecture of the board are based on the intent of use. But, I've also been stung by my own ignorance about the software configuration so that is why I mentioned astute management of the mods/plugins you select to run. Learning those pitfalls takes some patience and willingness to dig into the software configuration just as much as digging into the hardware specifications that are chosen.

I think the caffeine burn is starting to wear off so I'll stop here.. ;-)