My qualifications here: I was a NOC technician and sysadmin for several years at a hosting datacenter. I was good at it, but I left because sleeping 3 hours a day is not very good for your long-term health. Keep that in mind when you read the below
I run a server that handles around 5 people (FTB Unleashed, we've also used Mindcrack and Ultimate in the past). The primary bottleneck seems to be CPU usage on the server, and network latency between the server and the clients.
I have it set to start with 4gb of RAM and to grow to 6gb, and this seems to be far more than is necessary. (I have it though... unallocated RAM is wasted RAM!)
Watch your server with something like htop, start with a smaller minimum with a large maximum and just see what the server allocates during peak time. This gives you an idea of what you need. Set something respectably higher than that just to allow for unexpected peaks.
I think you'll find that RAM is
not where your "lag" comes from. It is most likely the fact that Minecraft, in 2013, remains single-threaded and is not thread-safe. See
TickThreading as this may help you, but keep in mind it is
experimental and it is likely to eat your children, after corrupting your chunks.
Network latency is, of course, mostly determined by your ISP, the client's ISP(s), and the geographical distance between. Now, if your server is overtaxed (load average is high) then you will see some additional latency as the kernel has trouble keeping up, but if that's the case then worrying about your RAM settings is the least of your concerns.