I've got an RS140 (with external RAID10 storage in a 1U can) and 16GB of RAM, it's my "overflow" server. Only has 9 2Mpx Vivoteks and runs 40-45% utilization. I have 15 more cameras in boxes, I figure I can hang another dozen off this little server. E-1225v3.
My POC server is an older Supermicro tower with an E3-1220 v2 and 16 2Mpx cameras, bounces between 65-80% utilization, might could squeeze another couple in if I had to. Primary is an E3-1270 v3 cruising along at 50-65% CPU load with 24 cameras, mostly 2 MPx Vivoteks but a 5 Mpx Bosch and a couple of 360 deg Panasonics as well, can bump up another 8-10 cameras when I need them.
The Xeon's are good even for the little weenie non-hyperthreaded versions, memory bandwidth is a good thing. Fast and redundant storage is good, too. Windows 8.1 Pro is perfectly happy on server hardware and a darn sight cheaper than Server. Will have to go to Windows 10 here pretty soon! If you have 8 GB RAM on a dedicated machine you're probably fine. All three servers together are only receiving about 110 Mbps, so network capacity is not an issue even with 50 cameras.
I don't think the cost premium for a Xeon over an i3 is that much, am I mis-remembering?