The first thing I do is establish session baselines for certain groups of server or for heavy use applications.
I will do an example with servers
For one month I select a silo of servers in my farm and get the following information
Avg Memory Per Active Session, and Avg Cpu % per active session from the XTS Starter templates for capacity planning (because I have quad core servers Avg CPU % has a max of 400% per server so I divide this number by 4 so i can give reports to managment)
I then take the information that I got from this report which is an avg session uses on average 60 Mb of ram and 1.24% CPU Utilization. I use this as a baseline.
the next thing I do is create a concurrency report. I create a custom field that is called Estimated Memory Usage that is broken down by server and day (User Concurrency * 60 Mb) this lets me know how much memory was likely being used around peak usage time.
Each of my servers has 4 Gb of ram, I want to keep the maximum ram usage at 3 Gb to avoid performance problems so I include on my graph a threshold line at 3 GB. now my report shows how many days exceded that threshold and how many came close. I have this report deleivered to managment each month it is easy for them to see if we are approching the threshold and need more resources. I had been using a spreadsheet where i plugged in valuses like avg memory, number of servers, total ram threshold % and other information, this makes it much easier so that I just have a report automaticaly generated.
The report can be run genralized for your farm, for a set of servers, or for specific applications.