It's possible to push a rate way, way down, with each method listed below, in order of importance:
FICO - Lenders work in 20 point ranges. If your score is 761, a rate might be 3.250. With a 781 FICO, the rate might be 3.125 and with an 801 score the rate might start at 3.00. For discussion purposes, let's use the middle FICO of 781 and a 3.125 rate.
LTV - Most of the best price products seen on line have a 75% Loan To Value or lower starting point. That 781 FICO, at 80% Loan To Value might rise to a 3.250 rate. At 70% the rate might drop back to 3.125. At 60%, perhaps the rate drops to 2.875.
Relationship. Got cash or securities to move? Big Banks's like the sound of that! Some scale their rate benefits at $100k, but the majority of banks will move the needle at $250k, $500k, $750k, and $1m.
If the starting rate at 781 FICO, and at 60% LTV is 2.875, and you move $750k to a bank, now you might see 2.50 or better. These funds aren't held hostage. It's possible to deposit, close, and move back, but that's kind of a hassle. The bank holding the funds might have incentives to keep the funds with them so consider leaving them for a bit if the benefit makes sense.
Relationship. Wait... didn't you just say this? Yes, but now I'm using it in a different context. Large companies with payroll services managed through some banks may have additional discounts offered to their employees for purchase or refinance loans. Not every company has a relationship based discount option, but it's a good idea to ask. If you're with a brand name silicon valley company, it's highly likely a discount exists.
Loan Size. Some borrowers took out $460,000 loans last year and enjoyed "Jumbo Pricing". The Conforming Loan Limit was approximately $450k so Big Bank's allow these loans to fund using portfolio pricing. Today, with the Standard Conforming Loan limit reaching $484k now, a $460k refinance must go Standard Conforming and the rates are not as sexy as those bank numbers. A good loan officer can work around some of these issues with you.
Occupancy. Owner occupied - best terms. Second Home - still pretty good. Investment.... ouch. If you've got a national average 30 year fixed rate of 3.99% (Freddie Mac) it's a pretty good bet you can improve on that average as noted above. When it comes to investment properties, add .75 or more to that average rate, even more with an ARM refinance product. Lenders aren't that enthusiastic about taking on rental property refinances so be ready for some pretty eye opening numbers when you call in.
Hope this helps. Questions? Ask away.
My .02c
SGIP