Not knowing the loan amount, it's over 4 points on a $510k loan, over 3 for a $700k loan (I'm assuming the $23k is "points and fees") I've had High Balance Conforming finance 2.0 points to get a rate very, very close to 2.0%, but 4 points? No thanks!
@Cares - some questions
"10 years?!?! Some people will have moved twice during that time frame." - For an investment property, something that's a long term hold, it might work. We're on the same page with a 3-4 point buy, but to lower costs for non-owner doesn't seem to be a bad idea in all circumstances. What say you?
"$23k? Invest in a conservative fund earning 3% compounded annually? FV of $23k after 10 years is $30k. So you lost $7k in investment growth which should be accounted for which then is the equivalent of 35 more months of mortgage payment savings to make up for it." -
Hard to find a guarantee of 3% without some risk - harder still as rates continue to level out at .001% As well the $23k is "post tax" money if paid out of pocket, not if financed. There is already plenty of ground to make up with that $23k really being $31ish to start with. This is also assuming one had $23k to spend. In a best case scenario you're starting out with untaxed funds being financed.
We could go very, very deep in the weeds to see each and every scenario case through to reasons why, or why not to buy a rate down. I'm neutral on it, except when we reach the level of 3-4 points. That's an automatic no-go. Less than 2.5 points? Never hurts to work through it and see. Sometimes it works, sometimes not.