You asked this question years ago and things have changed (like the real estate market) since you asked.
However, the first part of the answer is that the people to whom you are giving your house or its selling price are not responsible for tax. it is not taxable income to them. You as the donor are potentially responsible for gift tax. This is the gift tax land, not the income tax land.
Right now the annual exclusion is $13,000 per year, so you (and your spouse if applicable) can donate $13,000 to each person ($26,000 if you combine with your spouse--if you write one check you have to fill out IRS form 709) to each person.
That $13,000 is just the annual exclusion. There is also a lifetime unified credit. Currently it is more than $5,000,000 per spouse (and the spouses if they jump through the right hoops are able to combine their unified credits as well). Therefore unless your house is palatial you can give it or its proceeds to your mother and brother without likely being responsible for gift taxes (assuming you have not given millions away already during your life).
Now, some qualifications. The $5,000,000 unified credit (I believe it's actually $5.12 million right now) is scheduled to be drastically reduced to $1,000,000--however this issue is a sensitive political subject which is highly volatile and dependent on the 2012 elections--Republicans are likely to want to continue the very high unified credit and Democrats are likely to want to repeal it.
The annual exclusion, however, is less likely to fluctuate. Currently $13,000, it is indexed to grow over time in $1,000 increments and is about due for another increase to $14,000 (perhaps in 2013). The foundation tax advice for people with large estates is to give annually and continuously in increments equal to the maximum annual exclusion.