I know it's politically correct to say you have to decide for yourself, but I'm not going to say that. Assuming you can afford it, get the 20D. I faced the same decision last month.

You already mentioned one of the deciding factors, the size. The Rebel XT is too small for me and I'm guessing it's too small for you. What you didn't mention was the viewfinder. The 20D has a bigger viewfinder and the XT has a tiny viewfinder. Consider this, there are plenty of people who claim that even the 20D's viewfinder is too small. Finally, controls: that wheel on the back of the 20D that is missing from the XT is very handy for anybody but the point and shooter. The XT just doesn't have it. Also, to change the ISO and other settings on the XT you are forced to use an awkward button sequence plus the LCD monitor. On the 20D you use that handy wheel and a button. If you shoot inside and out you'll be changing ISO all the time. The wheel also make AE compensation trivial.

Skip the XT.