I dislike compiling myself because then I also have to deal with upgrading them. Sometimes building is a pain because programs have arcane build steps, or are less-secure by default. I certainly don't trust myself to compile network programs (apache etc) with the most secure options. Paradoxically, I think it's safer for me to rely on packages than to compile myself. Maybe I'm being naive. :)