I use Source Mage for my second OS (Gentoo just makes me want to kill something), and if you can get a desktop up and running on the first try, you're truly blessed.
Not only are there problems with building, but it will eat your CPU up for hours at a time. Oh, and you'll use more HDD space, due to having sources. Some sources, too, like the kernel, are many times larger than the binaries they make!
If you want to encounter problems, or want minimalism, source-based is fine. If you really want a usable desktop, go for the big downloads. Binaries may not compress as well as plain text, but it should Just Work™. If you want a fairly light and fast distro, but with good features, too, try Zenwalk or Yoper.
If building from source for every thing were made to work close to every time, it would be common. As it is, that's a pipe dream.