I've been interested in learning drums for some time, but felt I would not be coordinated enough, or at least that was my excuse. Well recently I decided to just go for it and here are my suggestions if you are looking at jumping into some uncharted waters.
First, shop around. If you are looking into a new instrument that tends to be expensive, make sure you check out eBay or Craigslist first. I especially like Craigslist, because you can deal locally and sometimes work out a better deal. Do this before laying out big cash for something that you may end up hating.
Second, buy "good enough". I think not buying in the right range is a problem. When you buy cheap stuff, it probably will never sound right. Do your research and make sure that what you are buying is decent stuff. That being said, unless you are rolling in the dough, don't go crazy expensive. I find that starting with gear that is too good, leaves little incentive to upgrade and learn about equipment. If you drop $1800 on your first guitar, chances are you won't try another one for a long time and you may have not picked the right one for you. Learn a little about your new instrument before making a huge investment in it. Besides, if you pick up a used instrument at a decent price, once you feel it's time to upgrade, you may be able to get out of it pretty much what you put into it and use that towards your upgrade. For me, I bought a used Yamaha kit from a friend, and got some old cymbals and hardware from another. I've already replaced the kick pedal with something new and have started thinking about a new cymbal or two.
Fourthly, submerse yourself in the culture of your new instrument. Check out magazines, websites, forums, etc. Learn about styles, gear, and anything else that will help you connect with it. After that, it's practice, practice, practice. Then once you start feeling confident, find an opportunity to play out on your new axe (no matter what form it takes). You will find that after you get to a level of comfort, you will start to see it effect your overall musicality. When you play your main gig, you will be thinking about where your other instrument fits in and not just about what space you can fill. You become a more well rounded musician with more than one point of view. Look out though, this could become addictive :)
What instruments have you taken up beyond your 'first love', and what did you find was helpful and what wasn't so great?