Gauge... friend or foe?!

Why is it that when I use the fancy yarns, the yarns that cost more than a latte or two, the yarns that feel sooooooooo good, that don't fall in the "muppet" category, I somehow manage to screw stuff up?  

I bought this awesome crochet hat pattern and have been making cute brim hats for my kids and then for my best friends kids.  When I use cheap acrylic or bargain store cotton, they come out fine.  When I use "fancy-schmancy" yarn I always screw up and the hats are too small.  I check gauge, sorta.  I make notes of what yarn and hook combo makes what hat size, but nope, my not so nimble fingers always get nervous with the fancy yarn.  I'm so worried I'm gonna run out, mess it up, ruin it, that subconciously I think I make myself screw up.  Blah.  

Off to frog some dreamy berreco Pure Merino and pray it will work up again in the right size this time.