Why does baking straight from the refrigerator not seem to work for me?

Every time I have tried it I end up with a very dense bread with inadequate size (oven spring is fine, it just always seems much smaller than it should be for the amount of dough). I think it would be much easier if I could bake straight from the fridge but it never seems to work for me. I use this recipe most of the time. Thanks for any help!