If the day was dry, I leave my ball in (provided the season isn't over). If temps are cool, I leave my gun in the cold; I'll take it inside only during the early season when it's often warmer outside than in.
I've never done this, but I know folks that will leave their powder and ball in all season - for a few months, and it still goes off fine.
I tend not to do this as my barrel is not a blued production barrel and tends to require anti-rust treatment every few weeks or so.
With a brass range rod and decent ball screw, pulling PRB is a breeze (I load 0.535 ball and 0.015 patch) in my .54-cal. [Quite a bit tougher with hickory ramrod and antler handle I made!!!] So I have few or no issues pulling when weather was wet/damp during the day. I'll often pour out the powder and let it dry for a few days, then put it back in the powder tin. Depending on condition, I'll even reuse the patch. I do use a fouling scraper to take out any excess stuck in the breech and run a couple dry patches to 'clean' after pulling ball.