basically you have 2 ball sizes: .490 and 0.495. On top of that you have 3 patch sizes: 0.010, 0.015, and 0.020 (sure, you might be able to find something in the 0.005 or 0.008 range, or even thicker than 0.020 like jeans, but your commercial offerings are the 3 above).
In my opinion, the larger ball for diameter (so 0.495 for .50-cal, 0.535 for .54-cal, etc.) is better with a reasonably tight patch. It stands to reason: what do you think would be more accurate in a .30-06 barrl, a .30-cal bullet or a .270-cal bullet. Use the larger bullet and fill the bore with whatever patch you can fit easily!
This will make loading somewhat more difficult, but in my opinion, more accurate shooting.
I shoot 0.535 in 0.015 patches in my .54-cal. 0.530 balls in any size patch just are less accurate.
One exception might be shooting your tighter ball-patch combo for your first shot while hunting, then going with the smaller ball and same patch for the quicker reload, which you might be able to load without a starter.