Perhaps I missed it, but are you shooting a rifle or smoothbore? If it's a smoothbore, you may need a slightly larger ball. I read in a recent magazine article of a fellow who did some "experimental archaeology", and found that his smoothbore shot better on the second or third shot, but never came to the conclusion that maybe he needed a slightly larger ball to begin with.
If it's a rifle, you may want to try a thicker patch, or again, a slightly larger ball, for the presence of fouling causing better accuracy is often due to the residue actually reducing bore size just a bit, at least in my experience.
LD