I think you've diagnosed your problem. I've made that mistake a few times, myself. Now I try to remember to swab with a dry patch or two thoroughly before loading for a range shoot or hunt. I also take care to wipe down the pan, frizzen, etc.
A couple of other things it could be: fouled flashhole and just gunk buildup in the breech.
I've found that even with a clean bore and using clean-burning powder, sharp flint and reknapping, I have main charge ignition difficulties after 15 or 20 rounds. A spit patch between rounds seems to wholly take care of this.
Lastly, a too-densely packed 'fuse' in the flashhole. I have to pick my flashhole, which is fairly small-diameter after loading, before each shot, to ensure fast ignition. I have an off-the shelf flashole pick, steel, very small diameter, that I push all the way into the main charge. Works like a charm.
This is not necessary on every flintlock (nor is the spit-patch), as I'm sure others here will think or say.
My wife's Trade Rifle, for example, and a different one I used to use, never needed the flashhole to be picked.
But if you don't do these - consider giving them a try, regardless of weather.
They do make the reloading sequence a bit more cumbersome, but it's worth it for me.