What load are you using?
Here's why I ask: as bore sizes increase, and charges increase, it seems to me the way the gun "twists" when fired due to the rifling tends to be noticed more. I start to notice notice it in the .50-54 range, especially when the velocity is up there (say, 1750-1800 fps or more). I also tend to notice it more when shooting FFFg than a load with the same velocity using FFg.
I suspect twist rate can also be a factor, at least I notice it more in a .58 with 1:48 rifling than I do in a .58 with 1:112 rifling.
I first noticed this in cartridge handguns. My 1911 had a bit of "hand twist" but a 45 Linebaugh had a definite "snap twist" (loads were, respectively, a 230 gr FP at 850 fps and at 1,700--the bullet weight is roughly the same as a .54 round ball).
Honestly, if it is consistent but group sizes are good, I'd just adjust the sights to bring the point of impact to the desired location.