Kevin, Pardon me for Monday morning quarterbacking, but in my experience in shooting .54's, and I have had several, I have never had one that shot worth a darn with your wimpy 40 - 50 grain charges. With the exception of my first rifle, a T/C Renegade, they all seemed to like charges in the 80 - 90 grain range. And I always used Ffg GOEX powder.
I really think you need to develop your load.
Off a bench, start with 60 grains and shoot a three shot group at 25 yards. Wipe the bore and increase the charge to 65 grains. Then increase to 65 grains, etc.
With each group you should see the group size decrease. When you notice your group starting to expand, you have found your ideal load. If you want to, then you can work on other variable, like ball size, patch thickness, lubes, etc.
When you are satisfied with the load, then start working on adjusting your sights.
Pretty soon, you will be shooting the hat off of Captain Morgan's head.
John