Good advice there from Riley. FWIW, on the occasions I shoot a .54, I typically use an 85 grain charge of Ffg.
If you are having issues with fouling, I have found that separating the powder charge from the PRB with a lubed shotgun cushion wad works wonders. When you load the wad it does two things, it first pushes whatever fouling is in the barrel down with it, and also lubricates the bore to help keep the fouling from the next shot softer. As I recall, a 28 gauge wad works pretty good in a .54.
When I had my .72 cal rifle, I was running 200 grain charges in a barrel that had rifling about .006 deep. Using a 12 gauge cushion wad I could fire as many shots as I wanted to without cleaning and they all loaded easily and shot well.
John