I really think a lot of it comes down to just personal preference.
In my case, I started off decades ago shooting a .54 Renegade. Over the years, I have had a lot of different rifles and the .54 has always been what I liked best and what shot best for me.
I have tried a few different .50's and was not able to make them do anything. OK, I did have one .50 that shot well, but I figured that was the exception that made the rule. I have had a couple of .62's and they shot well, I had a .66 and it shot like a .50. I had a .69 and it was OK, but not great, but my .72 and I got along just fine. My 8 bore was also good, but I could never get proficient with the 4 bore.
And then there is the problem with some clubs not wanting you to shoot anything bigger than a .58.
So, yeah, the .54 is the rifle that works best for me. As long as I am casting round ball, it is affordable enough to shoot, it is not abusive to the shooter and it has the power to knock down steel silhouettes at 200 yards. I think that checks off all the boxes. But again, it is mostly personal preference. If you have a .50 and it works good for you, why change. I have seen a lot of shooters do really well with a .50 in woods walks.
If you have the opportunity to try some different rifles in various calibers, that's not a bad idea. And also, the design of the rifle has a bit to do with things also.
John (Bigsmoke)