Mark, I have had that happen as well. Best thing to do (IMHO) is to leave the liner hole alone for now and get in the habit of, "pick and prime."
With this said, and with the accuracy you are now getting,,, you should be able to read your groups and know when it is time for a new liner in the future when the hole begins to enlarge. However, I have stainless steel liners in my flinters and I have not shot one out as yet - at least that I can tell. I also quit enlarging the hole.
Another option would be to shoot a heavier charge with the enlarged hole. But if you're getting 1 to 1 1/2 inch groupings at 100 yards from the bench, then it's awful tempting to "not" mess with your load.