it could be an issue with your eyes, or it could be a barrel issue, or the wind, or the light. Back when I was a kid building my first rifle, one of the local club members loaned me a TC Hawken so I'd have something to shoot in the monthly blanket matches over the summer while building my rifle. With it, I found that if I shot five shots fast (roughly 5 shots in 4 minutes, like you sometimes have to do to beat wind conditions) on a hot day, the shots would climb to one side (also left).
Over the years since I've seen a few others do the same. With the rifle Butch loaned me, 5 shots in 4 minutes would usually go about 3 1/2"-4" at 50 yards. If I waited a couple-three minutes between shots, I could cut the group size in half. I suspect that the barrel were tweaked a bit when they were breeched, but this is just a hunch.
Another possibility is that it would be your patch lube. What I mean by this is that maybe it is taking 3 shots for the fouling to build up to a "status" level that is consistent shot-to-shot. This could result in 2 groups like you seem to be running into. Back when I was switching back and forth between neatsfoot oil and olive oil, I had this happen a couple times with olive oil--only in one particular gun, mind you. It never happened with neatsfoot oil, and since most rifles would group subsequent shots from a fouled bore around the first shot from a clean bore, I've stuck with neatsfoot oil ever since.
This would be easy enough to test. Shoot a ten shot group at your normal cadence, and record POI of each shot.