Generally speaking:
Round barrels were cheaper to make since less metal was needed. Labor itself was cheap.
Octagon barrels were found on rifles. More weight out front = steadier hold.
18th century rifles' barrels were tapered and flared ("swamped") for better balance. After 1800 or so, barrels got shorter and became either straight or tapered.
Military guns had round barrels because of cost (see above) and because it was MUCH easier to fit a bayonet to a round barrel.
Some guns (fowling pieces, trade guns, etc) had barrels that were octagon at the breech end and transitioned to round 9-11" or so down the barrel. This helped put more metal around the breech to aid in strength, but allowed the front end of the gun to stay lively for shooting at moving targets.
Mario