That's a fair amount of sweet orange peel (I'm assuming you mean the dried...?)... I don't think all the flavor will be out of your beer. The later you add it in the boil, the better, but 5 minutes is a very reasonable time, assuming your wort was cooled fairly quickly.
Fermentation doesn't scrub out flavor/aroma as much as you'd think. Late hop/spice additions can still add quite a bit of both, depending on the beer.