There's really no need to filter the break or hops out before transferring to your fermenter. If you want to filter the beer, the best time would be when you're kegging it. To answer your question...the yeast don't really need that break at all. The wort itself has everything the yeast needs for healthy fermentation except oxygen and zinc.
In "How to Brew", Palmer says that the break material can lead to off-flavours, but he doesn't back that up with anything, and I've not noticed anything myself. In "Yeast", I wasn't able to find any reference to break material, and how it affects yeast and fermentation.
Filter if you like, but I suspect if you're using pellet hops, you'll have a difficult time. If you do try this, I'd be interested in hearing about the results.