I do this fairly frequently. If I run out of anything to build a starter I'll use extra runnings from the mash to build my starter, which means I can't pitch for a little while after brewing.
Before I built my immersion chiller I used to just rack it into a plastic bucket and throw it in my spare room until it reached pitching temp and I never had any problems. Don't put hot wort in a cold glass carboy though, i learned that the hard way...