Canon refurbished cameras and lenses are often actually new, unused items which for legal reasons cannot be sold as new. They may have been purchased and returned because a customer changed their mind. Most retailers have a 14-30 day return policy "no questions asked". Refurbished cameras and lenses may also be overstock items returned to the manufacturer by a retailer. They may be items in a box that suffered damage in shipping. It's also possible that refurbished cameras or lenses had some minor electrical or cosmetic defect on delivery which has been repaired by Canon. In all cases refurbished items are inspected and restored or repaired (if necessary) and certified to fully meet the specifications of new items. Refurbished Canon cameras and lenses generally have a 90 day manufacturers warranty rather then the typical 365 day warranty provided on cameras and lenses sold as new. They generally come in a different box (a white box) to the items sold as new. I guess this prevents retailers selling then as new!
Refurbished cameras and lenses aren't old and broken lenses that have been sent back for repair because of problems after extensive use. Those would be repaired under warranty and returned to the original purchaser.
So I don't personally have any hesitation in purchasing a refurbished camera or lens - as long as it was refurbished by the original manufacturer - and anything bought directly from Canon was obviously refurbished by their own service department.
Lots of major retailers have a few refurbished items for sale. However Canon also sell refurbished cameras and lenses directly from their online store. Much of the time quite a few of the items they list are out of stock because obviously the number of items available to be sold refurbished is somewhat out of their direct control. However if you are interested in purchasing a refurbished camera or lens, it's worth checking their store to see what's currently in stock.
Here are some refurbished items that seem to be in stock at the Canon Store as of 03/27/12 and which are significantly cheaper than new equivalents sold by the major discount vendors such as Amazon, Adorama and B&H:
If Canon's direct store had the EOS 5D Mark II Body Refurbished in stock today it would be $1759.20, but unfortunately it's out of stock as of 03/27/12 - UPDATE - 04/23/12 - The refurbished 5D MkII is back in stock. Don't know for how long though. UPDATE #2 - Well, it was about 2 days. As of 4/25/12 it's out of stock again. If you want one and it's in stock, don't wait!