I've never had to tension the belt, but I think you can do it with the alternator. If your belt looks cracked at all, I'd just replace the belt. I don't think it'd be too expensive. If you replace it, it might be tighter than the belt you have now, and nothing may need to be tensioned.
Haven't done it myself though. I will have to tackle it eventually though, as I think my PS Pump is dying. Just not entirely sure how to determine if it is or if it's my box.