PSP - portable console, good for portable gaming which is hardcore (DS is more casual). Looking at the specs, the PSP is nowhere near as good as the PS3 (after all it IS a portable console).
PS3 - currently the most technically advanced console out there (some would argue the Kinect is), and includes a blu-ray player. Generally has much better games like Call of Duty and Battlefield.
PS3 is more expensive (I think about double the PSP's price) but is worth it.
PSP's successor will be released soon (Vita) so the PSP's game market will significantly deteriorate.
PSP is quite easy to softmod, PS3 requires a hardmod which costs money.
I would recommend the PS3 - the PSP is dying - and the PS3 still is an excellent console and receives the latest games in high definition.
Hope this helped - let me know if you need more info