Distant and obscured supermassive black holes: discovering the missing AGN populations with current deep surveys and next-generation X-ray telescopes