Finding similar and relevant media content given a user query or sample image has been at the core of the multimedia retrieval community for a long time. In this talk, I will identify and address multimedia challenges that play a role at Yahoo!, and which go beyond relevancy of images and video to a given multimedia retrieval task. The true challenge for multimedia is to find a balance between relevancy, freshness, quality, interestingness and diversity in order to provide an engaging rich media experience to the user.