Face recognition under pose and facial expression variations is a challenging problem. In this paper, we present a method for pose and expression invariant face recognition using only a single sample image per person. The method utilizes the similarities of a face image against a set of faces from a prototype set taken at the same pose and expression to establish pose and expression invariant similarity vector which can be used for comparing face images of a person taken in different poses and facial expressions. Experimental results indicate that the proposed method achieves high recognition rate even for large pose and expression variations.