With the increasing demands of visual surveillance systems, human identification at a distance has gained more attention from the researchers recently. Gait analysis can be used as an unobtrusive biometric measure to identify people at a distance without any attention of the human subjects. We propose a novel effective method for both automatic viewpoint and person identification by using only the silhouette sequence of the gait. The gait silhouettes are nonlinearly transformed into low-dimensional embedding by Gaussian process latent variable model (GP-LVM), and the temporal dynamics of the gait sequences are modeled by hidden Markov models (HMMs). The experimental results show that our method has higher recognition rate than the other methods.