This research presents a depth estimation method that is based on vanishing point and object classification for 3D display applications. We use some image processing techniques such as mean shift color segmentation, Hough transform, and Canny edge detection to complete depth estimation. The method can be divided into four steps. The first step is object segmentation. We use mean shift algorithm to perform color segmentation. Then the segmented regions are removed noise by the thresholding technique. The second step is sky region detection and foreground and background separation. We use the criteria of blue, white, gray, and red color to extract sky regions in HSV color space. The foreground and background regions are then segmented and the horizon line is captured. The third step is vanishing point detection. We use the object segmented image to obtain the vanishing point. We assume that the vanishing point is the farthest place in an image. Then we use the coordinate of vanishing point to find out the background depth profile information. The fourth step is object classification and depth estimation. We use four features to complete object classification, horizon line, image borders, texture complexity, and object size. There are four object classes, sky, ground, high-type, and wide-type object classes. Then we assign the depth information to each object region and create the depth map. In the experiment, we compute the error between the ground truth depth map and the depth map produced by our method and compare with the error between the ground truth depth map and the background depth profile map. Experimental results show that our proposed method is superior to other methods.