A robust adaptive predictor is proposed to solve the time-varying and delayed control problem of an overhead crane system with stereo-vision servo. The predictor is based on the use a recurrent neural network (RNN) with tapped delays, and is used to supply the real-time signal of swing angle. There are two types of two discrete-time controllers under investigation: the proportional-integral-derivative (PID) controller and the sliding controller. First, a design principle of the neural predictor is developed to guarantee the convergence of its swing angle estimation. Next, an improved version of the Particle Swarm Optimization algorithm, the parallel particle swarm optimization (PPSO) method, is used to optimize the control parameters of these two types of controllers. Finally, a homemade overhead crane system equipped with the Kinect sensor for visual servo is used to verify the proposed scheme. Experimental results successfully demonstrate effectiveness of the approach, which also show the parameter convergence in the predictor.