WLAN has grown in popularity recently according to the characteristics of wireless access and mobility. Voice applications on WLAN usually require to access network resource un-interruptedly. To keep high quality of communication for mobile users employing these applications, WLAN system must provide the ability of fast handoff across different AP. However, the WPA authentication requires much time such that WLAN is difficult to support the fast handoff. In this paper, we propose a secure accelerated authentication scheme based on refined neighbor graph. Through key pre-agreement between each STA and neighbor APs in advance and the reduction of handover authentication process, our protocol can reduce greatly authentication latency of handoff. Moreover, a refined neighbor graph can only selects the certain of APs that STA most likely roams into further to anticipate in key pre-generation. Consequently, our scheme can minimize efficiently network traffic load that results from key pre-agreement while providing the fast authentication of handoff.