Many images suffer from interference by impulse noise, which can cause significant deterioration to the quality of an image. Efficient removal of the interference in a noise corrupted image is thus an important task for digital image signal processing. In this study we propose an efficient new approach to remove impulse noise by using a gain masker, which is adapted by switching variance and edge direction in a local window. Multiplying this gain masker with the neighbors of a corrupted center pixel, the impulse noise is then removed; meanwhile, the edge details of an image are well restored. Conversely, if the center pixel of a local window is classified as noise-free, this center pixel is kept unchanged to maintain image quality. Experimental results show that the proposed approach can efficiently remove salt-and-pepper noise from an image corrupted by various noise densities, ranging from 10% to 90%.