Polarization measurements done using Imaging Polarimeters such as the Robotic Polarimeter are very sensitive to the presence of artifacts in images. Artifacts can range from internal reflections in a telescope to satellite trails that could contaminate an area of interest in the image. With the advent of wide-field polarimetry surveys, it is imperative to develop methods that automatically flag artifacts in images. In this paper, we implement a Convolutional Neural Network to identify the most dominant artifacts in the images. We find that our model can successfully classify sources with 98% true positive and 97% true negative rates. Such models, combined with transfer learning, will give us a running start in artifact elimination for near-future surveys like the Wide Area Linear Optical Polarimeter (WALOP). This work was done at the Astronomy department of the California Institute of Technology in the US with Dr. Ashish Mahabal, Prof. Anthony Readhead, Prof. Kieran Cleary, Prof. A. N. Ramaprakash and Dr. Gina Panapoulou. This was a collaborative work with the PASIPHAE group which stands for Polar-Areas Stellar-Imaging in Polarization High-Accuracy Experiment. Pasiphae is an international collaboration between the University of Crete, Caltech, the South African Astronomical Observatory, the Inter-University Center for Astronomy and Astrophysics, and the University of Oslo. This experiment aims to map, with unprecedented accuracy, the polarization of millions of stars at areas of the sky away from the Galactic plane, in both the Northern and the Southern hemispheres.