Regression is a widely used statistical tool to discover associations between variables. The estimated relationship can be further utilized for predicting new observations. Obtaining reliable prediction outcomes is a challenging task. When building a regression model, several difficulties such as high dimensionality in predictors, non-linearity of the associations and the unreliable results caused by outliers could deteriorate the results. Furthermore, the prediction error increases if the newly acquired data might not be processed carefully. In this dissertation, we aim at improving prediction performance by enhancing the model robustness at the training stage and duly handling the query data at the testing stage. We propose two methods to build robust models. One focuses on adopting a parsimonious model to limit the number of parameters and a refinement technique to enhance model robustness. We design the procedure to be carried out on parallel systems and further extend their abilities of handling complex and large-scale datasets. The other method restricts the parameter space to avoid the singularity issue and takes up the trimming techniques to limit the influence of outlying observations. We build both approaches by using the mixture-modeling principle to accommodating data heterogeneity without uncontrollably increasing model complexity. Both methods show their abilities to improve prediction performance, compared to existing approaches, in applications such as magnetic resonance vascular fingerprinting and source separation in single-channel polyphonic music, among others. To evaluate model robustness, we develop an efficient approach to generating adversarial samples, which could induce large prediction errors yet are difficult to detect visually. Finally, we propose a preprocessing system to detect and repair different kinds of abnormal testing samples for prediction efficacy, when testing samples are either corrupted or adversarially perturbed.