Deep Neural Networks (DNNs) are widely used in automotive Cyber-Physical Systems (CPS) to implement au- tonomy related tasks. However, these networks have exhibited erroneous predictions to anomalous inputs that manifest either due to Out-of-Distribution (OOD) data or adversarial attacks. To detect these anomalies, a separate DNN called assurance monitor is used in parallel to the controller DNN, increasing the resource burden and latency. We hypothesize that a single network that can perform controller predictions and anomaly detection is necessary to reduce the resource requirements. Deep- Radial Basis Function (RBF) networks provide a rejection class alongside the class predictions, which can be used for anomaly detection. However, the use of RBF activation functions limits the applicability of these networks to only classification tasks. In this paper, we discuss the steps involved in detecting anomalies in CPS regression and classification tasks. Further, we design deep-RBF networks using popular DNNs such as NVIDIA DAVE- II and ResNet20 and then use the resulting rejection class for detecting physical and data poison adversarial attacks. We show that the deep-RBF network can effectively detect these attacks with limited resource requirements.