Abstract: Most of existing computer aided diagnosis (CAD) systems follow a rigid paradigm where the classifier's decision function is optimized during the training phase, and fixed during the test phase. These systems are often perceived as unfriendly as they do not allow clinicians to provide input. They are also unable to cope with the perpetual changes in data distribution caused by different sensing technologies, imaging protocols, and patient populations. To address these shortcomings, this paper proposes a novel CAD model capable of incorporating expert domain knowledge in real-time to improve its decision function. When the data distribution changes, our classification accuracy on lung nodule data remains above 90\% while popular deep networks' accuracies reduce to chance level. We demonstrate that the order of feedback samples affects the final accuracy. An information-gain sorting mechanism is proposed to compute an optimal order of feedback samples. We provide extensive experimental results on two lung nodule datasets to demonstrate that the proposed approach is promising for building a more reliable and radiologist-friendly CAD system.