The Professor: Supervising Model Predictions

The tests so far performed have exposed some model limitations. The most natural way to eliminate these would be to fix the model itself. Firstly, keeping in mind that a model reflects the data, we could augment a dataset, and add examples that the model has difficulties with predicting correctly. Since manual labeling is time-consuming, we might think about generating examples. Nonetheless, models adapt quickly to fixed structures, so the benefits might well be minimal. Secondly, of course, we could propose a new model architecture. This is an appealing direction from the perspective of a researcher, however, profits from the findings are uncertain.

Due to problematic model fixing, unknown model reasoning, and limited decision control, we have abandoned the idea of using a standard pipeline that relies exclusively on a single end-to-end model. Instead, before making a final decision, we have fortified the pipeline with a reviewing process. We have introduced a distinct component – the professor – to manage the reviewing process. The professor reviews the model’s hidden states and outputs, to both identify and correct suspicious predictions. It is composed of either simple or complex auxiliary models that examine the model reasoning and correct any model weaknesses. Because the professor takes into account information from aux. models in order to make the final decision, we have greater control over the decision-making process, and we can freely customize the model behavior.