yrenard wrote:
I'd like to point out two issues and propose one alternative solution here :
- Computing TP/FP supposes 2-class classifiers. Even if OpenViBE does not come with multi class classifiers for now, the classifier trainer box has been designed to handle multiple class classifiers and some users may use such classifiers making this TP/FP computation irrelevant
Actually, it can be used with a multiclass-classifier, if one class or a set of class represent a "target". For instance, for a multiclass-asynchronous experiment, let say with 3 classes : left hand motor imagery, right hand motor imagery and rest (or non-control state). In this case it is relevant to compute TP/FP, to estimate how good the asynchronous part (detection of commands) is. The target there would be both the left and right hand motor imagery (i.e., the control state). But I agree, it is not relevant for all type of classification problems. For a synchronous multi-class case, it is indeed irrelevant. Maybe we could compute and display the TP/FP only if required by the user, i.e., only if a target class (or list of target class if we want to consider the multiclass case) is specified? Or, to make things even simpler, we can compute and display the TP/FP only for two-class cases (since the box is aware of the number of classes). What do you think?
yrenard wrote:
[*] Adding a new setting to this box to choose which class is the target breaks the compatibility with existing scenarios, and even causes unpredictable behavior for scenarios that use classification algorithms which already add new settings (such as SVM for instance)[/list]
That could be a problem indeed, I didn't though about that. However, generally (and independently from this problem), I think we should dare to break the compatibility with existing scenarios from times to times (which can be made smoothly if we propose both a deprecated version of the modified box and the new version of the box for a while, to give time to users to switch to the new box) if it makes openvibe better. But it might not be necessary here. Otherwise, If we really want to avoid adding a new setting, we could compute and display the TP/FP only for binary problems, and for each class, i.e., using each of the two class as the target class. In this way we don't need to specify the target class.
yrenard wrote:
In my opinion, the best approach would be to compute the confusion matrix for each partition and print this to the Trace log level for interested users. The confusion matrix gives all the necessary information to know how good a classifier is.
I agree, the confusion matrix gives all the necessary information and should be displayed all the time, together with the accuracy (again, the accuracy is meaningful only for balanced problems, but is completely irrelevant for non-balanced ones. As such it should not be the only measure displayed). But it would be even more convenient for the user to also have aggregated measures ( such as TP, FP, precision, recall, etc.) when it is relevant, so that he won't have to compute them himself. Wouldn't it?
yrenard wrote:
More generally speaking, it is important to keep the boxes generic and not to make them specific to a particular problem in order to be able to re-use them later in different problems
Totally agree! And that was actually the initial problem here, since accuracy is a not a generic measure as it is meaningless for non-balanced problems. So that's good we have this discussion to find the best way to deal with that. Let's keep brainstorming! And thanks for your input Yann!