combine two loss

[liguopeng0923]

Hi @yoshitomo-matsubara ，

I cannot find the file of combining two losses in your legacy folder. For example, what should I do when I want to train the models with cross entropy loss and KD loss in your new manner (\ie, legacy of Important Notice).

[yoshitomo-matsubara]

Hi @liguopeng0923

Please use Q&A category for questions. (I changed the category here and the previous discussion)

There are many legacy config files that use a linear combination of multiple losses.
e.g. https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/legacy/sample/ilsvrc2012/single_stage/ghnd/custom_resnet152_from_resnet152.yaml#L104-L149
https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/legacy/sample/coco2017/multi_stage/ktaad/lraspp_mobilenet_v3_large_from_deeplabv3_resnet50.yaml#L228-L247

The KD loss that combines cross entropy and KL divergence is defined as KDLoss in torchdistill
e.g.,
https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/legacy/sample/ilsvrc2012/single_stage/kd/alexnet_from_resnet152.yaml#L95-L99

Note: Please read README carefully for types of config files

[liguopeng0923]

I get it. Thank you! But your examples are different types of losses (e.g. features). I want to combine multiple losses only according to outputs, not features.

[yoshitomo-matsubara]

Technically you can do that by extracting the last layer's output as a "feature"

[liguopeng0923]

ok，thanks!

[yoshitomo-matsubara]

When you close QA, please mark a reply as answer so that others can have better search experience if they have similar questions

[liguopeng0923]

And why we get a better result by appending a crossentropy loss in configs/sample/ilsvrc2012/kd/resnet18_from_resnet34.yaml

[yoshitomo-matsubara]

I don't understand the question.

Also, please keep discussion in one thread as I requested before #411 (reply in thread)

[liguopeng0923]

Sorry for that. I ignored that before. I have no problem. Great work!