A Gift from Knowledge Distillation #366

lansiyuan3 · 2023-06-13T03:07:53Z

lansiyuan3
Jun 13, 2023

HI @yoshitomo-matsubara ：
I'm looking for the code related to the paper A Gift from Knowledge Distillation, but I don't clear the location of the paper in your project. Or how you categorize these algorithms in general

yoshitomo-matsubara · 2023-06-13T03:19:20Z

yoshitomo-matsubara
Jun 13, 2023
Maintainer

Hi @lansiyuan3

With the current PyPI package (torchdistill==0.3.3), you can run it using this config file using this script

Note that you'll need to tune the hyperparameters as it's just a sample config (as explained here) because the original paper did not test the method for ImageNet dataset

24 replies

lansiyuan3 Aug 3, 2023
Author

First of all, I used thisconfig file and this script. All the experiments are based onthis
And then out of this error:
Files already downloaded and verified
2023/08/03 10:17:48 INFO torchdistill.common.main_util Getting ResNet34_Weights from torchvision.models.resnet
2023/08/03 10:17:48 INFO torchdistill.common.main_util Getting SequentialSampler from torch.utils.data
2023/08/03 10:17:48 INFO torchdistill.common.main_util Getting RandomSampler from torch.utils.data
2023/08/03 10:17:48 INFO torchdistill.common.main_util ckpt file path is None
2023/08/03 10:17:48 INFO torchdistill.common.main_util ckpt file path is None
2023/08/03 10:17:48 INFO main Start training
2023/08/03 10:17:50 INFO torchdistill.core.distillation [teacher model]
2023/08/03 10:17:50 INFO torchdistill.models.util Redesigning the teacher model with ['conv1', 'bn1', 'relu', 'maxpool', 'layer1', 'layer2', 'layer3', 'layer4']
2023/08/03 10:17:50 INFO torchdistill.core.distillation [student model]
2023/08/03 10:17:50 INFO torchdistill.models.util Redesigning the student model with ['conv1', 'bn1', 'relu', 'maxpool', 'layer1', 'layer2', 'layer3', 'layer4']
2023/08/03 10:17:50 INFO torchdistill.core.distillation Loss = 1.0 * FSPLoss()
2023/08/03 10:17:50 INFO torchdistill.core.distillation Freezing the whole teacher model
2023/08/03 10:17:50 INFO torchdistill.core.distillation Started stage 1
Traceback (most recent call last):
File "D:\Users\Miclan\PycharmProjects\pythonProject2\torchdistill-main\torchdistill-main\examples\torchvision\image_classification.py", line 213, in
main(argparser.parse_args())
File "D:\Users\Miclan\PycharmProjects\pythonProject2\torchdistill-main\torchdistill-main\examples\torchvision\image_classification.py", line 191, in main
train(teacher_model, student_model, dataset_dict, dst_ckpt_file_path,
File "D:\Users\Miclan\PycharmProjects\pythonProject2\torchdistill-main\torchdistill-main\examples\torchvision\image_classification.py", line 144, in train
train_one_epoch(training_box, device, epoch, log_freq)
File "D:\Users\Miclan\PycharmProjects\pythonProject2\torchdistill-main\torchdistill-main\examples\torchvision\image_classification.py", line 65, in train_one_epoch
for sample_batch, targets, supp_dict in
ValueError: not enough values to unpack (expected 3, got 2)
I don't know if this supp_dict returns any arguments，But it looks like loss = training_box.forward_process(sample_batch, targets, supp_dict) is needed

yoshitomo-matsubara Aug 3, 2023
Maintainer

Again, you are not using legacy scripts (examples/legacy/) with PyPI version of torchdistill v0.3.3.
Do pip3 install torchdistill, confirm torchdistill.__version__ returns 0.3.3 and use examples/legacy/image_classification.py.

training_box.forward_process is non-legacy implementation, which is not ready to be used right now

I've already emphasized this point at https://github.com/yoshitomo-matsubara/torchdistill/tree/main#important-notice , and you repeatedly ignore the message

lansiyuan3 Aug 3, 2023
Author

I have made some modifications according to your request, and I think this should be correct. about this script and yaml
I just want to experiment with whether this system can run and then add my own data loader, but I have found the following issues
C:\Users\Miclan\anaconda3\python.exe D:\torchdistill-0.3.3\torchdistill-main\torchdistill-main\examples\legacy\image_classification.py 2023/08/03 16:07:42 INFO torchdistill.common.main_util Not using distributed mode 2023/08/03 16:07:42 INFO __main__ Namespace(adjust_lr=False, config='D:\\torchdistill-0.3.3\\torchdistill-main\\torchdistill-main\\configs\\legacy\\sample\\cifar10\\kd\\resnet20_from_densenet_bc_k12_depth100-final_run.yaml', device='cuda', dist_url='env://', log=None, log_config=False, seed=None, start_epoch=0, student_only=False, test_only=False, world_size=1) 2023/08/03 16:07:42 INFO torchdistill.datasets.util Loading train data Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./resource/dataset/cifar10\cifar-10-python.tar.gz 100%|██████████| 170498071/170498071 [00:50<00:00, 3351126.91it/s] Extracting ./resource/dataset/cifar10\cifar-10-python.tar.gz to ./resource/dataset/cifar10 2023/08/03 16:08:40 INFO torchdistill.datasets.util dataset_id cifar10/train: 58.05756139755249 sec 2023/08/03 16:08:40 INFO torchdistill.datasets.util Loading val data Files already downloaded and verified 2023/08/03 16:08:41 INFO torchdistill.datasets.util dataset_id cifar10/val: 0.5656769275665283 sec 2023/08/03 16:08:41 INFO torchdistill.datasets.util Loading test data Files already downloaded and verified 2023/08/03 16:08:41 INFO torchdistill.datasets.util dataset_id cifar10/test: 0.5547194480895996 sec 2023/08/03 16:08:41 INFO torchdistill.common.main_util ckpt file is not found at ./resource/ckpt/cifar10/teacher/cifar10-densenet_bc_k12_depth100.pt2023/08/03 16:08:42 INFO torchdistill.common.main_util ckpt file is not found at./resource/ckpt/cifar10/kd/cifar10-resnet20_from_densenet_bc_k12_depth100-final_run.pt 2023/08/03 16:08:42 INFO __main__ Start training 2023/08/03 16:08:42 INFO torchdistill.models.util [teacher model] 2023/08/03 16:08:42 INFO torchdistill.models.util Using the original teacher model 2023/08/03 16:08:42 INFO torchdistill.models.util [student model] 2023/08/03 16:08:42 INFO torchdistill.models.util Using the original student model 2023/08/03 16:08:42 INFO torchdistill.core.distillation Loss = 1.0 * OrgLoss 2023/08/03 16:08:42 INFO torchdistill.core.distillation Freezing the whole teacher model Traceback (most recent call last): File "D:\torchdistill-0.3.3\torchdistill-main\torchdistill-main\examples\legacy\image_classification.py", line 188, in <module> main(argparser.parse_args()) File "D:\torchdistill-0.3.3\torchdistill-main\torchdistill-main\examples\legacy\image_classification.py", line 169, in main train(teacher_model, student_model, dataset_dict, ckpt_file_path, device, device_ids, distributed, config, args) File "D:\torchdistill-0.3.3\torchdistill-main\torchdistill-main\examples\legacy\image_classification.py", line 125, in train train_one_epoch(training_box, device, epoch, log_freq) File "D:\torchdistill-0.3.3\torchdistill-main\torchdistill-main\examples\legacy\image_classification.py", line 62, in train_one_epoch for sample_batch, targets, supp_dict in \ ValueError: not enough values to unpack (expected 3, got 2)
I discovered this issue while using your yaml file for training, which occurred in here
supp_dict cannot accept any parameters, but it is needed below

yoshitomo-matsubara Aug 3, 2023
Maintainer

No, I think you still use the code you copied from this repository and made some modifications.

Following

pip3 install torch==1.12.1 torchvision==0.13.1
pip3 install torchdistill
git clone https://github.com/yoshitomo-matsubara/torchdistill.git

I confirmed that the legacy script works for both training with/without teacher model with CIFAR-10 dataset

Training with teacher

python3 torchdistill/examples/legacy/image_classification.py --config torchdistill/configs/legacy/sample/cifar10/kd/resnet20_from_densenet_bc_k12_depth100-final_run.yaml --log log/cifar10/kd/resnet20_from_densenet_bc_k12_depth100-final_run.log

2023/08/03 15:46:14	INFO	torchdistill.common.main_util	Not using distributed mode
2023/08/03 15:46:14	INFO	__main__	Namespace(config='torchdistill/configs/legacy/sample/cifar10/kd/resnet20_from_densenet_bc_k12_depth100-final_run.yaml', device='cuda', log='log/cifar10/kd/resnet20_from_densenet_bc_k12_depth100-final_run.log', start_epoch=0, seed=None, test_only=False, student_only=False, log_config=False, world_size=1, dist_url='env://', adjust_lr=False)
2023/08/03 15:46:14	INFO	torchdistill.datasets.util	Loading train data
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./resource/dataset/cifar10/cifar-10-python.tar.gz
100% 170498071/170498071 [00:03<00:00, 48270607.07it/s]
Extracting ./resource/dataset/cifar10/cifar-10-python.tar.gz to ./resource/dataset/cifar10
2023/08/03 15:46:20	INFO	torchdistill.datasets.util	dataset_id `cifar10/train`: 6.466401100158691 sec
2023/08/03 15:46:20	INFO	torchdistill.datasets.util	Loading val data
Files already downloaded and verified
2023/08/03 15:46:21	INFO	torchdistill.datasets.util	dataset_id `cifar10/val`: 0.6924517154693604 sec
2023/08/03 15:46:21	INFO	torchdistill.datasets.util	Loading test data
Files already downloaded and verified
2023/08/03 15:46:21	INFO	torchdistill.datasets.util	dataset_id `cifar10/test`: 0.71073317527771 sec
Downloading: "https://github.com/yoshitomo-matsubara/torchdistill/releases/download/v0.1.1/cifar10-densenet_bc_k12_depth100.pt" to /root/.cache/torch/hub/checkpoints/cifar10-densenet_bc_k12_depth100.pt
100% 3.23M/3.23M [00:00<00:00, 85.1MB/s]
2023/08/03 15:46:22	INFO	torchdistill.common.main_util	ckpt file is not found at `./resource/ckpt/cifar10/teacher/cifar10-densenet_bc_k12_depth100.pt`
2023/08/03 15:46:25	INFO	torchdistill.common.main_util	ckpt file is not found at `./resource/ckpt/cifar10/kd/cifar10-resnet20_from_densenet_bc_k12_depth100-final_run.pt`
2023/08/03 15:46:25	INFO	__main__	Start training
/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
2023/08/03 15:46:25	INFO	torchdistill.models.util	[teacher model]
2023/08/03 15:46:25	INFO	torchdistill.models.util	Using the original teacher model
2023/08/03 15:46:25	INFO	torchdistill.models.util	[student model]
2023/08/03 15:46:25	INFO	torchdistill.models.util	Using the original student model
2023/08/03 15:46:25	INFO	torchdistill.core.distillation	Loss = 1.0 * OrgLoss
2023/08/03 15:46:25	INFO	torchdistill.core.distillation	Freezing the whole teacher model
2023/08/03 15:46:29	INFO	torchdistill.misc.log	Epoch: [0]  [  0/391]  eta: 0:28:49  lr: 0.1  img/s: 67.25317249488089  loss: 4.7402 (4.7402)  time: 4.4227  data: 2.5192  max mem: 700
2023/08/03 15:46:40	INFO	torchdistill.misc.log	Epoch: [0]  [100/391]  eta: 0:00:44  lr: 0.1  img/s: 1200.8439511673578  loss: 3.5644 (3.9403)  time: 0.1067  data: 0.0002  max mem: 700
2023/08/03 15:46:51	INFO	torchdistill.misc.log	Epoch: [0]  [200/391]  eta: 0:00:25  lr: 0.1  img/s: 1198.011563479755  loss: 3.4226 (3.7161)  time: 0.1069  data: 0.0002  max mem: 700
2023/08/03 15:47:03	INFO	torchdistill.misc.log	Epoch: [0]  [300/391]  eta: 0:00:11  lr: 0.1  img/s: 1117.7498719587604  loss: 3.1425 (3.5530)  time: 0.1173  data: 0.0036  max mem: 700
2023/08/03 15:47:13	INFO	torchdistill.misc.log	Epoch: [0] Total time: 0:00:48
2023/08/03 15:47:14	INFO	torchdistill.misc.log	Validation:  [ 0/79]  eta: 0:01:23  acc1: 42.1875 (42.1875)  acc5: 89.8438 (89.8438)  time: 1.0631  data: 1.0392  max mem: 700
2023/08/03 15:47:17	INFO	torchdistill.misc.log	Validation: Total time: 0:00:04
2023/08/03 15:47:17	INFO	__main__	 * Acc@1 47.4200	Acc@5 92.9200

2023/08/03 15:47:17	INFO	__main__	Best top-1 accuracy: 0.0000 -> 47.4200
2023/08/03 15:47:17	INFO	__main__	Updating ckpt at ./resource/ckpt/cifar10/kd/cifar10-resnet20_from_densenet_bc_k12_depth100-final_run.pt
2023/08/03 15:47:20	INFO	torchdistill.misc.log	Epoch: [1]  [  0/391]  eta: 0:16:32  lr: 0.1  img/s: 197.798169355944  loss: 2.8337 (2.8337)  time: 2.5374  data: 1.8900  max mem: 700
2023/08/03 15:47:31	INFO	torchdistill.misc.log	Epoch: [1]  [100/391]  eta: 0:00:39  lr: 0.1  img/s: 1180.7846782192917  loss: 2.5681 (2.7318)  time: 0.1092  data: 0.0002  max mem: 700

Training without teacher

python3 torchdistill/examples/legacy/image_classification.py --config torchdistill/configs/legacy/sample/cifar10/ce/resnet20-final_run.yaml --log log/cifar10/ce/resnet20-final_run.log

2023/08/03 15:38:45	INFO	torchdistill.common.main_util	Not using distributed mode
2023/08/03 15:38:45	INFO	__main__	Namespace(config='torchdistill/configs/legacy/sample/cifar10/ce/resnet20-final_run.yaml', device='cuda', log='log/cifar10/ce/resnet20-final_run.log', start_epoch=0, seed=None, test_only=False, student_only=False, log_config=False, world_size=1, dist_url='env://', adjust_lr=False)
2023/08/03 15:38:45	INFO	torchdistill.datasets.util	Loading train data
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./resource/dataset/cifar10/cifar-10-python.tar.gz
100% 170498071/170498071 [00:10<00:00, 16743363.48it/s]
Extracting ./resource/dataset/cifar10/cifar-10-python.tar.gz to ./resource/dataset/cifar10
2023/08/03 15:38:58	INFO	torchdistill.datasets.util	dataset_id `cifar10/train`: 13.498707056045532 sec
2023/08/03 15:38:58	INFO	torchdistill.datasets.util	Loading val data
Files already downloaded and verified
2023/08/03 15:38:59	INFO	torchdistill.datasets.util	dataset_id `cifar10/val`: 0.7275292873382568 sec
2023/08/03 15:38:59	INFO	torchdistill.datasets.util	Loading test data
Files already downloaded and verified
2023/08/03 15:39:00	INFO	torchdistill.datasets.util	dataset_id `cifar10/test`: 0.6866748332977295 sec
2023/08/03 15:39:00	INFO	torchdistill.common.main_util	ckpt file is not found at `./resource/ckpt/cifar10/ce/cifar10-resnet20-final_run.pt`
2023/08/03 15:39:02	INFO	__main__	Start training
/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
2023/08/03 15:39:02	INFO	torchdistill.models.util	[student model]
2023/08/03 15:39:02	INFO	torchdistill.models.util	Using the original student model
2023/08/03 15:39:02	INFO	torchdistill.core.training	Loss = 1.0 * OrgLoss
2023/08/03 15:39:05	INFO	torchdistill.misc.log	Epoch: [0]  [  0/391]  eta: 0:15:33  lr: 0.1  img/s: 115.78139357541274  loss: 2.6344 (2.6344)  time: 2.3876  data: 1.2818  max mem: 281
2023/08/03 15:39:10	INFO	torchdistill.misc.log	Epoch: [0]  [100/391]  eta: 0:00:22  lr: 0.1  img/s: 2267.8138509303653  loss: 1.7821 (1.9676)  time: 0.0710  data: 0.0058  max mem: 281
2023/08/03 15:39:17	INFO	torchdistill.misc.log	Epoch: [0]  [200/391]  eta: 0:00:14  lr: 0.1  img/s: 3803.279342589969  loss: 1.6063 (1.8255)  time: 0.0547  data: 0.0080  max mem: 281
2023/08/03 15:39:22	INFO	torchdistill.misc.log	Epoch: [0]  [300/391]  eta: 0:00:05  lr: 0.1  img/s: 3112.6740762643567  loss: 1.4312 (1.7326)  time: 0.0511  data: 0.0057  max mem: 281
2023/08/03 15:39:26	INFO	torchdistill.misc.log	Epoch: [0] Total time: 0:00:23
2023/08/03 15:39:28	INFO	torchdistill.misc.log	Validation:  [ 0/79]  eta: 0:01:49  acc1: 59.3750 (59.3750)  acc5: 93.7500 (93.7500)  time: 1.3893  data: 1.3608  max mem: 281
2023/08/03 15:39:31	INFO	torchdistill.misc.log	Validation: Total time: 0:00:05
2023/08/03 15:39:31	INFO	__main__	 * Acc@1 49.5600	Acc@5 94.1200

2023/08/03 15:39:31	INFO	__main__	Best top-1 accuracy: 0.0000 -> 49.5600
2023/08/03 15:39:31	INFO	__main__	Updating ckpt at ./resource/ckpt/cifar10/ce/cifar10-resnet20-final_run.pt
2023/08/03 15:39:33	INFO	torchdistill.misc.log	Epoch: [1]  [  0/391]  eta: 0:10:23  lr: 0.1  img/s: 943.0203439251023  loss: 1.2175 (1.2175)  time: 1.5934  data: 1.4574  max mem: 281
2023/08/03 15:39:39	INFO	torchdistill.misc.log	Epoch: [1]  [100/391]  eta: 0:00:21  lr: 0.1  img/s: 2045.4562883377148  loss: 1.2161 (1.2695)  time: 0.0517  data: 0.0050  max mem: 281
2023/08/03 15:39:44	INFO	torchdistill.misc.log	Epoch: [1]  [200/391]  eta: 0:00:12  lr: 0.1  img/s: 2955.1659694394293  loss: 1.1407 (1.2134)  time: 0.0759  data: 0.0053  max mem: 281
2023/08/03 15:39:52	INFO	torchdistill.misc.log	Epoch: [1]  [300/391]  eta: 0:00:06  lr: 0.1  img/s: 2962.160811727901  loss: 1.0447 (1.1683)  time: 0.0503  data: 0.0063  max mem: 281
2023/08/03 15:39:56	INFO	torchdistill.misc.log	Epoch: [1] Total time: 0:00:24
2023/08/03 15:39:57	INFO	torchdistill.misc.log	Validation:  [ 0/79]  eta: 0:01:28  acc1: 61.7188 (61.7188)  acc5: 99.2188 (99.2188)  time: 1.1196  data: 1.0868  max mem: 281
2023/08/03 15:39:59	INFO	torchdistill.misc.log	Validation: Total time: 0:00:03
2023/08/03 15:39:59	INFO	__main__	 * Acc@1 63.4100	Acc@5 96.4500

2023/08/03 15:39:59	INFO	__main__	Best top-1 accuracy: 49.5600 -> 63.4100
2023/08/03 15:39:59	INFO	__main__	Updating ckpt at ./resource/ckpt/cifar10/ce/cifar10-resnet20-final_run.pt
2023/08/03 15:40:02	INFO	torchdistill.misc.log	Epoch: [2]  [  0/391]  eta: 0:17:13  lr: 0.1  img/s: 959.3044746143098  loss: 0.9188 (0.9188)  time: 2.6443  data: 2.5106  max mem: 281

I have made some modifications according to your request, and I think this should be correct. about this script and yaml
I just want to experiment with whether this system can run and then add my own data loader, but I have found the following issues

Why don't you describe what modifications you made exactly? You just keep saying you made modifications, but never explain what modifications you made and how or differences from the legacy script in this repo either. You didn't even confirm that you used torchdistill=v0.3.3. I am not your personal tutor, and If you refuse to do those, I can not answer your questions anymore.

lansiyuan3 Aug 4, 2023
Author

Sorry to bother you, but I think I have figured out what the problem is and sincerely apologize for my low-level mistake

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A Gift from Knowledge Distillation #366

{{title}}

Replies: 1 comment 24 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

A Gift from Knowledge Distillation #366

lansiyuan3 Jun 13, 2023

Replies: 1 comment · 24 replies

yoshitomo-matsubara Jun 13, 2023 Maintainer

lansiyuan3 Aug 3, 2023 Author

yoshitomo-matsubara Aug 3, 2023 Maintainer

lansiyuan3 Aug 3, 2023 Author

yoshitomo-matsubara Aug 3, 2023 Maintainer

Training with teacher

Training without teacher

lansiyuan3 Aug 4, 2023 Author

lansiyuan3
Jun 13, 2023

Replies: 1 comment 24 replies

yoshitomo-matsubara
Jun 13, 2023
Maintainer

lansiyuan3 Aug 3, 2023
Author

yoshitomo-matsubara Aug 3, 2023
Maintainer

lansiyuan3 Aug 3, 2023
Author

yoshitomo-matsubara Aug 3, 2023
Maintainer

lansiyuan3 Aug 4, 2023
Author