-
Notifications
You must be signed in to change notification settings - Fork 4
/
HCQ_LSMDC_L15.txt
2581 lines (2581 loc) · 189 KB
/
HCQ_LSMDC_L15.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15
Preparing the dataloaders ...
Loading dataset LSMDC_full_trainval in ram ...
Finish loading dataset LSMDC_full_trainval in ram, taking 3034.4266057014465 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 20.605026960372925 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 12.207675695419312 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch0.pth ...
Done in 1.403s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch0.pth ...
Done in 2.791s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
LSMDC_full_test/t2v_metrics/R1: 0.1
LSMDC_full_test/t2v_metrics/R5: 0.4
LSMDC_full_test/t2v_metrics/R10: 1.0
LSMDC_full_test/t2v_metrics/R50: 5.0
LSMDC_full_test/t2v_metrics/MedR: 507.5
LSMDC_full_test/t2v_metrics/MeanR: 503.113
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3419951893353394
LSMDC_full_test/v2t_metrics/R1: 0.0
LSMDC_full_test/v2t_metrics/R5: 0.4
LSMDC_full_test/v2t_metrics/R10: 0.8
LSMDC_full_test/v2t_metrics/R50: 4.9
LSMDC_full_test/v2t_metrics/MedR: 502.0
LSMDC_full_test/v2t_metrics/MeanR: 501.174
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.3419951893353394
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 30.02033 (QuantReg: 22.58315) QuantErr: 22.58315 batch_time=12.60700
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 28.79362 (QuantReg: 22.55443) QuantErr: 22.55443 batch_time=0.61300
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 27.06283 (QuantReg: 22.60485) QuantErr: 22.60485 batch_time=0.60132
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 25.57468 (QuantReg: 22.57915) QuantErr: 22.57915 batch_time=0.59912
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 23.66705 (QuantReg: 22.67790) QuantErr: 22.67790 batch_time=0.60067
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 24.16905 (QuantReg: 22.65990) QuantErr: 22.65990 batch_time=0.61120
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 23.28362 (QuantReg: 22.65965) QuantErr: 22.65965 batch_time=0.71622
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 21.58935 (QuantReg: 22.62969) QuantErr: 22.62969 batch_time=0.60445
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 21.79805 (QuantReg: 22.64266) QuantErr: 22.64266 batch_time=0.59220
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 21.24980 (QuantReg: 22.66575) QuantErr: 22.66575 batch_time=0.60704
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 20.97132 (QuantReg: 22.65705) QuantErr: 22.65705 batch_time=0.60671
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 20.97261 (QuantReg: 22.67165) QuantErr: 22.67165 batch_time=0.61113
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 20.60770 (QuantReg: 22.65073) QuantErr: 22.65073 batch_time=1.13039
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 18.88217 (QuantReg: 22.65574) QuantErr: 22.65574 batch_time=1.05986
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 21.36782 (QuantReg: 22.64678) QuantErr: 22.64678 batch_time=0.61462
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 20.75636 (QuantReg: 22.61313) QuantErr: 22.61313 batch_time=0.62615
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 20.31746 (QuantReg: 22.65374) QuantErr: 22.65374 batch_time=0.59760
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 19.05165 (QuantReg: 22.68221) QuantErr: 22.68221 batch_time=0.60733
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 19.85156 (QuantReg: 22.68022) QuantErr: 22.68022 batch_time=0.61493
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 18.29740 (QuantReg: 22.65934) QuantErr: 22.65934 batch_time=2.22822
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 20.54351 (QuantReg: 22.65409) QuantErr: 22.65409 batch_time=0.60221
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 18.74141 (QuantReg: 22.67156) QuantErr: 22.67156 batch_time=0.62261
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 19.72237 (QuantReg: 22.66283) QuantErr: 22.66283 batch_time=0.62462
Train Epoch: 1 codebook_update_time=3.70383
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch1.pth ...
Done in 4.219s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch1.pth ...
Done in 8.043s
epoch : 1
loss : 21.857082893371583
quant_reg : 22.645687774658203
quant_err : 22.645687774658203
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
LSMDC_full_test/t2v_metrics/R1: 8.1
LSMDC_full_test/t2v_metrics/R5: 18.6
LSMDC_full_test/t2v_metrics/R10: 26.4
LSMDC_full_test/t2v_metrics/R50: 55.7
LSMDC_full_test/t2v_metrics/MedR: 37.0
LSMDC_full_test/t2v_metrics/MeanR: 99.792
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 15.844089853009011
LSMDC_full_test/v2t_metrics/R1: 7.0
LSMDC_full_test/v2t_metrics/R5: 19.3
LSMDC_full_test/v2t_metrics/R10: 27.1
LSMDC_full_test/v2t_metrics/R50: 56.4
LSMDC_full_test/v2t_metrics/MedR: 38.0
LSMDC_full_test/v2t_metrics/MeanR: 101.976
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 15.412563537547877
mnt_best : 15.844089853009011
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 19.84324 (QuantReg: 10.07362) QuantErr: 10.07362 batch_time=14.90569
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 19.32517 (QuantReg: 10.75407) QuantErr: 10.75407 batch_time=0.59524
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 17.44719 (QuantReg: 10.61300) QuantErr: 10.61300 batch_time=0.59313
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 19.89004 (QuantReg: 10.62928) QuantErr: 10.62928 batch_time=0.60890
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 19.07211 (QuantReg: 10.84240) QuantErr: 10.84240 batch_time=0.59211
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 18.13961 (QuantReg: 11.21694) QuantErr: 11.21694 batch_time=0.71144
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 18.68906 (QuantReg: 11.56114) QuantErr: 11.56114 batch_time=3.56899
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 18.34225 (QuantReg: 11.63389) QuantErr: 11.63389 batch_time=0.67172
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 16.91823 (QuantReg: 11.88288) QuantErr: 11.88288 batch_time=0.68101
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 17.47526 (QuantReg: 11.92499) QuantErr: 11.92499 batch_time=0.68004
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 17.08789 (QuantReg: 11.86138) QuantErr: 11.86138 batch_time=0.68433
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 16.46786 (QuantReg: 12.17782) QuantErr: 12.17782 batch_time=1.18291
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 18.88050 (QuantReg: 12.08986) QuantErr: 12.08986 batch_time=0.62020
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 18.51267 (QuantReg: 12.58485) QuantErr: 12.58485 batch_time=0.65308
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 16.40243 (QuantReg: 12.16273) QuantErr: 12.16273 batch_time=0.61900
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 18.42101 (QuantReg: 12.67813) QuantErr: 12.67813 batch_time=0.65433
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 17.41228 (QuantReg: 12.96694) QuantErr: 12.96694 batch_time=0.71770
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 17.12171 (QuantReg: 12.64137) QuantErr: 12.64137 batch_time=0.62077
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 16.30536 (QuantReg: 13.45646) QuantErr: 13.45646 batch_time=0.62808
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 16.94847 (QuantReg: 13.16302) QuantErr: 13.16302 batch_time=0.62548
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 17.08389 (QuantReg: 12.88763) QuantErr: 12.88763 batch_time=0.60713
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 17.16109 (QuantReg: 13.06465) QuantErr: 13.06465 batch_time=0.61670
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 16.37511 (QuantReg: 13.37994) QuantErr: 13.37994 batch_time=0.72046
Train Epoch: 2 codebook_update_time=3.61679
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch2.pth ...
Done in 5.412s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch2.pth ...
Done in 10.381s
removing stale ckpt [epoch 1] [took 0.04s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 17.86625744628906
quant_reg : 12.045957820892333
quant_err : 12.045957820892333
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
LSMDC_full_test/t2v_metrics/R1: 8.1
LSMDC_full_test/t2v_metrics/R5: 21.5
LSMDC_full_test/t2v_metrics/R10: 30.0
LSMDC_full_test/t2v_metrics/R50: 60.7
LSMDC_full_test/t2v_metrics/MedR: 32.0
LSMDC_full_test/t2v_metrics/MeanR: 87.669
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 17.35194827897326
LSMDC_full_test/v2t_metrics/R1: 7.7
LSMDC_full_test/v2t_metrics/R5: 22.2
LSMDC_full_test/v2t_metrics/R10: 31.2
LSMDC_full_test/v2t_metrics/R50: 59.7
LSMDC_full_test/v2t_metrics/MedR: 31.0
LSMDC_full_test/v2t_metrics/MeanR: 84.663
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 17.471603470854273
mnt_best : 17.35194827897326
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 17.03244 (QuantReg: 10.76640) QuantErr: 10.76640 batch_time=15.48008
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 15.69376 (QuantReg: 11.73874) QuantErr: 11.73874 batch_time=0.61964
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 15.14131 (QuantReg: 11.58997) QuantErr: 11.58997 batch_time=0.60807
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 16.71890 (QuantReg: 11.52647) QuantErr: 11.52647 batch_time=0.60527
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 16.54562 (QuantReg: 12.13310) QuantErr: 12.13310 batch_time=0.60531
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 16.34896 (QuantReg: 11.73885) QuantErr: 11.73885 batch_time=0.60657
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 16.08957 (QuantReg: 11.37755) QuantErr: 11.37755 batch_time=0.62120
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 16.67953 (QuantReg: 12.11074) QuantErr: 12.11074 batch_time=0.61115
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 15.90207 (QuantReg: 11.84967) QuantErr: 11.84967 batch_time=0.60727
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 15.56297 (QuantReg: 11.74323) QuantErr: 11.74323 batch_time=0.60764
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 15.29020 (QuantReg: 11.86427) QuantErr: 11.86427 batch_time=0.60481
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 16.08958 (QuantReg: 12.23417) QuantErr: 12.23417 batch_time=0.63860
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 16.54810 (QuantReg: 11.74092) QuantErr: 11.74092 batch_time=0.61927
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 18.60180 (QuantReg: 12.05889) QuantErr: 12.05889 batch_time=0.64283
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 15.23124 (QuantReg: 12.01852) QuantErr: 12.01852 batch_time=0.60450
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 16.37903 (QuantReg: 12.13189) QuantErr: 12.13189 batch_time=0.60083
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 16.54672 (QuantReg: 12.33619) QuantErr: 12.33619 batch_time=0.61085
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 17.02144 (QuantReg: 12.51968) QuantErr: 12.51968 batch_time=0.82983
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 15.24991 (QuantReg: 12.75379) QuantErr: 12.75379 batch_time=0.60484
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 17.06058 (QuantReg: 12.80133) QuantErr: 12.80133 batch_time=0.60948
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 15.40405 (QuantReg: 12.56951) QuantErr: 12.56951 batch_time=0.60881
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 15.18246 (QuantReg: 12.60635) QuantErr: 12.60635 batch_time=0.71551
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 15.11765 (QuantReg: 12.73829) QuantErr: 12.73829 batch_time=0.61552
Train Epoch: 3 codebook_update_time=3.32349
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch3.pth ...
Done in 4.300s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch3.pth ...
Done in 10.574s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 16.228526695251464
quant_reg : 12.037349704742432
quant_err : 12.037349704742432
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
LSMDC_full_test/t2v_metrics/R1: 8.4
LSMDC_full_test/t2v_metrics/R5: 25.1
LSMDC_full_test/t2v_metrics/R10: 33.7
LSMDC_full_test/t2v_metrics/R50: 62.2
LSMDC_full_test/t2v_metrics/MedR: 26.0
LSMDC_full_test/t2v_metrics/MeanR: 84.617
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.224761888123616
LSMDC_full_test/v2t_metrics/R1: 9.6
LSMDC_full_test/v2t_metrics/R5: 25.6
LSMDC_full_test/v2t_metrics/R10: 34.6
LSMDC_full_test/v2t_metrics/R50: 63.0
LSMDC_full_test/v2t_metrics/MedR: 25.0
LSMDC_full_test/v2t_metrics/MeanR: 81.638
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.410913038352767
mnt_best : 19.224761888123616
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 16.52729 (QuantReg: 11.37473) QuantErr: 11.37473 batch_time=20.54268
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 14.93070 (QuantReg: 11.96233) QuantErr: 11.96233 batch_time=0.61232
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 14.96355 (QuantReg: 12.23533) QuantErr: 12.23533 batch_time=1.41581
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 14.58323 (QuantReg: 12.54834) QuantErr: 12.54834 batch_time=0.84814
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 16.28649 (QuantReg: 12.25591) QuantErr: 12.25591 batch_time=0.62302
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 14.58000 (QuantReg: 12.18186) QuantErr: 12.18186 batch_time=0.64327
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 15.29297 (QuantReg: 12.21440) QuantErr: 12.21440 batch_time=0.60191
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 14.27994 (QuantReg: 12.66849) QuantErr: 12.66849 batch_time=0.61997
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 16.18463 (QuantReg: 12.43404) QuantErr: 12.43404 batch_time=0.60926
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 17.21956 (QuantReg: 12.16849) QuantErr: 12.16849 batch_time=0.68046
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 14.50321 (QuantReg: 12.38297) QuantErr: 12.38297 batch_time=0.58934
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 14.72744 (QuantReg: 12.41598) QuantErr: 12.41598 batch_time=0.61262
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 14.87195 (QuantReg: 12.56675) QuantErr: 12.56675 batch_time=0.61064
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 14.68629 (QuantReg: 12.86652) QuantErr: 12.86652 batch_time=0.81990
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 14.53106 (QuantReg: 12.57062) QuantErr: 12.57062 batch_time=0.62694
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 15.15368 (QuantReg: 13.11463) QuantErr: 13.11463 batch_time=0.65318
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 15.96990 (QuantReg: 12.64156) QuantErr: 12.64156 batch_time=0.67502
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 13.72798 (QuantReg: 12.98936) QuantErr: 12.98936 batch_time=0.59455
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 14.34755 (QuantReg: 12.75610) QuantErr: 12.75610 batch_time=0.59274
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 12.52170 (QuantReg: 12.50914) QuantErr: 12.50914 batch_time=0.58968
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 15.34996 (QuantReg: 12.64208) QuantErr: 12.64208 batch_time=0.60834
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 14.83895 (QuantReg: 12.78320) QuantErr: 12.78320 batch_time=0.80668
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 15.49538 (QuantReg: 13.30893) QuantErr: 13.30893 batch_time=0.59330
Train Epoch: 4 codebook_update_time=3.26337
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch4.pth ...
Done in 3.957s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch4.pth ...
Done in 8.819s
removing stale ckpt [epoch 3] [took 0.09s]
epoch : 4
loss : 15.125331176757813
quant_reg : 12.496103546142578
quant_err : 12.496103546142578
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
LSMDC_full_test/t2v_metrics/R1: 10.0
LSMDC_full_test/t2v_metrics/R5: 26.3
LSMDC_full_test/t2v_metrics/R10: 34.5
LSMDC_full_test/t2v_metrics/R50: 64.5
LSMDC_full_test/t2v_metrics/MedR: 23.0
LSMDC_full_test/t2v_metrics/MeanR: 75.017
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.857309285681975
LSMDC_full_test/v2t_metrics/R1: 9.9
LSMDC_full_test/v2t_metrics/R5: 27.4
LSMDC_full_test/v2t_metrics/R10: 36.4
LSMDC_full_test/v2t_metrics/R50: 63.6
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 74.462
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.453379420928414
mnt_best : 20.857309285681975
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 15.65095 (QuantReg: 12.47552) QuantErr: 12.47552 batch_time=19.10624
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 15.50409 (QuantReg: 11.80375) QuantErr: 11.80375 batch_time=0.60717
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 14.02215 (QuantReg: 12.69596) QuantErr: 12.69596 batch_time=0.61524
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 14.00186 (QuantReg: 11.84037) QuantErr: 11.84037 batch_time=0.60917
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 13.28046 (QuantReg: 12.77787) QuantErr: 12.77787 batch_time=0.59801
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 14.56681 (QuantReg: 12.76479) QuantErr: 12.76479 batch_time=0.62169
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 15.03531 (QuantReg: 12.29852) QuantErr: 12.29852 batch_time=0.60472
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 15.18400 (QuantReg: 12.62734) QuantErr: 12.62734 batch_time=0.60470
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 14.41819 (QuantReg: 12.86208) QuantErr: 12.86208 batch_time=0.60241
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 14.55834 (QuantReg: 12.68379) QuantErr: 12.68379 batch_time=0.61629
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 15.22928 (QuantReg: 13.00969) QuantErr: 13.00969 batch_time=0.61914
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 15.14270 (QuantReg: 13.33170) QuantErr: 13.33170 batch_time=0.61723
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 14.70173 (QuantReg: 13.18095) QuantErr: 13.18095 batch_time=0.62107
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 15.48029 (QuantReg: 13.07189) QuantErr: 13.07189 batch_time=0.60433
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 13.96784 (QuantReg: 12.90397) QuantErr: 12.90397 batch_time=0.60342
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 14.37729 (QuantReg: 12.76911) QuantErr: 12.76911 batch_time=0.60352
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 14.11780 (QuantReg: 13.01279) QuantErr: 13.01279 batch_time=0.61032
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 15.22858 (QuantReg: 12.84350) QuantErr: 12.84350 batch_time=0.60704
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 13.51091 (QuantReg: 12.92829) QuantErr: 12.92829 batch_time=0.61251
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 13.28470 (QuantReg: 13.10967) QuantErr: 13.10967 batch_time=0.60687
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 15.86467 (QuantReg: 13.05913) QuantErr: 13.05913 batch_time=0.61385
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 14.19203 (QuantReg: 13.26529) QuantErr: 13.26529 batch_time=1.20047
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 12.49185 (QuantReg: 12.82064) QuantErr: 12.82064 batch_time=0.61996
Train Epoch: 5 codebook_update_time=3.60669
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch5.pth ...
Done in 4.063s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch5.pth ...
Done in 8.351s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 14.279127712249755
quant_reg : 12.787982315063477
quant_err : 12.787982315063477
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
LSMDC_full_test/t2v_metrics/R1: 11.0
LSMDC_full_test/t2v_metrics/R5: 25.3
LSMDC_full_test/t2v_metrics/R10: 35.0
LSMDC_full_test/t2v_metrics/R50: 65.2
LSMDC_full_test/t2v_metrics/MedR: 23.0
LSMDC_full_test/t2v_metrics/MeanR: 77.194
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.35635264818574
LSMDC_full_test/v2t_metrics/R1: 9.1
LSMDC_full_test/v2t_metrics/R5: 26.9
LSMDC_full_test/v2t_metrics/R10: 37.2
LSMDC_full_test/v2t_metrics/R50: 64.7
LSMDC_full_test/v2t_metrics/MedR: 24.0
LSMDC_full_test/v2t_metrics/MeanR: 75.829
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.8823259692134
mnt_best : 21.35635264818574
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 14.54017 (QuantReg: 12.53513) QuantErr: 12.53513 batch_time=20.63973
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 12.26367 (QuantReg: 12.91728) QuantErr: 12.91728 batch_time=0.60945
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 15.01377 (QuantReg: 13.03047) QuantErr: 13.03047 batch_time=0.60607
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 13.88177 (QuantReg: 12.78374) QuantErr: 12.78374 batch_time=0.61935
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 12.16513 (QuantReg: 12.80488) QuantErr: 12.80488 batch_time=0.60574
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 15.34412 (QuantReg: 12.73779) QuantErr: 12.73779 batch_time=0.61511
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 12.70683 (QuantReg: 13.00432) QuantErr: 13.00432 batch_time=1.50923
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 11.84818 (QuantReg: 13.37119) QuantErr: 13.37119 batch_time=0.60673
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 13.87495 (QuantReg: 12.73455) QuantErr: 12.73455 batch_time=0.64532
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 12.89394 (QuantReg: 12.79802) QuantErr: 12.79802 batch_time=0.65273
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 12.93491 (QuantReg: 12.75261) QuantErr: 12.75261 batch_time=0.60906
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 13.97346 (QuantReg: 12.74315) QuantErr: 12.74315 batch_time=0.60671
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 14.30153 (QuantReg: 13.37123) QuantErr: 13.37123 batch_time=0.83242
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 13.87436 (QuantReg: 13.03050) QuantErr: 13.03050 batch_time=1.12896
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 13.07190 (QuantReg: 12.93735) QuantErr: 12.93735 batch_time=0.64458
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 13.26043 (QuantReg: 13.31617) QuantErr: 13.31617 batch_time=0.60286
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 13.32164 (QuantReg: 13.58489) QuantErr: 13.58489 batch_time=0.60946
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 13.90484 (QuantReg: 13.09283) QuantErr: 13.09283 batch_time=0.61628
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 14.31351 (QuantReg: 13.59076) QuantErr: 13.59076 batch_time=0.93056
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 15.01466 (QuantReg: 13.21178) QuantErr: 13.21178 batch_time=0.60851
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 13.13561 (QuantReg: 13.09336) QuantErr: 13.09336 batch_time=0.61521
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 14.43031 (QuantReg: 13.30154) QuantErr: 13.30154 batch_time=0.81547
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 12.13791 (QuantReg: 13.57454) QuantErr: 13.57454 batch_time=0.81974
Train Epoch: 6 codebook_update_time=3.34280
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch6.pth ...
Done in 5.863s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch6.pth ...
Done in 10.281s
removing stale ckpt [epoch 5] [took 0.25s]
epoch : 6
loss : 13.578512992858887
quant_reg : 13.13257447052002
quant_err : 13.13257447052002
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
LSMDC_full_test/t2v_metrics/R1: 11.2
LSMDC_full_test/t2v_metrics/R5: 27.7
LSMDC_full_test/t2v_metrics/R10: 37.4
LSMDC_full_test/t2v_metrics/R50: 65.8
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 73.007
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.638959609572165
LSMDC_full_test/v2t_metrics/R1: 10.9
LSMDC_full_test/v2t_metrics/R5: 27.6
LSMDC_full_test/v2t_metrics/R10: 37.7
LSMDC_full_test/v2t_metrics/R50: 66.4
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 71.983
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.467718645494752
mnt_best : 22.638959609572165
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 15.01007 (QuantReg: 12.67152) QuantErr: 12.67152 batch_time=18.90801
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 13.05971 (QuantReg: 13.09090) QuantErr: 13.09090 batch_time=0.61504
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 14.67229 (QuantReg: 13.48520) QuantErr: 13.48520 batch_time=1.19895
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 12.46377 (QuantReg: 13.46013) QuantErr: 13.46013 batch_time=0.71262
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 12.98610 (QuantReg: 13.15007) QuantErr: 13.15007 batch_time=0.61332
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 13.14126 (QuantReg: 13.60259) QuantErr: 13.60259 batch_time=0.92313
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 12.89764 (QuantReg: 13.41224) QuantErr: 13.41224 batch_time=1.64396
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 13.59983 (QuantReg: 13.22202) QuantErr: 13.22202 batch_time=0.59521
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 13.62476 (QuantReg: 13.39924) QuantErr: 13.39924 batch_time=0.59154
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 13.86290 (QuantReg: 13.35893) QuantErr: 13.35893 batch_time=1.25807
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 12.20646 (QuantReg: 13.17483) QuantErr: 13.17483 batch_time=0.60923
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 13.42391 (QuantReg: 13.73861) QuantErr: 13.73861 batch_time=0.59864
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 12.25331 (QuantReg: 13.52029) QuantErr: 13.52029 batch_time=0.59895
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 11.80625 (QuantReg: 13.56979) QuantErr: 13.56979 batch_time=1.46676
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 12.78524 (QuantReg: 13.47581) QuantErr: 13.47581 batch_time=0.61298
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 11.96917 (QuantReg: 13.40262) QuantErr: 13.40262 batch_time=0.61147
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 11.48593 (QuantReg: 13.66244) QuantErr: 13.66244 batch_time=0.59752
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 13.32694 (QuantReg: 14.06351) QuantErr: 14.06351 batch_time=0.60738
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 11.33280 (QuantReg: 13.77400) QuantErr: 13.77400 batch_time=0.59295
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 11.65698 (QuantReg: 13.98858) QuantErr: 13.98858 batch_time=1.64654
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 14.04742 (QuantReg: 13.73846) QuantErr: 13.73846 batch_time=0.61092
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 14.68911 (QuantReg: 13.45441) QuantErr: 13.45441 batch_time=0.61951
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 14.99003 (QuantReg: 13.22573) QuantErr: 13.22573 batch_time=0.60616
Train Epoch: 7 codebook_update_time=3.55067
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch7.pth ...
Done in 5.550s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch7.pth ...
Done in 10.896s
removing stale ckpt [epoch 6] [took 0.35s]
epoch : 7
loss : 12.928705284118653
quant_reg : 13.498936592102051
quant_err : 13.498936592102051
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
LSMDC_full_test/t2v_metrics/R1: 11.6
LSMDC_full_test/t2v_metrics/R5: 27.6
LSMDC_full_test/t2v_metrics/R10: 37.0
LSMDC_full_test/t2v_metrics/R50: 66.7
LSMDC_full_test/t2v_metrics/MedR: 19.5
LSMDC_full_test/t2v_metrics/MeanR: 72.2
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.795874907759682
LSMDC_full_test/v2t_metrics/R1: 10.8
LSMDC_full_test/v2t_metrics/R5: 28.6
LSMDC_full_test/v2t_metrics/R10: 37.5
LSMDC_full_test/v2t_metrics/R50: 66.7
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 68.999
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.625960196523028
mnt_best : 22.795874907759682
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 12.41065 (QuantReg: 13.60769) QuantErr: 13.60769 batch_time=21.63289
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 12.13455 (QuantReg: 13.07468) QuantErr: 13.07468 batch_time=0.60979
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 13.06882 (QuantReg: 13.51929) QuantErr: 13.51929 batch_time=0.61246
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 12.55411 (QuantReg: 13.44745) QuantErr: 13.44745 batch_time=0.60850
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 11.46028 (QuantReg: 13.31032) QuantErr: 13.31032 batch_time=1.25257
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 12.71348 (QuantReg: 13.35769) QuantErr: 13.35769 batch_time=0.61989
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 13.63869 (QuantReg: 13.62466) QuantErr: 13.62466 batch_time=0.61941
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 12.73510 (QuantReg: 13.68738) QuantErr: 13.68738 batch_time=0.60818
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 12.15156 (QuantReg: 13.73698) QuantErr: 13.73698 batch_time=0.61504
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 13.79112 (QuantReg: 13.36001) QuantErr: 13.36001 batch_time=0.61082
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 11.56484 (QuantReg: 13.77110) QuantErr: 13.77110 batch_time=0.61073
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 13.39492 (QuantReg: 13.85875) QuantErr: 13.85875 batch_time=0.72651
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 11.64097 (QuantReg: 13.71339) QuantErr: 13.71339 batch_time=0.62064
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 10.32263 (QuantReg: 13.89477) QuantErr: 13.89477 batch_time=1.25314
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 12.38623 (QuantReg: 14.10328) QuantErr: 14.10328 batch_time=0.61379
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 11.51237 (QuantReg: 13.87586) QuantErr: 13.87586 batch_time=0.61624
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 11.91717 (QuantReg: 13.97981) QuantErr: 13.97981 batch_time=0.61448
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 13.43176 (QuantReg: 13.89652) QuantErr: 13.89652 batch_time=1.02758
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 12.03799 (QuantReg: 13.71345) QuantErr: 13.71345 batch_time=0.61738
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 13.25762 (QuantReg: 13.79625) QuantErr: 13.79625 batch_time=0.82901
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 13.26049 (QuantReg: 13.73072) QuantErr: 13.73072 batch_time=0.60055
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 13.04654 (QuantReg: 13.61915) QuantErr: 13.61915 batch_time=0.61533
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 12.60939 (QuantReg: 14.07697) QuantErr: 14.07697 batch_time=0.61206
Train Epoch: 8 codebook_update_time=3.29834
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch8.pth ...
Done in 5.877s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch8.pth ...
Done in 10.560s
removing stale ckpt [epoch 7] [took 0.39s]
epoch : 8
loss : 12.399544597625733
quant_reg : 13.713250854492188
quant_err : 13.713250854492188
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
LSMDC_full_test/t2v_metrics/R1: 11.9
LSMDC_full_test/t2v_metrics/R5: 28.8
LSMDC_full_test/t2v_metrics/R10: 39.0
LSMDC_full_test/t2v_metrics/R50: 68.7
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 70.047
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.73201889226039
LSMDC_full_test/v2t_metrics/R1: 9.7
LSMDC_full_test/v2t_metrics/R5: 27.9
LSMDC_full_test/v2t_metrics/R10: 37.1
LSMDC_full_test/v2t_metrics/R50: 67.9
LSMDC_full_test/v2t_metrics/MedR: 22.5
LSMDC_full_test/v2t_metrics/MeanR: 71.026
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.573301632807873
mnt_best : 23.73201889226039
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 11.23595 (QuantReg: 13.35666) QuantErr: 13.35666 batch_time=19.43113
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 11.59472 (QuantReg: 13.61144) QuantErr: 13.61144 batch_time=0.61849
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 12.51466 (QuantReg: 13.56451) QuantErr: 13.56451 batch_time=0.60675
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 12.07237 (QuantReg: 13.61566) QuantErr: 13.61566 batch_time=0.61992
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 12.21066 (QuantReg: 13.76993) QuantErr: 13.76993 batch_time=0.60593
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 11.08503 (QuantReg: 13.71949) QuantErr: 13.71949 batch_time=0.61172
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 10.54916 (QuantReg: 14.08582) QuantErr: 14.08582 batch_time=1.47449
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 11.73991 (QuantReg: 14.03170) QuantErr: 14.03170 batch_time=0.61303
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 10.85295 (QuantReg: 14.12393) QuantErr: 14.12393 batch_time=0.60963
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 13.15962 (QuantReg: 13.87573) QuantErr: 13.87573 batch_time=0.61195
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 11.72135 (QuantReg: 13.97695) QuantErr: 13.97695 batch_time=0.60642
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 9.49249 (QuantReg: 14.28274) QuantErr: 14.28274 batch_time=0.63031
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 12.23348 (QuantReg: 14.09309) QuantErr: 14.09309 batch_time=0.62194
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 10.47746 (QuantReg: 13.93010) QuantErr: 13.93010 batch_time=0.61588
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 11.99329 (QuantReg: 13.93053) QuantErr: 13.93053 batch_time=0.61060
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 11.79040 (QuantReg: 14.15835) QuantErr: 14.15835 batch_time=0.68859
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 11.04743 (QuantReg: 13.95826) QuantErr: 13.95826 batch_time=0.60987
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 12.34173 (QuantReg: 14.29207) QuantErr: 14.29207 batch_time=1.02962
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 9.84372 (QuantReg: 13.97729) QuantErr: 13.97729 batch_time=0.60935
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 12.99015 (QuantReg: 13.95601) QuantErr: 13.95601 batch_time=0.59660
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 11.93090 (QuantReg: 14.00999) QuantErr: 14.00999 batch_time=0.60960
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 12.65433 (QuantReg: 13.79640) QuantErr: 13.79640 batch_time=0.61240
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 11.93162 (QuantReg: 14.21034) QuantErr: 14.21034 batch_time=0.61458
Train Epoch: 9 codebook_update_time=3.33379
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch9.pth ...
Done in 4.375s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch9.pth ...
Done in 8.972s
removing stale ckpt [epoch 8] [took 0.02s]
epoch : 9
loss : 11.834897148132324
quant_reg : 13.94833836364746
quant_err : 13.94833836364746
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
LSMDC_full_test/t2v_metrics/R1: 12.3
LSMDC_full_test/t2v_metrics/R5: 27.7
LSMDC_full_test/t2v_metrics/R10: 39.7
LSMDC_full_test/t2v_metrics/R50: 67.8
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 69.419
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.826401861705648
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 30.4
LSMDC_full_test/v2t_metrics/R10: 39.5
LSMDC_full_test/v2t_metrics/R50: 68.5
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 66.369
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.99143212686374
mnt_best : 23.826401861705648
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 12.24192 (QuantReg: 13.81917) QuantErr: 13.81917 batch_time=19.18655
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 12.18634 (QuantReg: 13.94332) QuantErr: 13.94332 batch_time=0.58875
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 10.69821 (QuantReg: 14.08848) QuantErr: 14.08848 batch_time=0.58762
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 12.42309 (QuantReg: 14.36632) QuantErr: 14.36632 batch_time=0.58916
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 10.32032 (QuantReg: 14.33670) QuantErr: 14.33670 batch_time=0.59389
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 12.14022 (QuantReg: 13.87764) QuantErr: 13.87764 batch_time=0.59089
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 11.40136 (QuantReg: 14.24421) QuantErr: 14.24421 batch_time=0.61753
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 10.55774 (QuantReg: 14.16905) QuantErr: 14.16905 batch_time=0.62581
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 12.02978 (QuantReg: 14.19364) QuantErr: 14.19364 batch_time=0.79570
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 11.40669 (QuantReg: 14.32292) QuantErr: 14.32292 batch_time=0.60647
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 9.71675 (QuantReg: 14.44164) QuantErr: 14.44164 batch_time=0.60648
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 13.28952 (QuantReg: 14.35420) QuantErr: 14.35420 batch_time=0.60934
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 11.14601 (QuantReg: 14.32770) QuantErr: 14.32770 batch_time=0.60574
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 12.05831 (QuantReg: 14.19058) QuantErr: 14.19058 batch_time=2.53834
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 11.16356 (QuantReg: 14.22740) QuantErr: 14.22740 batch_time=0.77431
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 11.80405 (QuantReg: 14.33746) QuantErr: 14.33746 batch_time=0.63248
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 11.31391 (QuantReg: 14.21816) QuantErr: 14.21816 batch_time=0.68064
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 11.25687 (QuantReg: 14.34811) QuantErr: 14.34811 batch_time=0.68806
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 11.58539 (QuantReg: 14.30680) QuantErr: 14.30680 batch_time=0.69193
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 10.02174 (QuantReg: 14.21518) QuantErr: 14.21518 batch_time=0.68688
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 11.31804 (QuantReg: 14.27033) QuantErr: 14.27033 batch_time=0.61629
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 10.45927 (QuantReg: 14.41690) QuantErr: 14.41690 batch_time=1.00871
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 10.66495 (QuantReg: 14.47048) QuantErr: 14.47048 batch_time=0.61068
Train Epoch: 10 codebook_update_time=3.28288
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch10.pth ...
Done in 4.212s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch10.pth ...
Done in 9.264s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 11.294474807739258
quant_reg : 14.206634197235108
quant_err : 14.206634197235108
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
LSMDC_full_test/t2v_metrics/R1: 13.4
LSMDC_full_test/t2v_metrics/R5: 31.1
LSMDC_full_test/t2v_metrics/R10: 40.8
LSMDC_full_test/t2v_metrics/R50: 70.1
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 66.19
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.71432430329307
LSMDC_full_test/v2t_metrics/R1: 12.5
LSMDC_full_test/v2t_metrics/R5: 31.5
LSMDC_full_test/v2t_metrics/R10: 40.1
LSMDC_full_test/v2t_metrics/R50: 68.2
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 67.4
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.087361033070675
mnt_best : 25.71432430329307
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 11.84523 (QuantReg: 13.72637) QuantErr: 13.72637 batch_time=16.74976
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 10.84587 (QuantReg: 14.00431) QuantErr: 14.00431 batch_time=0.60156
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 12.06230 (QuantReg: 14.11533) QuantErr: 14.11533 batch_time=0.60444
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 11.33605 (QuantReg: 14.19374) QuantErr: 14.19374 batch_time=0.59553
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 10.87994 (QuantReg: 14.53439) QuantErr: 14.53439 batch_time=0.69978
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 11.32503 (QuantReg: 13.95476) QuantErr: 13.95476 batch_time=0.59013
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 9.75440 (QuantReg: 14.47373) QuantErr: 14.47373 batch_time=0.59620
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 11.66253 (QuantReg: 14.22247) QuantErr: 14.22247 batch_time=0.59999
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 11.33386 (QuantReg: 14.44188) QuantErr: 14.44188 batch_time=0.61801
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 9.94281 (QuantReg: 14.48088) QuantErr: 14.48088 batch_time=0.62027
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 9.68110 (QuantReg: 14.13640) QuantErr: 14.13640 batch_time=0.63052
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 13.20342 (QuantReg: 14.41563) QuantErr: 14.41563 batch_time=0.61294
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 10.50571 (QuantReg: 14.24971) QuantErr: 14.24971 batch_time=1.91628
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 9.84334 (QuantReg: 14.56043) QuantErr: 14.56043 batch_time=0.69031
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 10.15594 (QuantReg: 14.38477) QuantErr: 14.38477 batch_time=1.08507
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 9.98536 (QuantReg: 14.35139) QuantErr: 14.35139 batch_time=0.60772
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 11.19665 (QuantReg: 14.51437) QuantErr: 14.51437 batch_time=0.61784
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 10.37062 (QuantReg: 14.24046) QuantErr: 14.24046 batch_time=0.60750
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 11.39776 (QuantReg: 14.25868) QuantErr: 14.25868 batch_time=0.62479
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 10.88501 (QuantReg: 14.47835) QuantErr: 14.47835 batch_time=0.59094
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 10.85540 (QuantReg: 14.52073) QuantErr: 14.52073 batch_time=0.62801
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 12.45994 (QuantReg: 14.60864) QuantErr: 14.60864 batch_time=0.59328
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 9.88872 (QuantReg: 14.63180) QuantErr: 14.63180 batch_time=0.59919
Train Epoch: 11 codebook_update_time=3.26398
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch11.pth ...
Done in 5.773s
removing stale ckpt [epoch 10] [took 0.04s]
epoch : 11
loss : 10.962527809143067
quant_reg : 14.36484125137329
quant_err : 14.36484125137329
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
LSMDC_full_test/t2v_metrics/R1: 13.0
LSMDC_full_test/t2v_metrics/R5: 30.3
LSMDC_full_test/t2v_metrics/R10: 40.5
LSMDC_full_test/t2v_metrics/R50: 69.2
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 70.222
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.173697045163998
LSMDC_full_test/v2t_metrics/R1: 12.8
LSMDC_full_test/v2t_metrics/R5: 30.9
LSMDC_full_test/v2t_metrics/R10: 39.6
LSMDC_full_test/v2t_metrics/R50: 69.0
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 67.14
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.020033009520017
mnt_best : 25.71432430329307
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 9.80194 (QuantReg: 14.51896) QuantErr: 14.51896 batch_time=20.95578
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 10.38375 (QuantReg: 14.44790) QuantErr: 14.44790 batch_time=0.67142
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 8.75633 (QuantReg: 14.58842) QuantErr: 14.58842 batch_time=0.61385
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 11.23824 (QuantReg: 14.45596) QuantErr: 14.45596 batch_time=1.00071
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 10.64482 (QuantReg: 14.47220) QuantErr: 14.47220 batch_time=0.60155
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 9.83745 (QuantReg: 14.69620) QuantErr: 14.69620 batch_time=0.60410
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 10.73915 (QuantReg: 14.44820) QuantErr: 14.44820 batch_time=0.60728
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 11.37879 (QuantReg: 14.38101) QuantErr: 14.38101 batch_time=0.61066
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 10.50496 (QuantReg: 14.73816) QuantErr: 14.73816 batch_time=0.66039
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 11.41503 (QuantReg: 14.40325) QuantErr: 14.40325 batch_time=0.60155
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 8.92809 (QuantReg: 14.31859) QuantErr: 14.31859 batch_time=0.60159
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 8.76371 (QuantReg: 14.43271) QuantErr: 14.43271 batch_time=0.60388
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 10.81063 (QuantReg: 14.76064) QuantErr: 14.76064 batch_time=0.60156
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 11.02927 (QuantReg: 14.57891) QuantErr: 14.57891 batch_time=0.61140
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 10.25170 (QuantReg: 14.55741) QuantErr: 14.55741 batch_time=0.61119
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 10.64407 (QuantReg: 14.45388) QuantErr: 14.45388 batch_time=0.61841
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 11.37710 (QuantReg: 14.42478) QuantErr: 14.42478 batch_time=0.61644
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 10.88657 (QuantReg: 15.04616) QuantErr: 15.04616 batch_time=0.60357
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 10.86959 (QuantReg: 14.84975) QuantErr: 14.84975 batch_time=0.60919
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 12.38786 (QuantReg: 14.40894) QuantErr: 14.40894 batch_time=0.60777
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 10.98099 (QuantReg: 14.24279) QuantErr: 14.24279 batch_time=0.61175
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 11.14709 (QuantReg: 14.50889) QuantErr: 14.50889 batch_time=0.61182
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 9.14912 (QuantReg: 14.96441) QuantErr: 14.96441 batch_time=0.61046
Train Epoch: 12 codebook_update_time=3.27866
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch12.pth ...
Done in 4.675s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 10.553606601715089
quant_reg : 14.526742370605469
quant_err : 14.526742370605469
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
LSMDC_full_test/t2v_metrics/R1: 12.1
LSMDC_full_test/t2v_metrics/R5: 29.9
LSMDC_full_test/t2v_metrics/R10: 40.2
LSMDC_full_test/t2v_metrics/R50: 68.5
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 72.118
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.40961168400568
LSMDC_full_test/v2t_metrics/R1: 12.1
LSMDC_full_test/v2t_metrics/R5: 29.5
LSMDC_full_test/v2t_metrics/R10: 39.1
LSMDC_full_test/v2t_metrics/R50: 68.2
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 70.012
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.07657543839008
mnt_best : 25.71432430329307
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 9.82759 (QuantReg: 14.19587) QuantErr: 14.19587 batch_time=19.79354
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 9.97946 (QuantReg: 14.56673) QuantErr: 14.56673 batch_time=0.61401
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 11.14087 (QuantReg: 14.72980) QuantErr: 14.72980 batch_time=0.61917
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 9.85460 (QuantReg: 14.60537) QuantErr: 14.60537 batch_time=0.61096
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 11.47962 (QuantReg: 14.46252) QuantErr: 14.46252 batch_time=0.62184
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 8.91585 (QuantReg: 14.64670) QuantErr: 14.64670 batch_time=0.62393
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 10.81782 (QuantReg: 14.27063) QuantErr: 14.27063 batch_time=0.61525
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 9.86942 (QuantReg: 14.75755) QuantErr: 14.75755 batch_time=0.60826
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 9.18625 (QuantReg: 14.48078) QuantErr: 14.48078 batch_time=0.59940
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 9.84595 (QuantReg: 14.64207) QuantErr: 14.64207 batch_time=0.61529
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 10.53617 (QuantReg: 14.63520) QuantErr: 14.63520 batch_time=0.61644
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 10.18975 (QuantReg: 14.67469) QuantErr: 14.67469 batch_time=0.60894
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 9.88774 (QuantReg: 14.66884) QuantErr: 14.66884 batch_time=0.82214
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 12.01293 (QuantReg: 14.62941) QuantErr: 14.62941 batch_time=0.60764
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 10.60888 (QuantReg: 14.74747) QuantErr: 14.74747 batch_time=0.63996
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 10.63278 (QuantReg: 14.68929) QuantErr: 14.68929 batch_time=0.60467
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 10.91287 (QuantReg: 14.93070) QuantErr: 14.93070 batch_time=0.60331
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 9.61338 (QuantReg: 14.77398) QuantErr: 14.77398 batch_time=0.61365
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 10.42229 (QuantReg: 14.79181) QuantErr: 14.79181 batch_time=0.60454
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 10.13615 (QuantReg: 14.67936) QuantErr: 14.67936 batch_time=0.61563
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 10.28155 (QuantReg: 14.42514) QuantErr: 14.42514 batch_time=0.61471
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 9.68196 (QuantReg: 14.71345) QuantErr: 14.71345 batch_time=0.60600
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 9.47473 (QuantReg: 14.55245) QuantErr: 14.55245 batch_time=0.61155
Train Epoch: 13 codebook_update_time=3.23061
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch13.pth ...
Done in 4.547s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 10.25339986038208
quant_reg : 14.662595268249511
quant_err : 14.662595268249511
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
LSMDC_full_test/t2v_metrics/R1: 11.9
LSMDC_full_test/t2v_metrics/R5: 30.1
LSMDC_full_test/t2v_metrics/R10: 40.1
LSMDC_full_test/t2v_metrics/R50: 69.8
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 69.687
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.308189306300076
LSMDC_full_test/v2t_metrics/R1: 13.1
LSMDC_full_test/v2t_metrics/R5: 30.4
LSMDC_full_test/v2t_metrics/R10: 40.3
LSMDC_full_test/v2t_metrics/R50: 70.1
LSMDC_full_test/v2t_metrics/MedR: 18.25
LSMDC_full_test/v2t_metrics/MeanR: 67.93
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.224155891895037
mnt_best : 25.71432430329307
not_improved_count: 3
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 10.08391 (QuantReg: 14.88337) QuantErr: 14.88337 batch_time=18.25756
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 8.45153 (QuantReg: 14.39071) QuantErr: 14.39071 batch_time=0.59703
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 10.07166 (QuantReg: 14.60140) QuantErr: 14.60140 batch_time=0.60463
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 9.42451 (QuantReg: 14.55158) QuantErr: 14.55158 batch_time=0.65513
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 10.85618 (QuantReg: 14.64555) QuantErr: 14.64555 batch_time=0.61202
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 10.06082 (QuantReg: 14.79963) QuantErr: 14.79963 batch_time=0.73458
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 9.97566 (QuantReg: 14.71584) QuantErr: 14.71584 batch_time=0.61891
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 9.73566 (QuantReg: 14.60666) QuantErr: 14.60666 batch_time=0.60851
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 10.00991 (QuantReg: 14.32896) QuantErr: 14.32896 batch_time=0.61083
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 10.52678 (QuantReg: 14.82185) QuantErr: 14.82185 batch_time=0.61067
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 8.88665 (QuantReg: 15.14307) QuantErr: 15.14307 batch_time=0.60780
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 9.48354 (QuantReg: 14.79430) QuantErr: 14.79430 batch_time=0.60030
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 9.11712 (QuantReg: 14.93866) QuantErr: 14.93866 batch_time=0.71355
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 8.74027 (QuantReg: 14.84952) QuantErr: 14.84952 batch_time=2.09037
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 11.58798 (QuantReg: 14.81262) QuantErr: 14.81262 batch_time=0.61680
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 12.11215 (QuantReg: 14.57476) QuantErr: 14.57476 batch_time=0.83467
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 9.43113 (QuantReg: 15.00649) QuantErr: 15.00649 batch_time=0.60690
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 10.95165 (QuantReg: 14.87674) QuantErr: 14.87674 batch_time=0.59132
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 10.54159 (QuantReg: 14.77240) QuantErr: 14.77240 batch_time=0.60320
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 9.07549 (QuantReg: 14.94482) QuantErr: 14.94482 batch_time=0.63985
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 9.01534 (QuantReg: 14.86050) QuantErr: 14.86050 batch_time=0.61459
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 10.96878 (QuantReg: 15.05782) QuantErr: 15.05782 batch_time=0.61165
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 10.03456 (QuantReg: 14.97967) QuantErr: 14.97967 batch_time=0.60613
Train Epoch: 14 codebook_update_time=3.24240
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch14.pth ...
Done in 4.050s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 9.815399806976318
quant_reg : 14.801168251037598
quant_err : 14.801168251037598
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
LSMDC_full_test/t2v_metrics/R1: 12.2
LSMDC_full_test/t2v_metrics/R5: 30.4
LSMDC_full_test/t2v_metrics/R10: 42.2
LSMDC_full_test/t2v_metrics/R50: 69.5
LSMDC_full_test/t2v_metrics/MedR: 16.0
LSMDC_full_test/t2v_metrics/MeanR: 70.668
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.013931435162498
LSMDC_full_test/v2t_metrics/R1: 12.0
LSMDC_full_test/v2t_metrics/R5: 30.6
LSMDC_full_test/v2t_metrics/R10: 40.3
LSMDC_full_test/v2t_metrics/R50: 70.0
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 67.501
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.551003010553774
mnt_best : 25.71432430329307
not_improved_count: 4
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 10.44200 (QuantReg: 14.65564) QuantErr: 14.65564 batch_time=19.43765
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 10.50081 (QuantReg: 14.65644) QuantErr: 14.65644 batch_time=0.68898
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 9.22588 (QuantReg: 14.82695) QuantErr: 14.82695 batch_time=1.44944
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 9.50589 (QuantReg: 14.82378) QuantErr: 14.82378 batch_time=0.61369
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 9.31036 (QuantReg: 14.80741) QuantErr: 14.80741 batch_time=1.24249
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 7.90524 (QuantReg: 14.85456) QuantErr: 14.85456 batch_time=0.60748
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 10.84799 (QuantReg: 14.77393) QuantErr: 14.77393 batch_time=0.90798
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 9.90963 (QuantReg: 14.98727) QuantErr: 14.98727 batch_time=0.60605
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 10.61770 (QuantReg: 14.77788) QuantErr: 14.77788 batch_time=0.60492
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 7.88240 (QuantReg: 15.08949) QuantErr: 15.08949 batch_time=0.60736
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 11.00048 (QuantReg: 14.86855) QuantErr: 14.86855 batch_time=0.60429
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 11.18736 (QuantReg: 14.98083) QuantErr: 14.98083 batch_time=0.60763
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 10.20367 (QuantReg: 15.00632) QuantErr: 15.00632 batch_time=0.61049
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 9.58632 (QuantReg: 15.00262) QuantErr: 15.00262 batch_time=0.61785
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 8.64438 (QuantReg: 15.15105) QuantErr: 15.15105 batch_time=0.60748
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 9.93977 (QuantReg: 14.97813) QuantErr: 14.97813 batch_time=0.61296
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 10.42943 (QuantReg: 15.08980) QuantErr: 15.08980 batch_time=0.60643
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 10.30455 (QuantReg: 15.02501) QuantErr: 15.02501 batch_time=0.62148
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 8.16834 (QuantReg: 14.89663) QuantErr: 14.89663 batch_time=0.61485
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 9.82557 (QuantReg: 15.05450) QuantErr: 15.05450 batch_time=0.61435
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 8.20494 (QuantReg: 15.02327) QuantErr: 15.02327 batch_time=0.61602
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 7.98553 (QuantReg: 14.95657) QuantErr: 14.95657 batch_time=0.61166
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 9.24749 (QuantReg: 14.84180) QuantErr: 14.84180 batch_time=0.61457
Train Epoch: 15 codebook_update_time=3.56151
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch15.pth ...
Done in 3.930s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 9.503763656616211
quant_reg : 14.91536022567749
quant_err : 14.91536022567749
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
LSMDC_full_test/t2v_metrics/R1: 12.1
LSMDC_full_test/t2v_metrics/R5: 31.0
LSMDC_full_test/t2v_metrics/R10: 41.2
LSMDC_full_test/t2v_metrics/R50: 69.5
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 69.985
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.908529735798282
LSMDC_full_test/v2t_metrics/R1: 12.2
LSMDC_full_test/v2t_metrics/R5: 31.1
LSMDC_full_test/v2t_metrics/R10: 40.3
LSMDC_full_test/v2t_metrics/R50: 70.2
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 67.341
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.82037975343605
mnt_best : 25.71432430329307
not_improved_count: 5
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 9.55281 (QuantReg: 14.63623) QuantErr: 14.63623 batch_time=18.42443
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 8.99454 (QuantReg: 14.73616) QuantErr: 14.73616 batch_time=0.60745
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 10.24321 (QuantReg: 14.80350) QuantErr: 14.80350 batch_time=0.60413
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 9.15388 (QuantReg: 15.02309) QuantErr: 15.02309 batch_time=0.60424
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 8.36614 (QuantReg: 15.07002) QuantErr: 15.07002 batch_time=0.58863
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 10.99701 (QuantReg: 14.92884) QuantErr: 14.92884 batch_time=0.59430
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 10.22569 (QuantReg: 14.85420) QuantErr: 14.85420 batch_time=1.19715
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 10.48994 (QuantReg: 14.95541) QuantErr: 14.95541 batch_time=0.59917
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 9.13636 (QuantReg: 14.77908) QuantErr: 14.77908 batch_time=0.60553
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 8.02680 (QuantReg: 14.98653) QuantErr: 14.98653 batch_time=0.62039
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 8.92604 (QuantReg: 15.03503) QuantErr: 15.03503 batch_time=0.61845
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 8.34885 (QuantReg: 14.84573) QuantErr: 14.84573 batch_time=0.61974
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 9.25324 (QuantReg: 15.19661) QuantErr: 15.19661 batch_time=0.59871
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 9.04737 (QuantReg: 15.10884) QuantErr: 15.10884 batch_time=2.55423
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 9.07604 (QuantReg: 15.18359) QuantErr: 15.18359 batch_time=1.31493
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 10.36550 (QuantReg: 14.79603) QuantErr: 14.79603 batch_time=0.59654
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 8.80256 (QuantReg: 15.14528) QuantErr: 15.14528 batch_time=0.60676
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 9.88279 (QuantReg: 15.15038) QuantErr: 15.15038 batch_time=0.59720
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 9.65215 (QuantReg: 15.11207) QuantErr: 15.11207 batch_time=0.61712
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 8.59362 (QuantReg: 14.92806) QuantErr: 14.92806 batch_time=0.59845
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 10.60663 (QuantReg: 15.14914) QuantErr: 15.14914 batch_time=0.65254
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 9.28027 (QuantReg: 15.00292) QuantErr: 15.00292 batch_time=0.59551
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 9.73251 (QuantReg: 14.99439) QuantErr: 14.99439 batch_time=0.62622
Train Epoch: 16 codebook_update_time=3.25264
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch16.pth ...
Done in 4.379s
removing stale ckpt [epoch 15] [took 0.00s]
epoch : 16
loss : 9.234854656219483
quant_reg : 15.004243679046631
quant_err : 15.004243679046631
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
LSMDC_full_test/t2v_metrics/R1: 12.2
LSMDC_full_test/t2v_metrics/R5: 32.4
LSMDC_full_test/t2v_metrics/R10: 42.7
LSMDC_full_test/t2v_metrics/R50: 69.5
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 70.418
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.651390095997474
LSMDC_full_test/v2t_metrics/R1: 11.9
LSMDC_full_test/v2t_metrics/R5: 31.4
LSMDC_full_test/v2t_metrics/R10: 40.9
LSMDC_full_test/v2t_metrics/R50: 69.7
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 69.498
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.816087160379848
mnt_best : 25.71432430329307
not_improved_count: 6
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 10.06749 (QuantReg: 14.92666) QuantErr: 14.92666 batch_time=22.50546
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 9.78600 (QuantReg: 14.72776) QuantErr: 14.72776 batch_time=0.60304
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 8.70359 (QuantReg: 15.01295) QuantErr: 15.01295 batch_time=0.61170
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 8.69114 (QuantReg: 14.74804) QuantErr: 14.74804 batch_time=0.60691
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 10.43168 (QuantReg: 15.04595) QuantErr: 15.04595 batch_time=0.63037
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 9.27435 (QuantReg: 14.86040) QuantErr: 14.86040 batch_time=0.61588
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 9.42183 (QuantReg: 15.14607) QuantErr: 15.14607 batch_time=0.61806
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 9.35505 (QuantReg: 15.01781) QuantErr: 15.01781 batch_time=0.60722
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 11.50849 (QuantReg: 15.00099) QuantErr: 15.00099 batch_time=0.59687
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 9.22188 (QuantReg: 15.02057) QuantErr: 15.02057 batch_time=1.35313
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 8.30336 (QuantReg: 15.38585) QuantErr: 15.38585 batch_time=0.61063
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 9.10856 (QuantReg: 15.23095) QuantErr: 15.23095 batch_time=0.59454
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 8.57568 (QuantReg: 14.93578) QuantErr: 14.93578 batch_time=0.61689
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 7.97439 (QuantReg: 15.68605) QuantErr: 15.68605 batch_time=0.61699
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 8.79490 (QuantReg: 15.20608) QuantErr: 15.20608 batch_time=0.93407
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 7.14728 (QuantReg: 15.22594) QuantErr: 15.22594 batch_time=0.60777
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 9.21318 (QuantReg: 15.00214) QuantErr: 15.00214 batch_time=0.61785
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 9.22752 (QuantReg: 15.11580) QuantErr: 15.11580 batch_time=0.60496
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 8.09451 (QuantReg: 15.23189) QuantErr: 15.23189 batch_time=0.61125
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 8.17847 (QuantReg: 14.78099) QuantErr: 14.78099 batch_time=0.59907
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 9.18122 (QuantReg: 15.31404) QuantErr: 15.31404 batch_time=1.13038
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 8.75016 (QuantReg: 15.20883) QuantErr: 15.20883 batch_time=0.61160
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 9.50711 (QuantReg: 15.15883) QuantErr: 15.15883 batch_time=0.61429
Train Epoch: 17 codebook_update_time=3.56888
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch17.pth ...
Done in 5.878s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 9.002302356719971
quant_reg : 15.115643875122071
quant_err : 15.115643875122071
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
LSMDC_full_test/t2v_metrics/R1: 12.5
LSMDC_full_test/t2v_metrics/R5: 31.6
LSMDC_full_test/t2v_metrics/R10: 42.2
LSMDC_full_test/t2v_metrics/R50: 69.2
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 72.51
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.54483972772254
LSMDC_full_test/v2t_metrics/R1: 12.0
LSMDC_full_test/v2t_metrics/R5: 31.2
LSMDC_full_test/v2t_metrics/R10: 40.3
LSMDC_full_test/v2t_metrics/R50: 68.8
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 69.666
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.71042957517801
mnt_best : 25.71432430329307
not_improved_count: 7
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 8.91108 (QuantReg: 15.12999) QuantErr: 15.12999 batch_time=16.02931
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 8.28392 (QuantReg: 15.08936) QuantErr: 15.08936 batch_time=0.60529
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 8.76417 (QuantReg: 14.91497) QuantErr: 14.91497 batch_time=0.62991
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 8.21286 (QuantReg: 15.20816) QuantErr: 15.20816 batch_time=0.59614
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 8.64304 (QuantReg: 14.89986) QuantErr: 14.89986 batch_time=0.61029
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 8.94984 (QuantReg: 15.07481) QuantErr: 15.07481 batch_time=0.61159
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 9.05601 (QuantReg: 15.14301) QuantErr: 15.14301 batch_time=0.60828
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 9.01465 (QuantReg: 15.04358) QuantErr: 15.04358 batch_time=1.10107
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 9.15671 (QuantReg: 15.04396) QuantErr: 15.04396 batch_time=0.61360
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 9.16819 (QuantReg: 15.11815) QuantErr: 15.11815 batch_time=0.61776
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 8.68681 (QuantReg: 15.02298) QuantErr: 15.02298 batch_time=0.62186
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 8.46416 (QuantReg: 15.26700) QuantErr: 15.26700 batch_time=0.61334
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 8.73122 (QuantReg: 15.35999) QuantErr: 15.35999 batch_time=0.60009
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 7.97505 (QuantReg: 15.03097) QuantErr: 15.03097 batch_time=2.50130
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 8.64121 (QuantReg: 15.09871) QuantErr: 15.09871 batch_time=0.61905
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 7.81922 (QuantReg: 15.40516) QuantErr: 15.40516 batch_time=0.61250
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 8.81807 (QuantReg: 15.41683) QuantErr: 15.41683 batch_time=0.62211
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 7.28024 (QuantReg: 15.42583) QuantErr: 15.42583 batch_time=0.61172
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 8.94115 (QuantReg: 15.53478) QuantErr: 15.53478 batch_time=0.61913
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 8.92296 (QuantReg: 15.02388) QuantErr: 15.02388 batch_time=0.61868
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 9.11698 (QuantReg: 15.24296) QuantErr: 15.24296 batch_time=0.62482
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 8.10173 (QuantReg: 15.27768) QuantErr: 15.27768 batch_time=0.61688
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 7.27042 (QuantReg: 15.27654) QuantErr: 15.27654 batch_time=0.61200
Train Epoch: 18 codebook_update_time=3.23683
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch18.pth ...
Done in 5.779s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 8.658070558547974
quant_reg : 15.202107158660889
quant_err : 15.202107158660889
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
LSMDC_full_test/t2v_metrics/R1: 12.0
LSMDC_full_test/t2v_metrics/R5: 33.0
LSMDC_full_test/t2v_metrics/R10: 42.3
LSMDC_full_test/t2v_metrics/R50: 69.2
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 72.181
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.586557069762687
LSMDC_full_test/v2t_metrics/R1: 11.9
LSMDC_full_test/v2t_metrics/R5: 32.2
LSMDC_full_test/v2t_metrics/R10: 41.5
LSMDC_full_test/v2t_metrics/R50: 68.5
LSMDC_full_test/v2t_metrics/MedR: 16.0
LSMDC_full_test/v2t_metrics/MeanR: 68.625
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.146853011979573
mnt_best : 25.71432430329307
not_improved_count: 8
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 8.48782 (QuantReg: 15.16522) QuantErr: 15.16522 batch_time=19.31856
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 7.22813 (QuantReg: 15.28173) QuantErr: 15.28173 batch_time=0.65235
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 9.74058 (QuantReg: 15.10396) QuantErr: 15.10396 batch_time=0.63933
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 10.50120 (QuantReg: 15.06824) QuantErr: 15.06824 batch_time=0.60365
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 8.10951 (QuantReg: 15.23589) QuantErr: 15.23589 batch_time=0.60764
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 9.85583 (QuantReg: 15.42675) QuantErr: 15.42675 batch_time=0.60536
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 8.07201 (QuantReg: 15.19647) QuantErr: 15.19647 batch_time=0.60799
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 8.07157 (QuantReg: 15.28503) QuantErr: 15.28503 batch_time=0.60422
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 9.00005 (QuantReg: 15.36798) QuantErr: 15.36798 batch_time=0.60670
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 7.71981 (QuantReg: 15.33989) QuantErr: 15.33989 batch_time=0.60424
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 7.87836 (QuantReg: 15.22970) QuantErr: 15.22970 batch_time=0.94474
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 8.48847 (QuantReg: 15.23149) QuantErr: 15.23149 batch_time=1.03116
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 10.15406 (QuantReg: 15.23305) QuantErr: 15.23305 batch_time=1.22446
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 8.72519 (QuantReg: 15.21094) QuantErr: 15.21094 batch_time=0.61728
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 8.57622 (QuantReg: 15.14652) QuantErr: 15.14652 batch_time=0.63151
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 7.77417 (QuantReg: 15.17074) QuantErr: 15.17074 batch_time=0.60332
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 10.08864 (QuantReg: 15.64270) QuantErr: 15.64270 batch_time=0.60594
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 8.14049 (QuantReg: 15.48792) QuantErr: 15.48792 batch_time=0.60474
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 7.05603 (QuantReg: 15.54663) QuantErr: 15.54663 batch_time=0.60874
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 8.29399 (QuantReg: 15.36980) QuantErr: 15.36980 batch_time=1.79434
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 9.15397 (QuantReg: 15.28071) QuantErr: 15.28071 batch_time=0.60452
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 9.11727 (QuantReg: 15.37587) QuantErr: 15.37587 batch_time=0.60122
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 8.82041 (QuantReg: 15.27108) QuantErr: 15.27108 batch_time=0.62814
Train Epoch: 19 codebook_update_time=3.25101
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_L15/checkpoint-epoch19.pth ...
Done in 4.196s
removing stale ckpt [epoch 18] [took 0.00s]
epoch : 19
loss : 8.375077934265137
quant_reg : 15.301295967102051
quant_err : 15.301295967102051
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
LSMDC_full_test/t2v_metrics/R1: 12.5
LSMDC_full_test/t2v_metrics/R5: 30.8
LSMDC_full_test/t2v_metrics/R10: 40.8
LSMDC_full_test/t2v_metrics/R50: 69.7
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 71.963
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.04418851565213
LSMDC_full_test/v2t_metrics/R1: 12.3
LSMDC_full_test/v2t_metrics/R5: 31.6
LSMDC_full_test/v2t_metrics/R10: 40.7
LSMDC_full_test/v2t_metrics/R50: 68.7
LSMDC_full_test/v2t_metrics/MedR: 18.0