-
Notifications
You must be signed in to change notification settings - Fork 4
/
HCQ_MSRVTT_full_t0.15.txt
3309 lines (3309 loc) · 233 KB
/
HCQ_MSRVTT_full_t0.15.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 1072.1231582164764 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 50.19671893119812 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 366.5434219837189 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 60.78925323486328 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch0.pth ...
Done in 1.847s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch0.pth ...
Done in 3.257s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
MSRVTT_full_val/t2v_metrics/MedR: 252.0
MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
MSRVTT_full_val/v2t_metrics/MedR: 243.0
MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
MSRVTT_full_test/t2v_metrics/MedR: 1515.0
MSRVTT_full_test/t2v_metrics/MeanR: 1498.5565217391304
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1471.5
MSRVTT_full_test/v2t_metrics/MeanR: 1495.3264214046824
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
mnt_best : 0.12154652794863813
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.71980 (QuantReg: 22.44377) QuantErr: 22.44377 batch_time=39.89561
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.85195 (QuantReg: 22.55325) QuantErr: 22.55325 batch_time=0.51681
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.36770 (QuantReg: 22.61552) QuantErr: 22.61552 batch_time=0.50155
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 7.21120 (QuantReg: 22.61015) QuantErr: 22.61015 batch_time=0.54296
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.73242 (QuantReg: 22.61219) QuantErr: 22.61219 batch_time=0.50793
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.47584 (QuantReg: 22.59515) QuantErr: 22.59515 batch_time=0.50933
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.45215 (QuantReg: 22.62016) QuantErr: 22.62016 batch_time=0.53975
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 6.18735 (QuantReg: 22.61082) QuantErr: 22.61082 batch_time=0.50017
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.97641 (QuantReg: 22.63460) QuantErr: 22.63460 batch_time=0.50294
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.02404 (QuantReg: 22.61957) QuantErr: 22.61957 batch_time=0.52662
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.75997 (QuantReg: 22.65413) QuantErr: 22.65413 batch_time=1.06623
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.60003 (QuantReg: 22.63161) QuantErr: 22.63161 batch_time=0.52704
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.56505 (QuantReg: 22.61616) QuantErr: 22.61616 batch_time=0.49868
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 5.48185 (QuantReg: 22.63235) QuantErr: 22.63235 batch_time=0.49919
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 5.49767 (QuantReg: 22.61691) QuantErr: 22.61691 batch_time=0.54870
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 5.63331 (QuantReg: 22.61498) QuantErr: 22.61498 batch_time=0.50558
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 5.46654 (QuantReg: 22.60433) QuantErr: 22.60433 batch_time=0.49204
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 5.21180 (QuantReg: 22.61203) QuantErr: 22.61203 batch_time=0.50412
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 5.23559 (QuantReg: 22.57929) QuantErr: 22.57929 batch_time=0.50963
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 5.05148 (QuantReg: 22.58998) QuantErr: 22.58998 batch_time=0.50222
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 5.23411 (QuantReg: 22.62759) QuantErr: 22.62759 batch_time=0.54205
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 5.15906 (QuantReg: 22.61801) QuantErr: 22.61801 batch_time=0.49781
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.85784 (QuantReg: 22.57612) QuantErr: 22.57612 batch_time=0.53121
Train Epoch: 1 codebook_update_time=1.74698
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch1.pth ...
Done in 4.112s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch1.pth ...
Done in 8.153s
epoch : 1
loss : 5.980801687240601
quant_reg : 22.60113138580322
quant_err : 22.60113138580322
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 16.498993963782695
MSRVTT_full_val/t2v_metrics/R5: 45.67404426559356
MSRVTT_full_val/t2v_metrics/R10: 61.56941649899397
MSRVTT_full_val/t2v_metrics/R50: 94.16498993963782
MSRVTT_full_val/t2v_metrics/MedR: 7.0
MSRVTT_full_val/t2v_metrics/MeanR: 16.080482897384307
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 35.93331820482901
MSRVTT_full_val/v2t_metrics/R1: 20.72434607645875
MSRVTT_full_val/v2t_metrics/R5: 50.503018108651915
MSRVTT_full_val/v2t_metrics/R10: 65.59356136820925
MSRVTT_full_val/v2t_metrics/R50: 93.56136820925553
MSRVTT_full_val/v2t_metrics/MedR: 5.0
MSRVTT_full_val/v2t_metrics/MeanR: 14.865191146881287
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 40.946783606268575
MSRVTT_full_test/t2v_metrics/R1: 5.117056856187291
MSRVTT_full_test/t2v_metrics/R5: 16.989966555183948
MSRVTT_full_test/t2v_metrics/R10: 26.622073578595316
MSRVTT_full_test/t2v_metrics/R50: 60.903010033444815
MSRVTT_full_test/t2v_metrics/MedR: 31.0
MSRVTT_full_test/t2v_metrics/MeanR: 87.74013377926421
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 13.227716607539838
MSRVTT_full_test/v2t_metrics/R1: 5.618729096989966
MSRVTT_full_test/v2t_metrics/R5: 18.26086956521739
MSRVTT_full_test/v2t_metrics/R10: 28.361204013377925
MSRVTT_full_test/v2t_metrics/R50: 64.04682274247492
MSRVTT_full_test/v2t_metrics/MedR: 29.0
MSRVTT_full_test/v2t_metrics/MeanR: 83.12140468227425
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 14.276707787678664
mnt_best : 13.227716607539838
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.80599 (QuantReg: 8.93690) QuantErr: 8.93690 batch_time=37.26274
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.86365 (QuantReg: 8.69148) QuantErr: 8.69148 batch_time=0.49905
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.84969 (QuantReg: 9.13306) QuantErr: 9.13306 batch_time=1.55224
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.73600 (QuantReg: 9.21926) QuantErr: 9.21926 batch_time=0.50782
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.76942 (QuantReg: 9.29076) QuantErr: 9.29076 batch_time=0.50231
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.61778 (QuantReg: 9.00625) QuantErr: 9.00625 batch_time=0.51719
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.92570 (QuantReg: 9.54889) QuantErr: 9.54889 batch_time=0.77187
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.67356 (QuantReg: 9.49042) QuantErr: 9.49042 batch_time=0.51220
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 5.02438 (QuantReg: 9.67104) QuantErr: 9.67104 batch_time=0.50411
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.64042 (QuantReg: 9.42451) QuantErr: 9.42451 batch_time=0.49991
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 4.58460 (QuantReg: 9.31475) QuantErr: 9.31475 batch_time=0.51309
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.55816 (QuantReg: 9.45503) QuantErr: 9.45503 batch_time=0.51296
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 4.29199 (QuantReg: 9.78026) QuantErr: 9.78026 batch_time=0.50891
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.42858 (QuantReg: 9.74329) QuantErr: 9.74329 batch_time=0.52729
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 4.32506 (QuantReg: 10.05841) QuantErr: 10.05841 batch_time=0.51863
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 4.48838 (QuantReg: 10.31563) QuantErr: 10.31563 batch_time=0.50773
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 4.52270 (QuantReg: 9.90040) QuantErr: 9.90040 batch_time=0.49990
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 4.55883 (QuantReg: 10.25563) QuantErr: 10.25563 batch_time=0.49759
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 4.15462 (QuantReg: 10.13753) QuantErr: 10.13753 batch_time=0.49257
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 4.34741 (QuantReg: 10.32012) QuantErr: 10.32012 batch_time=0.50113
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 4.12188 (QuantReg: 10.33519) QuantErr: 10.33519 batch_time=0.50045
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 4.69803 (QuantReg: 10.23737) QuantErr: 10.23737 batch_time=0.51584
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 4.55508 (QuantReg: 10.91043) QuantErr: 10.91043 batch_time=0.50243
Train Epoch: 2 codebook_update_time=1.63282
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch2.pth ...
Done in 4.433s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch2.pth ...
Done in 8.650s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 4.598513472557068
quant_reg : 9.676484245300292
quant_err : 9.676484245300292
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 16.70020120724346
MSRVTT_full_val/t2v_metrics/R5: 50.70422535211268
MSRVTT_full_val/t2v_metrics/R10: 65.79476861167002
MSRVTT_full_val/t2v_metrics/R50: 94.76861167002012
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 13.70824949698189
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 38.193172984595954
MSRVTT_full_val/v2t_metrics/R1: 21.730382293762574
MSRVTT_full_val/v2t_metrics/R5: 53.92354124748491
MSRVTT_full_val/v2t_metrics/R10: 73.44064386317908
MSRVTT_full_val/v2t_metrics/R50: 94.96981891348088
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 12.35010060362173
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 44.14966534047218
MSRVTT_full_test/t2v_metrics/R1: 5.8193979933110365
MSRVTT_full_test/t2v_metrics/R5: 20.367892976588628
MSRVTT_full_test/t2v_metrics/R10: 31.036789297658864
MSRVTT_full_test/t2v_metrics/R50: 66.82274247491638
MSRVTT_full_test/t2v_metrics/MedR: 24.0
MSRVTT_full_test/t2v_metrics/MeanR: 71.5304347826087
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 15.43714502060139
MSRVTT_full_test/v2t_metrics/R1: 6.989966555183947
MSRVTT_full_test/v2t_metrics/R5: 23.210702341137125
MSRVTT_full_test/v2t_metrics/R10: 36.08695652173913
MSRVTT_full_test/v2t_metrics/R50: 70.50167224080268
MSRVTT_full_test/v2t_metrics/MedR: 20.0
MSRVTT_full_test/v2t_metrics/MeanR: 64.14414715719064
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 18.023448036368976
mnt_best : 15.43714502060139
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 4.15265 (QuantReg: 7.77105) QuantErr: 7.77105 batch_time=43.18463
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 4.47773 (QuantReg: 8.03559) QuantErr: 8.03559 batch_time=0.48996
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 4.57136 (QuantReg: 8.36885) QuantErr: 8.36885 batch_time=0.48862
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 4.06489 (QuantReg: 7.38800) QuantErr: 7.38800 batch_time=0.53601
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 4.37694 (QuantReg: 7.85654) QuantErr: 7.85654 batch_time=0.51228
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 4.01733 (QuantReg: 7.78442) QuantErr: 7.78442 batch_time=0.52500
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 4.13443 (QuantReg: 7.78326) QuantErr: 7.78326 batch_time=0.50456
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 4.29860 (QuantReg: 7.73466) QuantErr: 7.73466 batch_time=0.62772
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 4.16914 (QuantReg: 7.75987) QuantErr: 7.75987 batch_time=0.50581
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 4.01711 (QuantReg: 7.99281) QuantErr: 7.99281 batch_time=0.49397
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 4.18565 (QuantReg: 8.07344) QuantErr: 8.07344 batch_time=0.49702
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.84076 (QuantReg: 8.06070) QuantErr: 8.06070 batch_time=0.49201
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 4.39604 (QuantReg: 8.28122) QuantErr: 8.28122 batch_time=0.48932
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 4.13635 (QuantReg: 8.56837) QuantErr: 8.56837 batch_time=0.49629
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.64793 (QuantReg: 8.23061) QuantErr: 8.23061 batch_time=0.49871
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.82570 (QuantReg: 8.56941) QuantErr: 8.56941 batch_time=1.15517
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.74509 (QuantReg: 8.26239) QuantErr: 8.26239 batch_time=0.50058
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 4.20095 (QuantReg: 8.27060) QuantErr: 8.27060 batch_time=0.51863
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.81882 (QuantReg: 8.23035) QuantErr: 8.23035 batch_time=0.49834
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 4.25564 (QuantReg: 8.45681) QuantErr: 8.45681 batch_time=0.49267
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.82808 (QuantReg: 8.57736) QuantErr: 8.57736 batch_time=0.51540
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.74846 (QuantReg: 8.65932) QuantErr: 8.65932 batch_time=0.51707
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.95860 (QuantReg: 8.64468) QuantErr: 8.64468 batch_time=0.50266
Train Epoch: 3 codebook_update_time=1.87990
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch3.pth ...
Done in 4.216s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch3.pth ...
Done in 8.661s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 4.154468665122986
quant_reg : 8.181871543884277
quant_err : 8.181871543884277
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 23.138832997987926
MSRVTT_full_val/t2v_metrics/R5: 56.74044265593562
MSRVTT_full_val/t2v_metrics/R10: 70.4225352112676
MSRVTT_full_val/t2v_metrics/R50: 95.57344064386318
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 12.267605633802816
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 45.21840878005637
MSRVTT_full_val/v2t_metrics/R1: 24.346076458752513
MSRVTT_full_val/v2t_metrics/R5: 61.3682092555332
MSRVTT_full_val/v2t_metrics/R10: 75.25150905432595
MSRVTT_full_val/v2t_metrics/R50: 95.57344064386318
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 10.953722334004024
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.2646560067785
MSRVTT_full_test/t2v_metrics/R1: 8.193979933110368
MSRVTT_full_test/t2v_metrics/R5: 24.782608695652176
MSRVTT_full_test/t2v_metrics/R10: 36.8561872909699
MSRVTT_full_test/t2v_metrics/R50: 71.83946488294315
MSRVTT_full_test/t2v_metrics/MedR: 19.0
MSRVTT_full_test/t2v_metrics/MeanR: 60.56421404682274
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.5606871584151
MSRVTT_full_test/v2t_metrics/R1: 8.361204013377927
MSRVTT_full_test/v2t_metrics/R5: 26.65551839464883
MSRVTT_full_test/v2t_metrics/R10: 39.23076923076923
MSRVTT_full_test/v2t_metrics/R50: 75.01672240802675
MSRVTT_full_test/v2t_metrics/MedR: 17.0
MSRVTT_full_test/v2t_metrics/MeanR: 59.00752508361204
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.601282573467344
mnt_best : 19.5606871584151
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 4.34748 (QuantReg: 7.34884) QuantErr: 7.34884 batch_time=41.53302
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 4.03640 (QuantReg: 7.34143) QuantErr: 7.34143 batch_time=0.50620
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 3.72095 (QuantReg: 7.39296) QuantErr: 7.39296 batch_time=0.50444
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.86138 (QuantReg: 7.31603) QuantErr: 7.31603 batch_time=0.50891
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 4.06921 (QuantReg: 7.37367) QuantErr: 7.37367 batch_time=0.51294
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 4.21882 (QuantReg: 7.39543) QuantErr: 7.39543 batch_time=0.48711
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 4.06902 (QuantReg: 7.45524) QuantErr: 7.45524 batch_time=0.50183
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 4.03109 (QuantReg: 7.43747) QuantErr: 7.43747 batch_time=0.49952
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.83535 (QuantReg: 7.33505) QuantErr: 7.33505 batch_time=0.50440
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.76933 (QuantReg: 7.42062) QuantErr: 7.42062 batch_time=0.50400
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.96317 (QuantReg: 7.68034) QuantErr: 7.68034 batch_time=0.76031
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.97566 (QuantReg: 7.32546) QuantErr: 7.32546 batch_time=0.50307
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.80945 (QuantReg: 7.40163) QuantErr: 7.40163 batch_time=0.48711
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 3.99487 (QuantReg: 7.94741) QuantErr: 7.94741 batch_time=0.50662
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.87656 (QuantReg: 7.85691) QuantErr: 7.85691 batch_time=0.50745
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 4.11499 (QuantReg: 8.18011) QuantErr: 8.18011 batch_time=0.50202
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 4.08940 (QuantReg: 7.90070) QuantErr: 7.90070 batch_time=0.50356
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 4.00631 (QuantReg: 8.17914) QuantErr: 8.17914 batch_time=0.50654
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 3.89514 (QuantReg: 7.80358) QuantErr: 7.80358 batch_time=0.50267
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 3.75872 (QuantReg: 7.78012) QuantErr: 7.78012 batch_time=0.52434
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 3.76781 (QuantReg: 8.15564) QuantErr: 8.15564 batch_time=1.03732
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 3.92667 (QuantReg: 7.79736) QuantErr: 7.79736 batch_time=0.51547
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 3.76574 (QuantReg: 7.77117) QuantErr: 7.77117 batch_time=0.50239
Train Epoch: 4 codebook_update_time=1.83728
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch4.pth ...
Done in 4.615s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 3.9034011993408204
quant_reg : 7.68858025932312
quant_err : 7.68858025932312
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 21.52917505030181
MSRVTT_full_val/t2v_metrics/R5: 53.521126760563384
MSRVTT_full_val/t2v_metrics/R10: 69.81891348088531
MSRVTT_full_val/t2v_metrics/R50: 95.57344064386318
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 13.078470824949699
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 43.16932330350488
MSRVTT_full_val/v2t_metrics/R1: 23.34004024144869
MSRVTT_full_val/v2t_metrics/R5: 61.77062374245473
MSRVTT_full_val/v2t_metrics/R10: 75.25150905432595
MSRVTT_full_val/v2t_metrics/R50: 96.17706237424547
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 10.69215291750503
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 47.69427669476766
MSRVTT_full_test/t2v_metrics/R1: 7.357859531772576
MSRVTT_full_test/t2v_metrics/R5: 23.812709030100333
MSRVTT_full_test/t2v_metrics/R10: 35.51839464882943
MSRVTT_full_test/t2v_metrics/R50: 70.20066889632108
MSRVTT_full_test/t2v_metrics/MedR: 20.0
MSRVTT_full_test/t2v_metrics/MeanR: 66.13344481605351
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.393789358849553
MSRVTT_full_test/v2t_metrics/R1: 8.929765886287626
MSRVTT_full_test/v2t_metrics/R5: 26.68896321070234
MSRVTT_full_test/v2t_metrics/R10: 39.39799331103679
MSRVTT_full_test/v2t_metrics/R50: 75.31772575250837
MSRVTT_full_test/v2t_metrics/MedR: 17.0
MSRVTT_full_test/v2t_metrics/MeanR: 58.095652173913045
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.096737189526046
mnt_best : 19.5606871584151
not_improved_count: 1
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.93718 (QuantReg: 7.42785) QuantErr: 7.42785 batch_time=34.74557
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.93427 (QuantReg: 7.34909) QuantErr: 7.34909 batch_time=0.49434
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 4.00329 (QuantReg: 7.36771) QuantErr: 7.36771 batch_time=0.49503
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 3.37238 (QuantReg: 7.76951) QuantErr: 7.76951 batch_time=0.49366
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 3.80719 (QuantReg: 7.29960) QuantErr: 7.29960 batch_time=0.49404
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 3.85595 (QuantReg: 7.77625) QuantErr: 7.77625 batch_time=0.52487
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 3.70333 (QuantReg: 7.43436) QuantErr: 7.43436 batch_time=0.53697
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.74419 (QuantReg: 7.15678) QuantErr: 7.15678 batch_time=0.52727
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 3.49349 (QuantReg: 7.60280) QuantErr: 7.60280 batch_time=0.49359
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 3.90931 (QuantReg: 7.34971) QuantErr: 7.34971 batch_time=0.65703
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 3.78285 (QuantReg: 7.53913) QuantErr: 7.53913 batch_time=0.49064
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 3.68614 (QuantReg: 7.38876) QuantErr: 7.38876 batch_time=0.48915
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 3.67955 (QuantReg: 7.62714) QuantErr: 7.62714 batch_time=0.50608
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 3.30855 (QuantReg: 7.63469) QuantErr: 7.63469 batch_time=0.50645
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 3.72163 (QuantReg: 7.86269) QuantErr: 7.86269 batch_time=0.50850
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 3.67747 (QuantReg: 7.75086) QuantErr: 7.75086 batch_time=0.49946
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 3.65081 (QuantReg: 7.90486) QuantErr: 7.90486 batch_time=0.49315
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 3.72766 (QuantReg: 7.50088) QuantErr: 7.50088 batch_time=0.49159
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 3.52279 (QuantReg: 7.49812) QuantErr: 7.49812 batch_time=0.72020
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 3.72127 (QuantReg: 7.50700) QuantErr: 7.50700 batch_time=2.77023
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 3.68901 (QuantReg: 7.62895) QuantErr: 7.62895 batch_time=0.50577
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 3.39699 (QuantReg: 7.69105) QuantErr: 7.69105 batch_time=0.49186
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.56347 (QuantReg: 7.51828) QuantErr: 7.51828 batch_time=0.50592
Train Epoch: 5 codebook_update_time=1.64931
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch5.pth ...
Done in 6.154s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch5.pth ...
Done in 10.779s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 3.682996241569519
quant_reg : 7.488429651260376
quant_err : 7.488429651260376
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 24.14486921529175
MSRVTT_full_val/t2v_metrics/R5: 55.734406438631794
MSRVTT_full_val/t2v_metrics/R10: 72.43460764587525
MSRVTT_full_val/t2v_metrics/R50: 95.57344064386318
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.818913480885312
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.02192530648933
MSRVTT_full_val/v2t_metrics/R1: 24.14486921529175
MSRVTT_full_val/v2t_metrics/R5: 62.17303822937626
MSRVTT_full_val/v2t_metrics/R10: 76.86116700201207
MSRVTT_full_val/v2t_metrics/R50: 96.17706237424547
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.953722334004024
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.683071863379155
MSRVTT_full_test/t2v_metrics/R1: 8.327759197324415
MSRVTT_full_test/t2v_metrics/R5: 25.7190635451505
MSRVTT_full_test/t2v_metrics/R10: 38.22742474916388
MSRVTT_full_test/t2v_metrics/R50: 72.74247491638796
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 58.969565217391306
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.155153842814332
MSRVTT_full_test/v2t_metrics/R1: 9.498327759197325
MSRVTT_full_test/v2t_metrics/R5: 28.127090301003346
MSRVTT_full_test/v2t_metrics/R10: 41.23745819397993
MSRVTT_full_test/v2t_metrics/R50: 76.88963210702342
MSRVTT_full_test/v2t_metrics/MedR: 15.0
MSRVTT_full_test/v2t_metrics/MeanR: 53.79247491638796
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.251260381546057
mnt_best : 20.155153842814332
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 3.77921 (QuantReg: 6.95490) QuantErr: 6.95490 batch_time=33.68612
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 3.75697 (QuantReg: 6.71087) QuantErr: 6.71087 batch_time=0.74623
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 3.33862 (QuantReg: 7.14453) QuantErr: 7.14453 batch_time=4.13271
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 3.48619 (QuantReg: 7.21054) QuantErr: 7.21054 batch_time=1.02203
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 3.21534 (QuantReg: 7.04239) QuantErr: 7.04239 batch_time=0.49378
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 3.40786 (QuantReg: 7.41000) QuantErr: 7.41000 batch_time=0.49241
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 3.40329 (QuantReg: 7.43735) QuantErr: 7.43735 batch_time=0.50281
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 3.26153 (QuantReg: 7.10493) QuantErr: 7.10493 batch_time=0.54003
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 3.85153 (QuantReg: 7.40626) QuantErr: 7.40626 batch_time=0.51120
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 3.47366 (QuantReg: 7.20345) QuantErr: 7.20345 batch_time=0.49804
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 3.43323 (QuantReg: 6.62281) QuantErr: 6.62281 batch_time=0.50959
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 3.36810 (QuantReg: 7.31895) QuantErr: 7.31895 batch_time=0.49390
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 3.41362 (QuantReg: 7.24448) QuantErr: 7.24448 batch_time=0.51485
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 3.43000 (QuantReg: 6.95406) QuantErr: 6.95406 batch_time=0.49876
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 3.26935 (QuantReg: 7.31224) QuantErr: 7.31224 batch_time=0.50606
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 3.27773 (QuantReg: 7.12463) QuantErr: 7.12463 batch_time=0.49206
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 3.69516 (QuantReg: 7.50438) QuantErr: 7.50438 batch_time=0.50588
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 3.51124 (QuantReg: 7.80155) QuantErr: 7.80155 batch_time=0.50166
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 3.48617 (QuantReg: 7.19032) QuantErr: 7.19032 batch_time=0.49891
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 3.48059 (QuantReg: 7.48277) QuantErr: 7.48277 batch_time=0.85314
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 3.39113 (QuantReg: 7.31902) QuantErr: 7.31902 batch_time=0.50092
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 3.64324 (QuantReg: 7.35715) QuantErr: 7.35715 batch_time=0.49242
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 3.50641 (QuantReg: 7.40096) QuantErr: 7.40096 batch_time=0.49288
Train Epoch: 6 codebook_update_time=1.82410
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch6.pth ...
Done in 6.655s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch6.pth ...
Done in 11.895s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 3.517167056083679
quant_reg : 7.2566107063293455
quant_err : 7.2566107063293455
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 24.949698189134807
MSRVTT_full_val/t2v_metrics/R5: 59.15492957746479
MSRVTT_full_val/t2v_metrics/R10: 73.2394366197183
MSRVTT_full_val/t2v_metrics/R50: 95.37223340040241
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.507042253521126
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 47.63583083752205
MSRVTT_full_val/v2t_metrics/R1: 29.77867203219316
MSRVTT_full_val/v2t_metrics/R5: 63.17907444668008
MSRVTT_full_val/v2t_metrics/R10: 76.05633802816901
MSRVTT_full_val/v2t_metrics/R50: 96.17706237424547
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 10.0261569416499
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 52.30437284203782
MSRVTT_full_test/t2v_metrics/R1: 8.963210702341136
MSRVTT_full_test/t2v_metrics/R5: 26.354515050167223
MSRVTT_full_test/t2v_metrics/R10: 38.96321070234114
MSRVTT_full_test/t2v_metrics/R50: 73.67892976588628
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 58.829765886287625
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.95677532026571
MSRVTT_full_test/v2t_metrics/R1: 9.464882943143813
MSRVTT_full_test/v2t_metrics/R5: 28.729096989966557
MSRVTT_full_test/v2t_metrics/R10: 41.67224080267559
MSRVTT_full_test/v2t_metrics/R50: 76.82274247491638
MSRVTT_full_test/v2t_metrics/MedR: 15.0
MSRVTT_full_test/v2t_metrics/MeanR: 53.30117056856187
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.460945060723205
mnt_best : 20.95677532026571
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 3.38228 (QuantReg: 6.58888) QuantErr: 6.58888 batch_time=33.23573
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 3.33196 (QuantReg: 7.43472) QuantErr: 7.43472 batch_time=0.53173
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 3.49908 (QuantReg: 7.34178) QuantErr: 7.34178 batch_time=0.48898
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 3.31943 (QuantReg: 7.28068) QuantErr: 7.28068 batch_time=0.48988
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 3.26083 (QuantReg: 6.74630) QuantErr: 6.74630 batch_time=0.50128
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 3.56537 (QuantReg: 6.85206) QuantErr: 6.85206 batch_time=0.52019
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 3.49358 (QuantReg: 7.23781) QuantErr: 7.23781 batch_time=0.50242
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 3.74996 (QuantReg: 7.31198) QuantErr: 7.31198 batch_time=0.49047
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 3.49473 (QuantReg: 7.29420) QuantErr: 7.29420 batch_time=0.63404
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 3.38944 (QuantReg: 7.33378) QuantErr: 7.33378 batch_time=0.56309
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 3.50655 (QuantReg: 7.29609) QuantErr: 7.29609 batch_time=0.52394
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 3.05021 (QuantReg: 7.65966) QuantErr: 7.65966 batch_time=0.50320
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 3.30137 (QuantReg: 7.09563) QuantErr: 7.09563 batch_time=1.20057
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 3.46413 (QuantReg: 7.28225) QuantErr: 7.28225 batch_time=1.14872
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 3.41473 (QuantReg: 6.94918) QuantErr: 6.94918 batch_time=0.49651
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 3.28326 (QuantReg: 7.00412) QuantErr: 7.00412 batch_time=0.48709
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 3.09899 (QuantReg: 7.29438) QuantErr: 7.29438 batch_time=0.51813
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 3.39965 (QuantReg: 6.84342) QuantErr: 6.84342 batch_time=0.50683
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.95492 (QuantReg: 7.40744) QuantErr: 7.40744 batch_time=0.49024
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 3.19325 (QuantReg: 7.23470) QuantErr: 7.23470 batch_time=0.49156
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 3.67165 (QuantReg: 7.27642) QuantErr: 7.27642 batch_time=0.50361
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 3.32082 (QuantReg: 7.04107) QuantErr: 7.04107 batch_time=0.49989
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 3.32756 (QuantReg: 7.13990) QuantErr: 7.13990 batch_time=0.49890
Train Epoch: 7 codebook_update_time=1.68247
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch7.pth ...
Done in 4.915s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch7.pth ...
Done in 9.729s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 3.3896362657547
quant_reg : 7.198209774017334
quant_err : 7.198209774017334
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 24.748490945674043
MSRVTT_full_val/t2v_metrics/R5: 59.95975855130785
MSRVTT_full_val/t2v_metrics/R10: 71.83098591549296
MSRVTT_full_val/t2v_metrics/R50: 95.77464788732394
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.364185110663984
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 47.41402326619343
MSRVTT_full_val/v2t_metrics/R1: 26.358148893360163
MSRVTT_full_val/v2t_metrics/R5: 63.17907444668008
MSRVTT_full_val/v2t_metrics/R10: 75.25150905432595
MSRVTT_full_val/v2t_metrics/R50: 96.37826961770624
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 10.116700201207243
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 50.04197710948591
MSRVTT_full_test/t2v_metrics/R1: 8.561872909698996
MSRVTT_full_test/t2v_metrics/R5: 27.45819397993311
MSRVTT_full_test/t2v_metrics/R10: 40.03344481605351
MSRVTT_full_test/t2v_metrics/R50: 74.04682274247492
MSRVTT_full_test/t2v_metrics/MedR: 17.0
MSRVTT_full_test/t2v_metrics/MeanR: 58.01220735785953
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.11322467717102
MSRVTT_full_test/v2t_metrics/R1: 9.464882943143813
MSRVTT_full_test/v2t_metrics/R5: 30.234113712374583
MSRVTT_full_test/v2t_metrics/R10: 42.876254180602004
MSRVTT_full_test/v2t_metrics/R50: 77.82608695652173
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 53.62658862876254
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.064450316063336
mnt_best : 21.11322467717102
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 3.37515 (QuantReg: 7.38991) QuantErr: 7.38991 batch_time=38.02983
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.44084 (QuantReg: 7.24196) QuantErr: 7.24196 batch_time=0.50524
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 3.61112 (QuantReg: 7.03379) QuantErr: 7.03379 batch_time=2.10483
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 3.29328 (QuantReg: 7.34944) QuantErr: 7.34944 batch_time=0.52335
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 3.63177 (QuantReg: 7.24476) QuantErr: 7.24476 batch_time=0.54117
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 3.52044 (QuantReg: 7.11571) QuantErr: 7.11571 batch_time=0.50098
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 3.22770 (QuantReg: 7.43191) QuantErr: 7.43191 batch_time=0.51230
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 3.14738 (QuantReg: 7.41061) QuantErr: 7.41061 batch_time=0.55356
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 3.60759 (QuantReg: 7.40740) QuantErr: 7.40740 batch_time=0.50540
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 3.15083 (QuantReg: 6.89683) QuantErr: 6.89683 batch_time=0.50923
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 3.31703 (QuantReg: 7.28928) QuantErr: 7.28928 batch_time=0.49777
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 3.57752 (QuantReg: 7.21335) QuantErr: 7.21335 batch_time=0.49065
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 3.22428 (QuantReg: 7.13583) QuantErr: 7.13583 batch_time=0.53998
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 3.06091 (QuantReg: 7.07867) QuantErr: 7.07867 batch_time=2.13574
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 3.50426 (QuantReg: 7.35705) QuantErr: 7.35705 batch_time=0.49710
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 3.18297 (QuantReg: 7.08635) QuantErr: 7.08635 batch_time=0.54204
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 3.37645 (QuantReg: 7.22483) QuantErr: 7.22483 batch_time=0.49717
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 3.52000 (QuantReg: 7.46441) QuantErr: 7.46441 batch_time=1.10094
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.95634 (QuantReg: 7.25012) QuantErr: 7.25012 batch_time=1.12538
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.96925 (QuantReg: 7.04629) QuantErr: 7.04629 batch_time=0.52352
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 3.05393 (QuantReg: 7.54748) QuantErr: 7.54748 batch_time=0.50219
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 2.75112 (QuantReg: 7.00264) QuantErr: 7.00264 batch_time=0.52310
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.84784 (QuantReg: 7.07934) QuantErr: 7.07934 batch_time=0.50997
Train Epoch: 8 codebook_update_time=1.64044
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch8.pth ...
Done in 4.446s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 3.306207092285156
quant_reg : 7.139089624404908
quant_err : 7.139089624404908
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 22.132796780684103
MSRVTT_full_val/t2v_metrics/R5: 59.15492957746479
MSRVTT_full_val/t2v_metrics/R10: 71.42857142857143
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.235412474849095
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 45.39064988166291
MSRVTT_full_val/v2t_metrics/R1: 23.943661971830984
MSRVTT_full_val/v2t_metrics/R5: 63.17907444668008
MSRVTT_full_val/v2t_metrics/R10: 76.25754527162978
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.841046277665995
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.67981111253257
MSRVTT_full_test/t2v_metrics/R1: 8.695652173913043
MSRVTT_full_test/t2v_metrics/R5: 27.49163879598662
MSRVTT_full_test/t2v_metrics/R10: 39.297658862876254
MSRVTT_full_test/t2v_metrics/R50: 73.54515050167224
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 57.55150501672241
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.10035792946751
MSRVTT_full_test/v2t_metrics/R1: 9.397993311036789
MSRVTT_full_test/v2t_metrics/R5: 28.494983277591974
MSRVTT_full_test/v2t_metrics/R10: 41.67224080267559
MSRVTT_full_test/v2t_metrics/R50: 77.52508361204013
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 51.47742474916388
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.34687487751429
mnt_best : 21.11322467717102
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 3.30861 (QuantReg: 6.70388) QuantErr: 6.70388 batch_time=39.23145
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 3.04909 (QuantReg: 6.77814) QuantErr: 6.77814 batch_time=0.51765
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 3.35671 (QuantReg: 6.97213) QuantErr: 6.97213 batch_time=0.50822
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 3.45221 (QuantReg: 7.03738) QuantErr: 7.03738 batch_time=0.50369
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 3.50453 (QuantReg: 6.78877) QuantErr: 6.78877 batch_time=0.50677
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 3.46782 (QuantReg: 7.23554) QuantErr: 7.23554 batch_time=0.50073
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 3.15138 (QuantReg: 7.13071) QuantErr: 7.13071 batch_time=0.51718
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 3.01827 (QuantReg: 7.05359) QuantErr: 7.05359 batch_time=0.51656
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 3.32108 (QuantReg: 6.80154) QuantErr: 6.80154 batch_time=0.49257
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 3.23644 (QuantReg: 6.88372) QuantErr: 6.88372 batch_time=0.49223
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 3.22980 (QuantReg: 7.03508) QuantErr: 7.03508 batch_time=0.50465
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 3.17072 (QuantReg: 7.09756) QuantErr: 7.09756 batch_time=0.52273
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 3.28836 (QuantReg: 7.20635) QuantErr: 7.20635 batch_time=0.51575
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 3.31384 (QuantReg: 7.13113) QuantErr: 7.13113 batch_time=0.54778
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 3.24187 (QuantReg: 7.47392) QuantErr: 7.47392 batch_time=0.49555
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 3.41031 (QuantReg: 6.93121) QuantErr: 6.93121 batch_time=0.49682
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.89664 (QuantReg: 7.08065) QuantErr: 7.08065 batch_time=0.50896
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 3.02996 (QuantReg: 6.96301) QuantErr: 6.96301 batch_time=0.49769
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.99549 (QuantReg: 7.22959) QuantErr: 7.22959 batch_time=0.49705
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 3.14304 (QuantReg: 6.97511) QuantErr: 6.97511 batch_time=0.50919
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 3.22639 (QuantReg: 7.08047) QuantErr: 7.08047 batch_time=0.50147
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 3.26344 (QuantReg: 7.08806) QuantErr: 7.08806 batch_time=0.55122
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 2.79373 (QuantReg: 7.14815) QuantErr: 7.14815 batch_time=0.52405
Train Epoch: 9 codebook_update_time=1.94174
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch9.pth ...
Done in 20.856s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 3.199481548309326
quant_reg : 7.071276475906372
quant_err : 7.071276475906372
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 24.547283702213278
MSRVTT_full_val/t2v_metrics/R5: 58.95372233400403
MSRVTT_full_val/t2v_metrics/R10: 71.0261569416499
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.191146881287727
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.842959978220726
MSRVTT_full_val/v2t_metrics/R1: 22.93762575452716
MSRVTT_full_val/v2t_metrics/R5: 63.38028169014085
MSRVTT_full_val/v2t_metrics/R10: 78.06841046277665
MSRVTT_full_val/v2t_metrics/R50: 96.17706237424547
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.740442655935613
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.416417891595756
MSRVTT_full_test/t2v_metrics/R1: 8.66220735785953
MSRVTT_full_test/t2v_metrics/R5: 26.923076923076923
MSRVTT_full_test/t2v_metrics/R10: 40.10033444816054
MSRVTT_full_test/t2v_metrics/R50: 75.01672240802675
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 57.012040133779266
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.068506678095225
MSRVTT_full_test/v2t_metrics/R1: 9.565217391304348
MSRVTT_full_test/v2t_metrics/R5: 29.331103678929765
MSRVTT_full_test/v2t_metrics/R10: 42.74247491638796
MSRVTT_full_test/v2t_metrics/R50: 77.59197324414716
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 52.56605351170568
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.88904318020072
mnt_best : 21.11322467717102
not_improved_count: 2
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 2.94157 (QuantReg: 6.57951) QuantErr: 6.57951 batch_time=34.77409
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 3.05040 (QuantReg: 6.98982) QuantErr: 6.98982 batch_time=0.50329
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.96333 (QuantReg: 6.67600) QuantErr: 6.67600 batch_time=0.49582
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 3.01911 (QuantReg: 6.75033) QuantErr: 6.75033 batch_time=0.49900
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 3.07335 (QuantReg: 7.20976) QuantErr: 7.20976 batch_time=0.49768
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 3.36524 (QuantReg: 6.99350) QuantErr: 6.99350 batch_time=0.50170
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.96606 (QuantReg: 6.93462) QuantErr: 6.93462 batch_time=0.56377
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 3.01489 (QuantReg: 6.79505) QuantErr: 6.79505 batch_time=1.89227
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 3.23295 (QuantReg: 7.05604) QuantErr: 7.05604 batch_time=0.50103
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 3.45297 (QuantReg: 6.73645) QuantErr: 6.73645 batch_time=0.54317
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.90508 (QuantReg: 6.99706) QuantErr: 6.99706 batch_time=0.52840
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 3.22559 (QuantReg: 7.11338) QuantErr: 7.11338 batch_time=0.57338
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 3.23721 (QuantReg: 7.03380) QuantErr: 7.03380 batch_time=0.97982
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 3.44165 (QuantReg: 6.99943) QuantErr: 6.99943 batch_time=1.08542
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 2.86296 (QuantReg: 7.09146) QuantErr: 7.09146 batch_time=0.49875
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.81544 (QuantReg: 6.94550) QuantErr: 6.94550 batch_time=0.50934
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.92630 (QuantReg: 7.12082) QuantErr: 7.12082 batch_time=0.49530
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.92608 (QuantReg: 7.15204) QuantErr: 7.15204 batch_time=0.50686
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 3.03272 (QuantReg: 7.46787) QuantErr: 7.46787 batch_time=0.50621
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.92182 (QuantReg: 6.94975) QuantErr: 6.94975 batch_time=1.70003
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.83570 (QuantReg: 6.67651) QuantErr: 6.67651 batch_time=0.49530
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 3.53930 (QuantReg: 7.11345) QuantErr: 7.11345 batch_time=0.50106
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 3.12736 (QuantReg: 6.88202) QuantErr: 6.88202 batch_time=0.49627
Train Epoch: 10 codebook_update_time=2.03381
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch10.pth ...
Done in 22.337s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 3.110167073249817
quant_reg : 6.984330781936645
quant_err : 6.984330781936645
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 24.346076458752513
MSRVTT_full_val/t2v_metrics/R5: 58.75251509054326
MSRVTT_full_val/t2v_metrics/R10: 73.2394366197183
MSRVTT_full_val/t2v_metrics/R50: 95.97585513078471
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.653923541247485
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 47.14114712896537
MSRVTT_full_val/v2t_metrics/R1: 24.346076458752513
MSRVTT_full_val/v2t_metrics/R5: 61.77062374245473
MSRVTT_full_val/v2t_metrics/R10: 75.65392354124748
MSRVTT_full_val/v2t_metrics/R50: 96.579476861167
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.841046277665995
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.45599042529375
MSRVTT_full_test/t2v_metrics/R1: 8.461538461538462
MSRVTT_full_test/t2v_metrics/R5: 27.123745819397993
MSRVTT_full_test/t2v_metrics/R10: 39.59866220735786
MSRVTT_full_test/t2v_metrics/R50: 73.9799331103679
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 59.37826086956522
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.868593088232437
MSRVTT_full_test/v2t_metrics/R1: 9.431438127090301
MSRVTT_full_test/v2t_metrics/R5: 29.096989966555185
MSRVTT_full_test/v2t_metrics/R10: 42.642140468227424
MSRVTT_full_test/v2t_metrics/R50: 77.09030100334448
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 53.57374581939799
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.70326544467121
mnt_best : 21.11322467717102
not_improved_count: 3
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.97706 (QuantReg: 6.94210) QuantErr: 6.94210 batch_time=34.66133
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.81618 (QuantReg: 7.04720) QuantErr: 7.04720 batch_time=0.50237
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 3.07220 (QuantReg: 6.63745) QuantErr: 6.63745 batch_time=0.48878
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 3.12037 (QuantReg: 7.03701) QuantErr: 7.03701 batch_time=0.52066
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 3.20908 (QuantReg: 7.30381) QuantErr: 7.30381 batch_time=0.51880
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 3.04699 (QuantReg: 6.89558) QuantErr: 6.89558 batch_time=0.53772
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 3.12282 (QuantReg: 6.72826) QuantErr: 6.72826 batch_time=4.21673
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.84884 (QuantReg: 6.99554) QuantErr: 6.99554 batch_time=0.50146
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 3.09453 (QuantReg: 6.74710) QuantErr: 6.74710 batch_time=0.51242
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 3.40247 (QuantReg: 6.81929) QuantErr: 6.81929 batch_time=0.50672
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 3.21841 (QuantReg: 6.65894) QuantErr: 6.65894 batch_time=0.50105
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 3.11645 (QuantReg: 7.17621) QuantErr: 7.17621 batch_time=0.50840
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 3.42628 (QuantReg: 6.76453) QuantErr: 6.76453 batch_time=0.50385
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 3.25992 (QuantReg: 7.01387) QuantErr: 7.01387 batch_time=0.50339
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 3.00094 (QuantReg: 6.58572) QuantErr: 6.58572 batch_time=0.51180
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.99864 (QuantReg: 6.58507) QuantErr: 6.58507 batch_time=0.53173
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.98020 (QuantReg: 6.68885) QuantErr: 6.68885 batch_time=0.50726
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 2.84475 (QuantReg: 6.87744) QuantErr: 6.87744 batch_time=0.58918
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 2.94750 (QuantReg: 7.34649) QuantErr: 7.34649 batch_time=0.51290
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 3.30315 (QuantReg: 7.09049) QuantErr: 7.09049 batch_time=0.50353
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 3.14666 (QuantReg: 7.24463) QuantErr: 7.24463 batch_time=0.50208
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.95714 (QuantReg: 7.07733) QuantErr: 7.07733 batch_time=0.52540
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 2.87285 (QuantReg: 7.31442) QuantErr: 7.31442 batch_time=0.50102
Train Epoch: 11 codebook_update_time=1.86478
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch11.pth ...
Done in 4.039s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch11.pth ...
Done in 7.958s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 3.0403765058517456
quant_reg : 6.955853029251099
quant_err : 6.955853029251099
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 25.35211267605634
MSRVTT_full_val/t2v_metrics/R5: 58.75251509054326
MSRVTT_full_val/t2v_metrics/R10: 72.63581488933602
MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.303822937625755
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 47.65009938095121
MSRVTT_full_val/v2t_metrics/R1: 25.35211267605634
MSRVTT_full_val/v2t_metrics/R5: 63.58148893360161
MSRVTT_full_val/v2t_metrics/R10: 77.2635814889336
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.736418511066399
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 49.93900617468437
MSRVTT_full_test/t2v_metrics/R1: 8.62876254180602
MSRVTT_full_test/t2v_metrics/R5: 27.49163879598662
MSRVTT_full_test/t2v_metrics/R10: 39.69899665551839
MSRVTT_full_test/t2v_metrics/R50: 74.51505016722408
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 58.20685618729097
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.117518946200104
MSRVTT_full_test/v2t_metrics/R1: 9.297658862876254
MSRVTT_full_test/v2t_metrics/R5: 29.39799331103679
MSRVTT_full_test/v2t_metrics/R10: 43.44481605351171
MSRVTT_full_test/v2t_metrics/R50: 77.75919732441471
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 52.147491638795984
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.814436808929724
mnt_best : 21.117518946200104
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 3.06344 (QuantReg: 6.70243) QuantErr: 6.70243 batch_time=50.35710
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 3.15004 (QuantReg: 6.93292) QuantErr: 6.93292 batch_time=0.49819
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 2.86933 (QuantReg: 6.84413) QuantErr: 6.84413 batch_time=0.49393
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 3.02737 (QuantReg: 6.80477) QuantErr: 6.80477 batch_time=0.50566
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.71628 (QuantReg: 6.98611) QuantErr: 6.98611 batch_time=0.50661
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 3.08456 (QuantReg: 6.68708) QuantErr: 6.68708 batch_time=0.48430
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 2.77372 (QuantReg: 6.84448) QuantErr: 6.84448 batch_time=0.49538
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 2.96872 (QuantReg: 6.90805) QuantErr: 6.90805 batch_time=0.50926
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 2.84356 (QuantReg: 6.82792) QuantErr: 6.82792 batch_time=0.49733
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 3.21405 (QuantReg: 6.79331) QuantErr: 6.79331 batch_time=0.49214
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 2.67607 (QuantReg: 6.80269) QuantErr: 6.80269 batch_time=0.50412
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 3.12262 (QuantReg: 6.76076) QuantErr: 6.76076 batch_time=0.49200
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 2.69960 (QuantReg: 6.84551) QuantErr: 6.84551 batch_time=0.49187
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 2.79205 (QuantReg: 7.30056) QuantErr: 7.30056 batch_time=0.51750
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.98572 (QuantReg: 6.88027) QuantErr: 6.88027 batch_time=0.51186
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 2.66979 (QuantReg: 7.01861) QuantErr: 7.01861 batch_time=0.49592
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.82318 (QuantReg: 6.81635) QuantErr: 6.81635 batch_time=0.51692
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 3.17443 (QuantReg: 6.59024) QuantErr: 6.59024 batch_time=0.51031
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 3.26146 (QuantReg: 6.78980) QuantErr: 6.78980 batch_time=0.49874
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 3.19960 (QuantReg: 6.87550) QuantErr: 6.87550 batch_time=0.50292
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 2.98337 (QuantReg: 6.29371) QuantErr: 6.29371 batch_time=0.53465
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.87456 (QuantReg: 7.03366) QuantErr: 7.03366 batch_time=0.50822
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 3.11399 (QuantReg: 7.35692) QuantErr: 7.35692 batch_time=0.51078
Train Epoch: 12 codebook_update_time=1.63595
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch12.pth ...
Done in 4.460s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 2.974168775558472
quant_reg : 6.8922064514160155
quant_err : 6.8922064514160155
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 25.150905432595575
MSRVTT_full_val/t2v_metrics/R5: 59.758551307847085
MSRVTT_full_val/t2v_metrics/R10: 71.0261569416499
MSRVTT_full_val/t2v_metrics/R50: 95.97585513078471
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.428571428571429
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 47.43774109767644
MSRVTT_full_val/v2t_metrics/R1: 27.364185110663986
MSRVTT_full_val/v2t_metrics/R5: 64.38631790744466
MSRVTT_full_val/v2t_metrics/R10: 75.45271629778672
MSRVTT_full_val/v2t_metrics/R50: 95.77464788732394
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.937625754527163
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 51.03682660269616
MSRVTT_full_test/t2v_metrics/R1: 8.193979933110368
MSRVTT_full_test/t2v_metrics/R5: 27.224080267558527
MSRVTT_full_test/t2v_metrics/R10: 39.89966555183946
MSRVTT_full_test/t2v_metrics/R50: 74.91638795986623
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 58.47123745819398
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.723946190878333
MSRVTT_full_test/v2t_metrics/R1: 10.033444816053512
MSRVTT_full_test/v2t_metrics/R5: 29.03010033444816
MSRVTT_full_test/v2t_metrics/R10: 42.94314381270903
MSRVTT_full_test/v2t_metrics/R50: 78.16053511705685
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 51.75852842809365
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.21297549235124
mnt_best : 21.117518946200104
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.81744 (QuantReg: 6.81854) QuantErr: 6.81854 batch_time=41.31141
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 3.36040 (QuantReg: 6.43294) QuantErr: 6.43294 batch_time=0.53461
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 3.11299 (QuantReg: 6.96034) QuantErr: 6.96034 batch_time=0.49091
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 2.71574 (QuantReg: 6.89176) QuantErr: 6.89176 batch_time=0.49905
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 3.03789 (QuantReg: 6.93190) QuantErr: 6.93190 batch_time=0.50072
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.91667 (QuantReg: 6.96262) QuantErr: 6.96262 batch_time=0.53332
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 2.86122 (QuantReg: 7.12910) QuantErr: 7.12910 batch_time=1.36921
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 3.20714 (QuantReg: 7.22162) QuantErr: 7.22162 batch_time=0.49463
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 2.69773 (QuantReg: 6.46968) QuantErr: 6.46968 batch_time=0.49259
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 3.06590 (QuantReg: 6.87448) QuantErr: 6.87448 batch_time=0.49162
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 2.98360 (QuantReg: 6.65036) QuantErr: 6.65036 batch_time=0.54923
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 2.78778 (QuantReg: 6.74662) QuantErr: 6.74662 batch_time=0.55666
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 2.84942 (QuantReg: 6.80664) QuantErr: 6.80664 batch_time=0.86958
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 3.03356 (QuantReg: 7.10688) QuantErr: 7.10688 batch_time=0.53683
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.03969 (QuantReg: 6.77678) QuantErr: 6.77678 batch_time=0.49312
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 2.76036 (QuantReg: 6.79708) QuantErr: 6.79708 batch_time=0.50071
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 2.68592 (QuantReg: 7.19464) QuantErr: 7.19464 batch_time=0.49663
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 3.15377 (QuantReg: 6.77977) QuantErr: 6.77977 batch_time=0.48933
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 2.73091 (QuantReg: 7.04461) QuantErr: 7.04461 batch_time=0.49546
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 3.26087 (QuantReg: 6.84005) QuantErr: 6.84005 batch_time=0.49871
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 3.17764 (QuantReg: 7.04189) QuantErr: 7.04189 batch_time=0.49749
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 2.87068 (QuantReg: 7.09382) QuantErr: 7.09382 batch_time=0.48936
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.79147 (QuantReg: 6.89140) QuantErr: 6.89140 batch_time=0.50150
Train Epoch: 13 codebook_update_time=1.68137
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch13.pth ...
Done in 4.162s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch13.pth ...
Done in 8.025s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 2.9167723903656007
quant_reg : 6.890765857696533
quant_err : 6.890765857696533
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 26.760563380281692
MSRVTT_full_val/t2v_metrics/R5: 60.96579476861167
MSRVTT_full_val/t2v_metrics/R10: 74.24547283702213
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.285714285714286
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 49.47857194324977
MSRVTT_full_val/v2t_metrics/R1: 28.571428571428573
MSRVTT_full_val/v2t_metrics/R5: 64.1851106639839
MSRVTT_full_val/v2t_metrics/R10: 77.66599597585513
MSRVTT_full_val/v2t_metrics/R50: 96.78068410462777
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.179074446680081
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 52.22346960705232
MSRVTT_full_test/t2v_metrics/R1: 8.929765886287626
MSRVTT_full_test/t2v_metrics/R5: 27.558528428093645
MSRVTT_full_test/t2v_metrics/R10: 40.468227424749166
MSRVTT_full_test/t2v_metrics/R50: 76.72240802675586
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 55.86354515050167
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.51477256498174
MSRVTT_full_test/v2t_metrics/R1: 9.765886287625419
MSRVTT_full_test/v2t_metrics/R5: 29.19732441471572
MSRVTT_full_test/v2t_metrics/R10: 45.21739130434783
MSRVTT_full_test/v2t_metrics/R50: 79.36454849498328
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 50.53076923076923
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.448770383369656
mnt_best : 21.51477256498174
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 2.74862 (QuantReg: 6.97068) QuantErr: 6.97068 batch_time=35.10607
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 3.22625 (QuantReg: 7.06847) QuantErr: 7.06847 batch_time=0.50303
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 2.85327 (QuantReg: 7.04782) QuantErr: 7.04782 batch_time=0.49979
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.79826 (QuantReg: 6.76159) QuantErr: 6.76159 batch_time=0.50129
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.94738 (QuantReg: 6.85628) QuantErr: 6.85628 batch_time=0.57974
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.79500 (QuantReg: 6.62019) QuantErr: 6.62019 batch_time=0.51321
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 3.07304 (QuantReg: 6.84111) QuantErr: 6.84111 batch_time=0.49170
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 3.00692 (QuantReg: 6.34240) QuantErr: 6.34240 batch_time=0.48720
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.68266 (QuantReg: 6.51284) QuantErr: 6.51284 batch_time=0.50011
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 2.78439 (QuantReg: 6.63057) QuantErr: 6.63057 batch_time=0.52213
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.83237 (QuantReg: 6.83128) QuantErr: 6.83128 batch_time=0.50840
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 2.92454 (QuantReg: 6.67166) QuantErr: 6.67166 batch_time=0.51746
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 2.91377 (QuantReg: 7.29218) QuantErr: 7.29218 batch_time=0.50347
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 2.67623 (QuantReg: 7.03976) QuantErr: 7.03976 batch_time=0.50311
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 2.64439 (QuantReg: 6.87627) QuantErr: 6.87627 batch_time=0.50982
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 2.67920 (QuantReg: 6.78719) QuantErr: 6.78719 batch_time=0.50808
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 2.72678 (QuantReg: 6.77453) QuantErr: 6.77453 batch_time=0.56848
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 2.93462 (QuantReg: 7.16695) QuantErr: 7.16695 batch_time=0.50577
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 2.64052 (QuantReg: 6.56670) QuantErr: 6.56670 batch_time=0.50742
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 3.07432 (QuantReg: 6.62739) QuantErr: 6.62739 batch_time=0.51540
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.99894 (QuantReg: 7.03994) QuantErr: 7.03994 batch_time=0.49863
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.78502 (QuantReg: 6.99693) QuantErr: 6.99693 batch_time=1.29177
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.74280 (QuantReg: 6.94890) QuantErr: 6.94890 batch_time=0.51980
Train Epoch: 14 codebook_update_time=1.61991
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch14.pth ...
Done in 4.044s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 2.8615364961624143
quant_reg : 6.848707180023194
quant_err : 6.848707180023194
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 24.14486921529175
MSRVTT_full_val/t2v_metrics/R5: 58.75251509054326
MSRVTT_full_val/t2v_metrics/R10: 73.03822937625755
MSRVTT_full_val/t2v_metrics/R50: 95.17102615694165
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.392354124748492
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.967832523431035
MSRVTT_full_val/v2t_metrics/R1: 25.75452716297787
MSRVTT_full_val/v2t_metrics/R5: 62.57545271629779
MSRVTT_full_val/v2t_metrics/R10: 77.2635814889336
MSRVTT_full_val/v2t_metrics/R50: 96.17706237424547
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.849094567404427
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 49.935661298172384
MSRVTT_full_test/t2v_metrics/R1: 9.063545150501673
MSRVTT_full_test/t2v_metrics/R5: 27.357859531772576
MSRVTT_full_test/t2v_metrics/R10: 39.23076923076923
MSRVTT_full_test/t2v_metrics/R50: 74.61538461538461
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 60.8556856187291
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.346942514094387
MSRVTT_full_test/v2t_metrics/R1: 9.03010033444816
MSRVTT_full_test/v2t_metrics/R5: 28.929765886287626
MSRVTT_full_test/v2t_metrics/R10: 43.41137123745819
MSRVTT_full_test/v2t_metrics/R50: 77.82608695652173
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 54.591471571906354
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.467099036239695
mnt_best : 21.51477256498174
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 3.11685 (QuantReg: 6.57115) QuantErr: 6.57115 batch_time=36.10734
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.79968 (QuantReg: 6.64398) QuantErr: 6.64398 batch_time=0.50866
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.87550 (QuantReg: 6.65415) QuantErr: 6.65415 batch_time=0.51041
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.90109 (QuantReg: 6.54777) QuantErr: 6.54777 batch_time=0.51131
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.94042 (QuantReg: 6.94475) QuantErr: 6.94475 batch_time=0.50502
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 2.84560 (QuantReg: 6.49313) QuantErr: 6.49313 batch_time=1.06857
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 3.13449 (QuantReg: 6.54219) QuantErr: 6.54219 batch_time=0.51859
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.83997 (QuantReg: 6.75037) QuantErr: 6.75037 batch_time=0.54450
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.97187 (QuantReg: 6.83611) QuantErr: 6.83611 batch_time=0.49878
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 2.73401 (QuantReg: 6.80704) QuantErr: 6.80704 batch_time=0.52198
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.86532 (QuantReg: 6.70668) QuantErr: 6.70668 batch_time=0.54607
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.62362 (QuantReg: 6.54200) QuantErr: 6.54200 batch_time=0.49742
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.64048 (QuantReg: 6.37432) QuantErr: 6.37432 batch_time=2.02560
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 3.24114 (QuantReg: 6.90306) QuantErr: 6.90306 batch_time=1.61401
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.87866 (QuantReg: 6.55702) QuantErr: 6.55702 batch_time=0.50927
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.92454 (QuantReg: 7.07053) QuantErr: 7.07053 batch_time=0.52758
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 3.20164 (QuantReg: 6.97658) QuantErr: 6.97658 batch_time=0.49998
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 2.70073 (QuantReg: 6.80041) QuantErr: 6.80041 batch_time=0.53871
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 2.64389 (QuantReg: 6.90570) QuantErr: 6.90570 batch_time=0.53600
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.86315 (QuantReg: 6.68955) QuantErr: 6.68955 batch_time=0.49838
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 3.16379 (QuantReg: 6.96690) QuantErr: 6.96690 batch_time=0.50446
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.98203 (QuantReg: 7.27427) QuantErr: 7.27427 batch_time=0.50223
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 3.01709 (QuantReg: 6.62417) QuantErr: 6.62417 batch_time=0.52596
Train Epoch: 15 codebook_update_time=1.86173
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch15.pth ...
Done in 14.898s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.15/checkpoint-epoch15.pth ...
Done in 18.717s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 2.8239939546585084
quant_reg : 6.804800344467163
quant_err : 6.804800344467163
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_full_val/t2v_metrics/R1: 27.364185110663986
MSRVTT_full_val/t2v_metrics/R5: 60.563380281690144