-
Notifications
You must be signed in to change notification settings - Fork 4
/
HCQ_MSRVTT_full_bs256.txt
4469 lines (4469 loc) · 260 KB
/
HCQ_MSRVTT_full_bs256.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 721.2272913455963 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 37.05474376678467 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 235.09013748168945 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 153.15345907211304 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch0.pth ...
Done in 1.756s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch0.pth ...
Done in 4.421s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
MSRVTT_full_val/t2v_metrics/MedR: 252.0
MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
MSRVTT_full_val/v2t_metrics/MedR: 243.0
MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
MSRVTT_full_test/t2v_metrics/MedR: 1515.0
MSRVTT_full_test/t2v_metrics/MeanR: 1498.5294314381272
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1471.5
MSRVTT_full_test/v2t_metrics/MeanR: 1495.3260869565217
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
mnt_best : 0.12154652794863813
not_improved_count: 0
Train Epoch: 1 [1/125 256/32000 (1%)] Loss: 11.22009 (QuantReg: 22.46302) QuantErr: 22.46302 batch_time=29.62151
Train Epoch: 1 [17/125 4352/32000 (14%)] Loss: 8.86114 (QuantReg: 22.61963) QuantErr: 22.61963 batch_time=0.72778
Train Epoch: 1 [33/125 8448/32000 (26%)] Loss: 7.70387 (QuantReg: 22.65079) QuantErr: 22.65079 batch_time=0.75476
Train Epoch: 1 [49/125 12544/32000 (39%)] Loss: 7.14685 (QuantReg: 22.62908) QuantErr: 22.62908 batch_time=0.77014
Train Epoch: 1 [65/125 16640/32000 (52%)] Loss: 6.48895 (QuantReg: 22.64130) QuantErr: 22.64130 batch_time=0.77264
Train Epoch: 1 [81/125 20736/32000 (65%)] Loss: 6.38125 (QuantReg: 22.64185) QuantErr: 22.64185 batch_time=0.86151
Train Epoch: 1 [97/125 24832/32000 (78%)] Loss: 5.89802 (QuantReg: 22.64419) QuantErr: 22.64419 batch_time=0.80036
Train Epoch: 1 [113/125 28928/32000 (90%)] Loss: 5.77948 (QuantReg: 22.67823) QuantErr: 22.67823 batch_time=0.93882
Train Epoch: 1 codebook_update_time=1.81953
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch1.pth ...
Done in 5.000s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch1.pth ...
Done in 10.040s
epoch : 1
loss : 7.1198236198425295
quant_reg : 22.622706283569336
quant_err : 22.622706283569336
learning_rate : 5e-05
n_samples : 32000
n_steps : 125
MSRVTT_full_val/t2v_metrics/R1: 17.10261569416499
MSRVTT_full_val/t2v_metrics/R5: 47.48490945674044
MSRVTT_full_val/t2v_metrics/R10: 61.971830985915496
MSRVTT_full_val/t2v_metrics/R50: 93.158953722334
MSRVTT_full_val/t2v_metrics/MedR: 6.0
MSRVTT_full_val/t2v_metrics/MeanR: 15.396378269617706
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 36.920776500497816
MSRVTT_full_val/v2t_metrics/R1: 20.321931589537222
MSRVTT_full_val/v2t_metrics/R5: 53.72233400402415
MSRVTT_full_val/v2t_metrics/R10: 69.21529175050301
MSRVTT_full_val/v2t_metrics/R50: 95.37223340040241
MSRVTT_full_val/v2t_metrics/MedR: 5.0
MSRVTT_full_val/v2t_metrics/MeanR: 12.472837022132797
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 42.27730585320558
MSRVTT_full_test/t2v_metrics/R1: 5.919732441471572
MSRVTT_full_test/t2v_metrics/R5: 19.163879598662206
MSRVTT_full_test/t2v_metrics/R10: 29.39799331103679
MSRVTT_full_test/t2v_metrics/R50: 63.24414715719063
MSRVTT_full_test/t2v_metrics/MedR: 29.0
MSRVTT_full_test/t2v_metrics/MeanR: 80.16923076923077
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 14.940589477644663
MSRVTT_full_test/v2t_metrics/R1: 6.789297658862877
MSRVTT_full_test/v2t_metrics/R5: 22.040133779264213
MSRVTT_full_test/v2t_metrics/R10: 33.37792642140468
MSRVTT_full_test/v2t_metrics/R50: 68.96321070234114
MSRVTT_full_test/v2t_metrics/MedR: 23.0
MSRVTT_full_test/v2t_metrics/MeanR: 68.71973244147158
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 17.09357137010099
mnt_best : 14.940589477644663
not_improved_count: 0
Train Epoch: 2 [1/125 256/32000 (1%)] Loss: 5.23188 (QuantReg: 11.77032) QuantErr: 11.77032 batch_time=37.84993
Train Epoch: 2 [17/125 4352/32000 (14%)] Loss: 5.21253 (QuantReg: 12.31350) QuantErr: 12.31350 batch_time=0.75476
Train Epoch: 2 [33/125 8448/32000 (26%)] Loss: 5.12016 (QuantReg: 12.70065) QuantErr: 12.70065 batch_time=0.73138
Train Epoch: 2 [49/125 12544/32000 (39%)] Loss: 5.04035 (QuantReg: 13.15375) QuantErr: 13.15375 batch_time=0.75140
Train Epoch: 2 [65/125 16640/32000 (52%)] Loss: 4.75050 (QuantReg: 14.02011) QuantErr: 14.02011 batch_time=4.09410
Train Epoch: 2 [81/125 20736/32000 (65%)] Loss: 4.46351 (QuantReg: 14.15531) QuantErr: 14.15531 batch_time=0.88543
Train Epoch: 2 [97/125 24832/32000 (78%)] Loss: 4.63671 (QuantReg: 14.45342) QuantErr: 14.45342 batch_time=0.74318
Train Epoch: 2 [113/125 28928/32000 (90%)] Loss: 4.60893 (QuantReg: 15.28817) QuantErr: 15.28817 batch_time=0.93086
Train Epoch: 2 codebook_update_time=1.61281
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch2.pth ...
Done in 21.621s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch2.pth ...
Done in 25.344s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 4.942960472106933
quant_reg : 13.67909097290039
quant_err : 13.67909097290039
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 22.736418511066397
MSRVTT_full_val/t2v_metrics/R5: 56.33802816901409
MSRVTT_full_val/t2v_metrics/R10: 72.43460764587525
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.613682092555331
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 45.27133169190418
MSRVTT_full_val/v2t_metrics/R1: 26.760563380281692
MSRVTT_full_val/v2t_metrics/R5: 64.58752515090544
MSRVTT_full_val/v2t_metrics/R10: 78.47082494969818
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.95774647887324
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 51.37880962661494
MSRVTT_full_test/t2v_metrics/R1: 8.729096989966555
MSRVTT_full_test/t2v_metrics/R5: 25.986622073578594
MSRVTT_full_test/t2v_metrics/R10: 38.862876254180605
MSRVTT_full_test/t2v_metrics/R50: 72.10702341137124
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 60.19565217391305
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.657829869393094
MSRVTT_full_test/v2t_metrics/R1: 9.899665551839465
MSRVTT_full_test/v2t_metrics/R5: 29.765886287625417
MSRVTT_full_test/v2t_metrics/R10: 42.508361204013376
MSRVTT_full_test/v2t_metrics/R50: 76.72240802675586
MSRVTT_full_test/v2t_metrics/MedR: 14.5
MSRVTT_full_test/v2t_metrics/MeanR: 51.20200668896321
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.224046965809478
mnt_best : 20.657829869393094
not_improved_count: 0
Train Epoch: 3 [1/125 256/32000 (1%)] Loss: 4.55657 (QuantReg: 11.97677) QuantErr: 11.97677 batch_time=36.24997
Train Epoch: 3 [17/125 4352/32000 (14%)] Loss: 4.41748 (QuantReg: 12.37944) QuantErr: 12.37944 batch_time=2.86910
Train Epoch: 3 [33/125 8448/32000 (26%)] Loss: 4.43360 (QuantReg: 12.47073) QuantErr: 12.47073 batch_time=0.89022
Train Epoch: 3 [49/125 12544/32000 (39%)] Loss: 4.12300 (QuantReg: 12.75680) QuantErr: 12.75680 batch_time=0.75958
Train Epoch: 3 [65/125 16640/32000 (52%)] Loss: 4.22704 (QuantReg: 12.95197) QuantErr: 12.95197 batch_time=0.75180
Train Epoch: 3 [81/125 20736/32000 (65%)] Loss: 4.05506 (QuantReg: 13.24821) QuantErr: 13.24821 batch_time=2.86447
Train Epoch: 3 [97/125 24832/32000 (78%)] Loss: 3.69180 (QuantReg: 13.35273) QuantErr: 13.35273 batch_time=0.73935
Train Epoch: 3 [113/125 28928/32000 (90%)] Loss: 4.22632 (QuantReg: 13.55894) QuantErr: 13.55894 batch_time=0.74957
Train Epoch: 3 codebook_update_time=1.81088
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch3.pth ...
Done in 4.459s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch3.pth ...
Done in 8.518s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 4.211776826858521
quant_reg : 12.919039520263672
quant_err : 12.919039520263672
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 375
MSRVTT_full_val/t2v_metrics/R1: 24.949698189134807
MSRVTT_full_val/t2v_metrics/R5: 61.16700201207244
MSRVTT_full_val/t2v_metrics/R10: 75.25150905432595
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.523138832997988
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.607046723187516
MSRVTT_full_val/v2t_metrics/R1: 31.388329979879277
MSRVTT_full_val/v2t_metrics/R5: 68.41046277665995
MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.800804828973843
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.434224298216506
MSRVTT_full_test/t2v_metrics/R1: 8.996655518394649
MSRVTT_full_test/t2v_metrics/R5: 28.62876254180602
MSRVTT_full_test/t2v_metrics/R10: 40.53511705685619
MSRVTT_full_test/t2v_metrics/R50: 74.81605351170569
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 53.336120401337794
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.856051223368222
MSRVTT_full_test/v2t_metrics/R1: 11.070234113712374
MSRVTT_full_test/v2t_metrics/R5: 32.675585284280935
MSRVTT_full_test/v2t_metrics/R10: 46.65551839464883
MSRVTT_full_test/v2t_metrics/R50: 80.13377926421404
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 43.615050167224084
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.65041523954077
mnt_best : 21.856051223368222
not_improved_count: 0
Train Epoch: 4 [1/125 256/32000 (1%)] Loss: 4.25866 (QuantReg: 12.29319) QuantErr: 12.29319 batch_time=45.59648
Train Epoch: 4 [17/125 4352/32000 (14%)] Loss: 3.76131 (QuantReg: 12.46964) QuantErr: 12.46964 batch_time=0.84708
Train Epoch: 4 [33/125 8448/32000 (26%)] Loss: 3.54967 (QuantReg: 12.69598) QuantErr: 12.69598 batch_time=0.73501
Train Epoch: 4 [49/125 12544/32000 (39%)] Loss: 3.74672 (QuantReg: 12.77463) QuantErr: 12.77463 batch_time=0.75831
Train Epoch: 4 [65/125 16640/32000 (52%)] Loss: 3.94518 (QuantReg: 12.93646) QuantErr: 12.93646 batch_time=1.69216
Train Epoch: 4 [81/125 20736/32000 (65%)] Loss: 3.99261 (QuantReg: 12.83701) QuantErr: 12.83701 batch_time=0.84018
Train Epoch: 4 [97/125 24832/32000 (78%)] Loss: 3.40698 (QuantReg: 13.13470) QuantErr: 13.13470 batch_time=0.84378
Train Epoch: 4 [113/125 28928/32000 (90%)] Loss: 3.40417 (QuantReg: 13.04777) QuantErr: 13.04777 batch_time=0.75109
Train Epoch: 4 codebook_update_time=1.66647
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch4.pth ...
Done in 5.182s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch4.pth ...
Done in 11.070s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 3.781222272872925
quant_reg : 12.842113235473633
quant_err : 12.842113235473633
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 26.961770623742456
MSRVTT_full_val/t2v_metrics/R5: 62.97786720321932
MSRVTT_full_val/t2v_metrics/R10: 77.2635814889336
MSRVTT_full_val/t2v_metrics/R50: 98.18913480885311
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.847082494969818
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.81248070529641
MSRVTT_full_val/v2t_metrics/R1: 30.18108651911469
MSRVTT_full_val/v2t_metrics/R5: 71.42857142857143
MSRVTT_full_val/v2t_metrics/R10: 83.29979879275653
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.482897384305835
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.41790159773574
MSRVTT_full_test/t2v_metrics/R1: 9.59866220735786
MSRVTT_full_test/t2v_metrics/R5: 29.698996655518396
MSRVTT_full_test/t2v_metrics/R10: 43.24414715719063
MSRVTT_full_test/t2v_metrics/R50: 77.1571906354515
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 46.84314381270903
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.10077804998149
MSRVTT_full_test/v2t_metrics/R1: 12.14046822742475
MSRVTT_full_test/v2t_metrics/R5: 35.25083612040134
MSRVTT_full_test/v2t_metrics/R10: 49.56521739130435
MSRVTT_full_test/v2t_metrics/R50: 82.74247491638796
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 37.07006688963211
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.68177652050579
mnt_best : 23.10077804998149
not_improved_count: 0
Train Epoch: 5 [1/125 256/32000 (1%)] Loss: 3.55801 (QuantReg: 12.42609) QuantErr: 12.42609 batch_time=37.26921
Train Epoch: 5 [17/125 4352/32000 (14%)] Loss: 3.51526 (QuantReg: 12.81252) QuantErr: 12.81252 batch_time=0.74224
Train Epoch: 5 [33/125 8448/32000 (26%)] Loss: 3.24577 (QuantReg: 12.87114) QuantErr: 12.87114 batch_time=0.77548
Train Epoch: 5 [49/125 12544/32000 (39%)] Loss: 3.20709 (QuantReg: 12.79152) QuantErr: 12.79152 batch_time=0.77892
Train Epoch: 5 [65/125 16640/32000 (52%)] Loss: 3.08974 (QuantReg: 13.06595) QuantErr: 13.06595 batch_time=0.97878
Train Epoch: 5 [81/125 20736/32000 (65%)] Loss: 3.28793 (QuantReg: 13.06563) QuantErr: 13.06563 batch_time=0.95083
Train Epoch: 5 [97/125 24832/32000 (78%)] Loss: 3.13906 (QuantReg: 12.95514) QuantErr: 12.95514 batch_time=0.75231
Train Epoch: 5 [113/125 28928/32000 (90%)] Loss: 3.55830 (QuantReg: 13.29980) QuantErr: 13.29980 batch_time=0.73709
Train Epoch: 5 codebook_update_time=1.63723
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch5.pth ...
Done in 4.353s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch5.pth ...
Done in 16.948s
removing stale ckpt [epoch 4] [took 0.16s]
epoch : 5
loss : 3.4096395950317384
quant_reg : 13.011991203308105
quant_err : 13.011991203308105
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 625
MSRVTT_full_val/t2v_metrics/R1: 28.37022132796781
MSRVTT_full_val/t2v_metrics/R5: 64.1851106639839
MSRVTT_full_val/t2v_metrics/R10: 77.2635814889336
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.794768611670019
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.01045079117703
MSRVTT_full_val/v2t_metrics/R1: 30.784708249496983
MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
MSRVTT_full_val/v2t_metrics/R10: 82.09255533199195
MSRVTT_full_val/v2t_metrics/R50: 98.79275653923541
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.132796780684105
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.67460276047788
MSRVTT_full_test/t2v_metrics/R1: 10.535117056856187
MSRVTT_full_test/t2v_metrics/R5: 29.598662207357858
MSRVTT_full_test/t2v_metrics/R10: 43.17725752508361
MSRVTT_full_test/t2v_metrics/R50: 77.89297658862876
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.489297658862874
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.789692798110735
MSRVTT_full_test/v2t_metrics/R1: 12.37458193979933
MSRVTT_full_test/v2t_metrics/R5: 35.35117056856188
MSRVTT_full_test/v2t_metrics/R10: 50.23411371237458
MSRVTT_full_test/v2t_metrics/R50: 83.47826086956522
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 38.11672240802675
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.009865690753163
mnt_best : 23.789692798110735
not_improved_count: 0
Train Epoch: 6 [1/125 256/32000 (1%)] Loss: 3.40289 (QuantReg: 12.84797) QuantErr: 12.84797 batch_time=42.81861
Train Epoch: 6 [17/125 4352/32000 (14%)] Loss: 3.39512 (QuantReg: 12.81523) QuantErr: 12.81523 batch_time=4.06002
Train Epoch: 6 [33/125 8448/32000 (26%)] Loss: 3.43623 (QuantReg: 12.99560) QuantErr: 12.99560 batch_time=0.75743
Train Epoch: 6 [49/125 12544/32000 (39%)] Loss: 2.94881 (QuantReg: 13.33653) QuantErr: 13.33653 batch_time=0.73456
Train Epoch: 6 [65/125 16640/32000 (52%)] Loss: 3.45603 (QuantReg: 12.93412) QuantErr: 12.93412 batch_time=3.70007
Train Epoch: 6 [81/125 20736/32000 (65%)] Loss: 3.11369 (QuantReg: 13.32969) QuantErr: 13.32969 batch_time=3.52348
Train Epoch: 6 [97/125 24832/32000 (78%)] Loss: 3.11262 (QuantReg: 13.54766) QuantErr: 13.54766 batch_time=0.74327
Train Epoch: 6 [113/125 28928/32000 (90%)] Loss: 2.78031 (QuantReg: 13.48894) QuantErr: 13.48894 batch_time=0.73746
Train Epoch: 6 codebook_update_time=1.63220
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch6.pth ...
Done in 3.831s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch6.pth ...
Done in 8.356s
removing stale ckpt [epoch 5] [took 0.06s]
epoch : 6
loss : 3.154381580352783
quant_reg : 13.051425605773925
quant_err : 13.051425605773925
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 28.169014084507044
MSRVTT_full_val/t2v_metrics/R5: 62.57545271629779
MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
MSRVTT_full_val/t2v_metrics/R50: 98.18913480885311
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.567404426559357
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 51.89256764337161
MSRVTT_full_val/v2t_metrics/R1: 33.80281690140845
MSRVTT_full_val/v2t_metrics/R5: 70.4225352112676
MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.901408450704225
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.6403767535992
MSRVTT_full_test/t2v_metrics/R1: 10.735785953177258
MSRVTT_full_test/t2v_metrics/R5: 31.137123745819398
MSRVTT_full_test/t2v_metrics/R10: 45.1505016722408
MSRVTT_full_test/t2v_metrics/R50: 79.4648829431438
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.14163879598662
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.712971726253034
MSRVTT_full_test/v2t_metrics/R1: 13.311036789297658
MSRVTT_full_test/v2t_metrics/R5: 38.32775919732441
MSRVTT_full_test/v2t_metrics/R10: 51.90635451505017
MSRVTT_full_test/v2t_metrics/R50: 84.64882943143813
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.788628762541805
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.806794991589733
mnt_best : 24.712971726253034
not_improved_count: 0
Train Epoch: 7 [1/125 256/32000 (1%)] Loss: 2.64780 (QuantReg: 12.82744) QuantErr: 12.82744 batch_time=55.32466
Train Epoch: 7 [17/125 4352/32000 (14%)] Loss: 2.76277 (QuantReg: 12.94643) QuantErr: 12.94643 batch_time=1.04641
Train Epoch: 7 [33/125 8448/32000 (26%)] Loss: 3.27648 (QuantReg: 13.02782) QuantErr: 13.02782 batch_time=0.74685
Train Epoch: 7 [49/125 12544/32000 (39%)] Loss: 3.18682 (QuantReg: 13.19938) QuantErr: 13.19938 batch_time=1.99944
Train Epoch: 7 [65/125 16640/32000 (52%)] Loss: 2.92595 (QuantReg: 13.13552) QuantErr: 13.13552 batch_time=13.33293
Train Epoch: 7 [81/125 20736/32000 (65%)] Loss: 2.96854 (QuantReg: 13.23915) QuantErr: 13.23915 batch_time=0.98241
Train Epoch: 7 [97/125 24832/32000 (78%)] Loss: 2.93184 (QuantReg: 13.30652) QuantErr: 13.30652 batch_time=0.75448
Train Epoch: 7 [113/125 28928/32000 (90%)] Loss: 2.69038 (QuantReg: 13.16595) QuantErr: 13.16595 batch_time=2.01076
Train Epoch: 7 codebook_update_time=1.68812
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch7.pth ...
Done in 4.266s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch7.pth ...
Done in 9.280s
removing stale ckpt [epoch 6] [took 0.43s]
epoch : 7
loss : 2.970870113372803
quant_reg : 13.143326766967773
quant_err : 13.143326766967773
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 875
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 62.17303822937626
MSRVTT_full_val/t2v_metrics/R10: 78.26961770623743
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.038229376257545
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.287425256709916
MSRVTT_full_val/v2t_metrics/R1: 32.796780684104625
MSRVTT_full_val/v2t_metrics/R5: 71.62977867203219
MSRVTT_full_val/v2t_metrics/R10: 86.31790744466801
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.788732394366197
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.75009539418348
MSRVTT_full_test/t2v_metrics/R1: 11.17056856187291
MSRVTT_full_test/t2v_metrics/R5: 32.441471571906355
MSRVTT_full_test/t2v_metrics/R10: 45.38461538461539
MSRVTT_full_test/t2v_metrics/R50: 78.39464882943143
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 46.12675585284281
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.43088620700667
MSRVTT_full_test/v2t_metrics/R1: 13.177257525083611
MSRVTT_full_test/v2t_metrics/R5: 38.12709030100334
MSRVTT_full_test/v2t_metrics/R10: 52.90969899665552
MSRVTT_full_test/v2t_metrics/R50: 84.68227424749163
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.49280936454849
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.844524403442886
mnt_best : 25.43088620700667
not_improved_count: 0
Train Epoch: 8 [1/125 256/32000 (1%)] Loss: 2.97525 (QuantReg: 13.14789) QuantErr: 13.14789 batch_time=50.78277
Train Epoch: 8 [17/125 4352/32000 (14%)] Loss: 2.83680 (QuantReg: 13.07020) QuantErr: 13.07020 batch_time=0.89752
Train Epoch: 8 [33/125 8448/32000 (26%)] Loss: 2.79471 (QuantReg: 12.83999) QuantErr: 12.83999 batch_time=0.75163
Train Epoch: 8 [49/125 12544/32000 (39%)] Loss: 3.03696 (QuantReg: 12.91600) QuantErr: 12.91600 batch_time=0.73972
Train Epoch: 8 [65/125 16640/32000 (52%)] Loss: 3.00537 (QuantReg: 13.06215) QuantErr: 13.06215 batch_time=8.81928
Train Epoch: 8 [81/125 20736/32000 (65%)] Loss: 2.99956 (QuantReg: 13.14621) QuantErr: 13.14621 batch_time=0.75193
Train Epoch: 8 [97/125 24832/32000 (78%)] Loss: 2.70725 (QuantReg: 13.13393) QuantErr: 13.13393 batch_time=0.79249
Train Epoch: 8 [113/125 28928/32000 (90%)] Loss: 2.74703 (QuantReg: 13.05886) QuantErr: 13.05886 batch_time=0.74951
Train Epoch: 8 codebook_update_time=1.78971
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch8.pth ...
Done in 4.819s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch8.pth ...
Done in 9.511s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 2.7841379928588865
quant_reg : 13.160666114807128
quant_err : 13.160666114807128
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 30.58350100603622
MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.909456740442656
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.87563003940867
MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
MSRVTT_full_val/v2t_metrics/R5: 75.65392354124748
MSRVTT_full_val/v2t_metrics/R10: 86.11670020120724
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.064386317907445
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.09944031287618
MSRVTT_full_test/t2v_metrics/R1: 12.040133779264215
MSRVTT_full_test/t2v_metrics/R5: 32.94314381270903
MSRVTT_full_test/t2v_metrics/R10: 47.625418060200666
MSRVTT_full_test/t2v_metrics/R50: 80.60200668896321
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.21103678929766
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.632486682817603
MSRVTT_full_test/v2t_metrics/R1: 14.280936454849499
MSRVTT_full_test/v2t_metrics/R5: 38.929765886287626
MSRVTT_full_test/v2t_metrics/R10: 53.31103678929766
MSRVTT_full_test/v2t_metrics/R50: 85.35117056856187
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.967224080267556
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.94699863814622
mnt_best : 26.632486682817603
not_improved_count: 0
Train Epoch: 9 [1/125 256/32000 (1%)] Loss: 3.04971 (QuantReg: 12.99928) QuantErr: 12.99928 batch_time=44.21777
Train Epoch: 9 [17/125 4352/32000 (14%)] Loss: 3.10184 (QuantReg: 13.06532) QuantErr: 13.06532 batch_time=1.74077
Train Epoch: 9 [33/125 8448/32000 (26%)] Loss: 2.75977 (QuantReg: 13.27556) QuantErr: 13.27556 batch_time=1.61944
Train Epoch: 9 [49/125 12544/32000 (39%)] Loss: 2.65655 (QuantReg: 13.16294) QuantErr: 13.16294 batch_time=0.74571
Train Epoch: 9 [65/125 16640/32000 (52%)] Loss: 2.74634 (QuantReg: 13.05528) QuantErr: 13.05528 batch_time=2.19704
Train Epoch: 9 [81/125 20736/32000 (65%)] Loss: 2.65549 (QuantReg: 13.31017) QuantErr: 13.31017 batch_time=1.70364
Train Epoch: 9 [97/125 24832/32000 (78%)] Loss: 2.60720 (QuantReg: 13.35579) QuantErr: 13.35579 batch_time=1.51848
Train Epoch: 9 [113/125 28928/32000 (90%)] Loss: 2.47861 (QuantReg: 13.33753) QuantErr: 13.33753 batch_time=0.78452
Train Epoch: 9 codebook_update_time=1.77836
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch9.pth ...
Done in 4.088s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 2.6520782585144045
quant_reg : 13.261753967285156
quant_err : 13.261753967285156
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 1125
MSRVTT_full_val/t2v_metrics/R1: 32.394366197183096
MSRVTT_full_val/t2v_metrics/R5: 65.3923541247485
MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.209255533199194
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.17115774417963
MSRVTT_full_val/v2t_metrics/R1: 32.394366197183096
MSRVTT_full_val/v2t_metrics/R5: 73.44064386317908
MSRVTT_full_val/v2t_metrics/R10: 85.51307847082495
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.47887323943662
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.81383535550033
MSRVTT_full_test/t2v_metrics/R1: 11.57190635451505
MSRVTT_full_test/t2v_metrics/R5: 33.87959866220736
MSRVTT_full_test/t2v_metrics/R10: 46.92307692307692
MSRVTT_full_test/t2v_metrics/R50: 79.49832775919732
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.30468227424749
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.39833586475874
MSRVTT_full_test/v2t_metrics/R1: 14.481605351170568
MSRVTT_full_test/v2t_metrics/R5: 38.69565217391305
MSRVTT_full_test/v2t_metrics/R10: 54.147157190635454
MSRVTT_full_test/v2t_metrics/R50: 84.84949832775919
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.63327759197325
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.190201444641787
mnt_best : 26.632486682817603
not_improved_count: 1
Train Epoch: 10 [1/125 256/32000 (1%)] Loss: 2.87962 (QuantReg: 12.87302) QuantErr: 12.87302 batch_time=39.66266
Train Epoch: 10 [17/125 4352/32000 (14%)] Loss: 2.69879 (QuantReg: 13.15061) QuantErr: 13.15061 batch_time=0.76510
Train Epoch: 10 [33/125 8448/32000 (26%)] Loss: 2.12039 (QuantReg: 12.98295) QuantErr: 12.98295 batch_time=0.85966
Train Epoch: 10 [49/125 12544/32000 (39%)] Loss: 2.47367 (QuantReg: 13.19082) QuantErr: 13.19082 batch_time=0.76362
Train Epoch: 10 [65/125 16640/32000 (52%)] Loss: 2.56730 (QuantReg: 13.36712) QuantErr: 13.36712 batch_time=5.10368
Train Epoch: 10 [81/125 20736/32000 (65%)] Loss: 2.49105 (QuantReg: 13.24953) QuantErr: 13.24953 batch_time=0.76815
Train Epoch: 10 [97/125 24832/32000 (78%)] Loss: 2.75520 (QuantReg: 13.28102) QuantErr: 13.28102 batch_time=0.86606
Train Epoch: 10 [113/125 28928/32000 (90%)] Loss: 2.75627 (QuantReg: 13.30014) QuantErr: 13.30014 batch_time=0.74222
Train Epoch: 10 codebook_update_time=1.70740
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch10.pth ...
Done in 13.463s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 2.536181224822998
quant_reg : 13.30095012664795
quant_err : 13.30095012664795
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 31.790744466800806
MSRVTT_full_val/t2v_metrics/R5: 64.1851106639839
MSRVTT_full_val/t2v_metrics/R10: 79.87927565392354
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.474849094567404
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.62478033376326
MSRVTT_full_val/v2t_metrics/R1: 33.80281690140845
MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
MSRVTT_full_val/v2t_metrics/R10: 86.11670020120724
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.674044265593562
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.22787940200011
MSRVTT_full_test/t2v_metrics/R1: 11.438127090301004
MSRVTT_full_test/t2v_metrics/R5: 33.34448160535117
MSRVTT_full_test/t2v_metrics/R10: 46.020066889632105
MSRVTT_full_test/t2v_metrics/R50: 79.1638795986622
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.792642140468224
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.988150776515248
MSRVTT_full_test/v2t_metrics/R1: 14.548494983277592
MSRVTT_full_test/v2t_metrics/R5: 39.86622073578595
MSRVTT_full_test/v2t_metrics/R10: 54.48160535117057
MSRVTT_full_test/v2t_metrics/R50: 85.75250836120401
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.09364548494983
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.614841626048708
mnt_best : 26.632486682817603
not_improved_count: 2
Train Epoch: 11 [1/125 256/32000 (1%)] Loss: 2.56824 (QuantReg: 13.24133) QuantErr: 13.24133 batch_time=45.62533
Train Epoch: 11 [17/125 4352/32000 (14%)] Loss: 2.25104 (QuantReg: 13.34244) QuantErr: 13.34244 batch_time=0.75695
Train Epoch: 11 [33/125 8448/32000 (26%)] Loss: 2.11745 (QuantReg: 13.63294) QuantErr: 13.63294 batch_time=0.74868
Train Epoch: 11 [49/125 12544/32000 (39%)] Loss: 2.38136 (QuantReg: 13.40882) QuantErr: 13.40882 batch_time=1.18530
Train Epoch: 11 [65/125 16640/32000 (52%)] Loss: 2.31578 (QuantReg: 13.45031) QuantErr: 13.45031 batch_time=10.12938
Train Epoch: 11 [81/125 20736/32000 (65%)] Loss: 2.15176 (QuantReg: 13.45653) QuantErr: 13.45653 batch_time=0.75413
Train Epoch: 11 [97/125 24832/32000 (78%)] Loss: 2.33216 (QuantReg: 13.61406) QuantErr: 13.61406 batch_time=0.85616
Train Epoch: 11 [113/125 28928/32000 (90%)] Loss: 2.55033 (QuantReg: 13.40026) QuantErr: 13.40026 batch_time=0.96672
Train Epoch: 11 codebook_update_time=1.75658
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch11.pth ...
Done in 4.411s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 2.4074890842437746
quant_reg : 13.366433380126953
quant_err : 13.366433380126953
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 1375
MSRVTT_full_val/t2v_metrics/R1: 27.56539235412475
MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
MSRVTT_full_val/t2v_metrics/R10: 79.47686116700201
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.253521126760564
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.2176119263171
MSRVTT_full_val/v2t_metrics/R1: 35.412474849094565
MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.346076458752515
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.26422297765961
MSRVTT_full_test/t2v_metrics/R1: 10.501672240802675
MSRVTT_full_test/t2v_metrics/R5: 32.0066889632107
MSRVTT_full_test/t2v_metrics/R10: 45.719063545150505
MSRVTT_full_test/t2v_metrics/R50: 78.62876254180603
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 46.36120401337793
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.861777674934743
MSRVTT_full_test/v2t_metrics/R1: 14.347826086956522
MSRVTT_full_test/v2t_metrics/R5: 39.531772575250834
MSRVTT_full_test/v2t_metrics/R10: 53.812709030100336
MSRVTT_full_test/v2t_metrics/R50: 85.08361204013377
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.94899665551839
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.251611438934773
mnt_best : 26.632486682817603
not_improved_count: 3
Train Epoch: 12 [1/125 256/32000 (1%)] Loss: 2.43776 (QuantReg: 13.13254) QuantErr: 13.13254 batch_time=36.86617
Train Epoch: 12 [17/125 4352/32000 (14%)] Loss: 2.62807 (QuantReg: 13.37611) QuantErr: 13.37611 batch_time=0.93156
Train Epoch: 12 [33/125 8448/32000 (26%)] Loss: 2.06361 (QuantReg: 13.48146) QuantErr: 13.48146 batch_time=0.99717
Train Epoch: 12 [49/125 12544/32000 (39%)] Loss: 2.38954 (QuantReg: 13.41766) QuantErr: 13.41766 batch_time=0.74566
Train Epoch: 12 [65/125 16640/32000 (52%)] Loss: 2.36757 (QuantReg: 13.55765) QuantErr: 13.55765 batch_time=1.98633
Train Epoch: 12 [81/125 20736/32000 (65%)] Loss: 2.24145 (QuantReg: 13.52949) QuantErr: 13.52949 batch_time=0.77344
Train Epoch: 12 [97/125 24832/32000 (78%)] Loss: 2.36366 (QuantReg: 13.41046) QuantErr: 13.41046 batch_time=1.14446
Train Epoch: 12 [113/125 28928/32000 (90%)] Loss: 2.31451 (QuantReg: 13.37399) QuantErr: 13.37399 batch_time=0.74556
Train Epoch: 12 codebook_update_time=1.59254
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch12.pth ...
Done in 4.380s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch12.pth ...
Done in 9.512s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 2.347339246749878
quant_reg : 13.422491744995117
quant_err : 13.422491744995117
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 31.58953722334004
MSRVTT_full_val/t2v_metrics/R5: 65.3923541247485
MSRVTT_full_val/t2v_metrics/R10: 81.69014084507042
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.865191146881288
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.26030703404006
MSRVTT_full_val/v2t_metrics/R1: 35.2112676056338
MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
MSRVTT_full_val/v2t_metrics/R10: 87.72635814889335
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.72635814889336
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.265930221758836
MSRVTT_full_test/t2v_metrics/R1: 12.54180602006689
MSRVTT_full_test/t2v_metrics/R5: 34.247491638795985
MSRVTT_full_test/t2v_metrics/R10: 47.69230769230769
MSRVTT_full_test/t2v_metrics/R50: 80.10033444816054
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 42.42307692307692
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.361867047091895
MSRVTT_full_test/v2t_metrics/R1: 14.280936454849499
MSRVTT_full_test/v2t_metrics/R5: 41.37123745819398
MSRVTT_full_test/v2t_metrics/R10: 55.35117056856188
MSRVTT_full_test/v2t_metrics/R50: 86.35451505016722
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 30.707859531772574
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.978690019696508
mnt_best : 27.361867047091895
not_improved_count: 0
Train Epoch: 13 [1/125 256/32000 (1%)] Loss: 2.31656 (QuantReg: 13.48718) QuantErr: 13.48718 batch_time=41.61112
Train Epoch: 13 [17/125 4352/32000 (14%)] Loss: 2.05901 (QuantReg: 13.40941) QuantErr: 13.40941 batch_time=3.14408
Train Epoch: 13 [33/125 8448/32000 (26%)] Loss: 2.39475 (QuantReg: 13.55238) QuantErr: 13.55238 batch_time=0.95801
Train Epoch: 13 [49/125 12544/32000 (39%)] Loss: 2.21443 (QuantReg: 13.48490) QuantErr: 13.48490 batch_time=0.73362
Train Epoch: 13 [65/125 16640/32000 (52%)] Loss: 2.25883 (QuantReg: 13.18168) QuantErr: 13.18168 batch_time=5.07854
Train Epoch: 13 [81/125 20736/32000 (65%)] Loss: 2.64793 (QuantReg: 13.36213) QuantErr: 13.36213 batch_time=3.63918
Train Epoch: 13 [97/125 24832/32000 (78%)] Loss: 2.33002 (QuantReg: 13.50118) QuantErr: 13.50118 batch_time=0.98904
Train Epoch: 13 [113/125 28928/32000 (90%)] Loss: 2.39035 (QuantReg: 13.62226) QuantErr: 13.62226 batch_time=0.74860
Train Epoch: 13 codebook_update_time=1.65125
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch13.pth ...
Done in 3.940s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch13.pth ...
Done in 8.371s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 2.2696165857315065
quant_reg : 13.494962272644043
quant_err : 13.494962272644043
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 1625
MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
MSRVTT_full_val/t2v_metrics/R5: 67.40442655935614
MSRVTT_full_val/t2v_metrics/R10: 82.09255533199195
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.635814889336016
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.50023742628318
MSRVTT_full_val/v2t_metrics/R1: 36.41851106639839
MSRVTT_full_val/v2t_metrics/R5: 75.25150905432595
MSRVTT_full_val/v2t_metrics/R10: 88.32997987927565
MSRVTT_full_val/v2t_metrics/R50: 98.79275653923541
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.830985915492958
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.32302294111043
MSRVTT_full_test/t2v_metrics/R1: 12.37458193979933
MSRVTT_full_test/t2v_metrics/R5: 34.51505016722408
MSRVTT_full_test/t2v_metrics/R10: 48.2943143812709
MSRVTT_full_test/t2v_metrics/R50: 81.00334448160535
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 42.20535117056856
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.42489771827609
MSRVTT_full_test/v2t_metrics/R1: 14.68227424749164
MSRVTT_full_test/v2t_metrics/R5: 40.96989966555184
MSRVTT_full_test/v2t_metrics/R10: 56.38795986622073
MSRVTT_full_test/v2t_metrics/R50: 86.08695652173913
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 30.610702341137124
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.370410342108066
mnt_best : 27.42489771827609
not_improved_count: 0
Train Epoch: 14 [1/125 256/32000 (1%)] Loss: 2.09460 (QuantReg: 13.50841) QuantErr: 13.50841 batch_time=46.51388
Train Epoch: 14 [17/125 4352/32000 (14%)] Loss: 2.31007 (QuantReg: 13.21856) QuantErr: 13.21856 batch_time=0.73553
Train Epoch: 14 [33/125 8448/32000 (26%)] Loss: 2.28561 (QuantReg: 13.44373) QuantErr: 13.44373 batch_time=0.73653
Train Epoch: 14 [49/125 12544/32000 (39%)] Loss: 2.26397 (QuantReg: 13.48887) QuantErr: 13.48887 batch_time=0.77770
Train Epoch: 14 [65/125 16640/32000 (52%)] Loss: 2.43147 (QuantReg: 13.64102) QuantErr: 13.64102 batch_time=8.10784
Train Epoch: 14 [81/125 20736/32000 (65%)] Loss: 2.33766 (QuantReg: 13.51908) QuantErr: 13.51908 batch_time=0.82544
Train Epoch: 14 [97/125 24832/32000 (78%)] Loss: 2.05150 (QuantReg: 13.41814) QuantErr: 13.41814 batch_time=0.82469
Train Epoch: 14 [113/125 28928/32000 (90%)] Loss: 2.01952 (QuantReg: 13.63159) QuantErr: 13.63159 batch_time=0.74893
Train Epoch: 14 codebook_update_time=1.64450
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch14.pth ...
Done in 16.268s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch14.pth ...
Done in 19.969s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 2.206376944541931
quant_reg : 13.530414100646972
quant_err : 13.530414100646972
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 32.19315895372233
MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.03420523138833
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.598543157742654
MSRVTT_full_val/v2t_metrics/R1: 36.82092555331992
MSRVTT_full_val/v2t_metrics/R5: 75.65392354124748
MSRVTT_full_val/v2t_metrics/R10: 88.93360160965794
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.82897384305835
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.80545206292597
MSRVTT_full_test/t2v_metrics/R1: 12.876254180602007
MSRVTT_full_test/t2v_metrics/R5: 34.74916387959866
MSRVTT_full_test/t2v_metrics/R10: 48.36120401337793
MSRVTT_full_test/t2v_metrics/R50: 80.8361204013378
MSRVTT_full_test/t2v_metrics/MedR: 11.5
MSRVTT_full_test/t2v_metrics/MeanR: 42.855183946488296
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.866151998684543
MSRVTT_full_test/v2t_metrics/R1: 15.418060200668897
MSRVTT_full_test/v2t_metrics/R5: 42.541806020066886
MSRVTT_full_test/v2t_metrics/R10: 57.458193979933114
MSRVTT_full_test/v2t_metrics/R50: 86.5551839464883
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 29.88494983277592
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.52734862862026
mnt_best : 27.866151998684543
not_improved_count: 0
Train Epoch: 15 [1/125 256/32000 (1%)] Loss: 2.49299 (QuantReg: 13.23431) QuantErr: 13.23431 batch_time=41.75253
Train Epoch: 15 [17/125 4352/32000 (14%)] Loss: 2.15547 (QuantReg: 13.27426) QuantErr: 13.27426 batch_time=0.74653
Train Epoch: 15 [33/125 8448/32000 (26%)] Loss: 2.11740 (QuantReg: 13.35264) QuantErr: 13.35264 batch_time=0.74219
Train Epoch: 15 [49/125 12544/32000 (39%)] Loss: 1.95543 (QuantReg: 13.46893) QuantErr: 13.46893 batch_time=0.73045
Train Epoch: 15 [65/125 16640/32000 (52%)] Loss: 2.01758 (QuantReg: 13.43160) QuantErr: 13.43160 batch_time=6.58724
Train Epoch: 15 [81/125 20736/32000 (65%)] Loss: 2.02260 (QuantReg: 13.61464) QuantErr: 13.61464 batch_time=0.88546
Train Epoch: 15 [97/125 24832/32000 (78%)] Loss: 2.10495 (QuantReg: 13.63645) QuantErr: 13.63645 batch_time=0.73662
Train Epoch: 15 [113/125 28928/32000 (90%)] Loss: 1.90299 (QuantReg: 13.68564) QuantErr: 13.68564 batch_time=0.74098
Train Epoch: 15 codebook_update_time=1.70331
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch15.pth ...
Done in 10.872s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 2.1380276861190795
quant_reg : 13.508374969482421
quant_err : 13.508374969482421
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 1875
MSRVTT_full_val/t2v_metrics/R1: 30.985915492957748
MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.002012072434606
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.22571637902403
MSRVTT_full_val/v2t_metrics/R1: 36.21730382293762
MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
MSRVTT_full_val/v2t_metrics/R10: 87.32394366197182
MSRVTT_full_val/v2t_metrics/R50: 98.79275653923541
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.776659959758551
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.91567806186523
MSRVTT_full_test/t2v_metrics/R1: 12.341137123745819
MSRVTT_full_test/t2v_metrics/R5: 34.147157190635454
MSRVTT_full_test/t2v_metrics/R10: 48.79598662207358
MSRVTT_full_test/t2v_metrics/R50: 80.63545150501672
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.325752508361205
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.396680502545337
MSRVTT_full_test/v2t_metrics/R1: 16.153846153846153
MSRVTT_full_test/v2t_metrics/R5: 42.541806020066886
MSRVTT_full_test/v2t_metrics/R10: 56.62207357859532
MSRVTT_full_test/v2t_metrics/R50: 86.5551839464883
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 30.556521739130435
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.88643475212785
mnt_best : 27.866151998684543
not_improved_count: 1
Train Epoch: 16 [1/125 256/32000 (1%)] Loss: 2.05611 (QuantReg: 13.51428) QuantErr: 13.51428 batch_time=54.77479
Train Epoch: 16 [17/125 4352/32000 (14%)] Loss: 2.15506 (QuantReg: 13.44423) QuantErr: 13.44423 batch_time=0.74232
Train Epoch: 16 [33/125 8448/32000 (26%)] Loss: 1.94737 (QuantReg: 13.60122) QuantErr: 13.60122 batch_time=0.76274
Train Epoch: 16 [49/125 12544/32000 (39%)] Loss: 1.77963 (QuantReg: 13.54796) QuantErr: 13.54796 batch_time=0.73750
Train Epoch: 16 [65/125 16640/32000 (52%)] Loss: 1.97634 (QuantReg: 13.57346) QuantErr: 13.57346 batch_time=12.45718
Train Epoch: 16 [81/125 20736/32000 (65%)] Loss: 2.07200 (QuantReg: 13.65373) QuantErr: 13.65373 batch_time=0.75078
Train Epoch: 16 [97/125 24832/32000 (78%)] Loss: 1.92398 (QuantReg: 13.93964) QuantErr: 13.93964 batch_time=0.87035
Train Epoch: 16 [113/125 28928/32000 (90%)] Loss: 1.89852 (QuantReg: 13.46017) QuantErr: 13.46017 batch_time=0.73997
Train Epoch: 16 codebook_update_time=1.74844
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch16.pth ...
Done in 14.379s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch16.pth ...
Done in 18.167s
removing stale ckpt [epoch 15] [took 0.00s]
epoch : 16
loss : 2.0726689710617063
quant_reg : 13.587524971008301
quant_err : 13.587524971008301
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
MSRVTT_full_val/t2v_metrics/R5: 69.21529175050301
MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.006036217303823
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.46195634942725
MSRVTT_full_val/v2t_metrics/R1: 37.42454728370221
MSRVTT_full_val/v2t_metrics/R5: 74.24547283702213
MSRVTT_full_val/v2t_metrics/R10: 88.53118712273641
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.830985915492958
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.6576746839343
MSRVTT_full_test/t2v_metrics/R1: 12.642140468227424
MSRVTT_full_test/t2v_metrics/R5: 35.752508361204015
MSRVTT_full_test/t2v_metrics/R10: 49.66555183946488
MSRVTT_full_test/t2v_metrics/R50: 82.04013377926421
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 40.84615384615385
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.20941833131043
MSRVTT_full_test/v2t_metrics/R1: 16.08695652173913
MSRVTT_full_test/v2t_metrics/R5: 42.94314381270903
MSRVTT_full_test/v2t_metrics/R10: 57.05685618729097
MSRVTT_full_test/v2t_metrics/R50: 87.62541806020067
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 28.891638795986623
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.032343337463125
mnt_best : 28.20941833131043
not_improved_count: 0
Train Epoch: 17 [1/125 256/32000 (1%)] Loss: 1.96222 (QuantReg: 13.48277) QuantErr: 13.48277 batch_time=39.67730
Train Epoch: 17 [17/125 4352/32000 (14%)] Loss: 1.94576 (QuantReg: 13.60441) QuantErr: 13.60441 batch_time=1.75576
Train Epoch: 17 [33/125 8448/32000 (26%)] Loss: 1.91727 (QuantReg: 13.51670) QuantErr: 13.51670 batch_time=0.73464
Train Epoch: 17 [49/125 12544/32000 (39%)] Loss: 2.29541 (QuantReg: 13.65309) QuantErr: 13.65309 batch_time=0.98498
Train Epoch: 17 [65/125 16640/32000 (52%)] Loss: 1.97408 (QuantReg: 13.87374) QuantErr: 13.87374 batch_time=3.13784
Train Epoch: 17 [81/125 20736/32000 (65%)] Loss: 1.91609 (QuantReg: 13.75805) QuantErr: 13.75805 batch_time=1.77229
Train Epoch: 17 [97/125 24832/32000 (78%)] Loss: 2.02721 (QuantReg: 13.63551) QuantErr: 13.63551 batch_time=0.77353
Train Epoch: 17 [113/125 28928/32000 (90%)] Loss: 1.85816 (QuantReg: 13.76242) QuantErr: 13.76242 batch_time=0.74031
Train Epoch: 17 codebook_update_time=1.63715
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch17.pth ...
Done in 3.805s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch17.pth ...
Done in 7.512s
removing stale ckpt [epoch 16] [took 0.00s]
epoch : 17
loss : 1.9942949647903443
quant_reg : 13.642178482055664
quant_err : 13.642178482055664
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 2125
MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
MSRVTT_full_val/t2v_metrics/R50: 98.59154929577464
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.851106639839034
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.14638065454504
MSRVTT_full_val/v2t_metrics/R1: 38.22937625754527
MSRVTT_full_val/v2t_metrics/R5: 75.65392354124748
MSRVTT_full_val/v2t_metrics/R10: 90.3420523138833
MSRVTT_full_val/v2t_metrics/R50: 98.99396378269617
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.800804828973843
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 63.930221169011666
MSRVTT_full_test/t2v_metrics/R1: 13.311036789297658
MSRVTT_full_test/t2v_metrics/R5: 36.52173913043478
MSRVTT_full_test/t2v_metrics/R10: 50.13377926421405
MSRVTT_full_test/t2v_metrics/R50: 81.83946488294315
MSRVTT_full_test/t2v_metrics/MedR: 10.0
MSRVTT_full_test/t2v_metrics/MeanR: 39.57123745819398
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.993318479064907
MSRVTT_full_test/v2t_metrics/R1: 15.585284280936454
MSRVTT_full_test/v2t_metrics/R5: 43.47826086956522
MSRVTT_full_test/v2t_metrics/R10: 57.725752508361204
MSRVTT_full_test/v2t_metrics/R50: 87.59197324414716
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 27.868729096989966
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.94575703589024
mnt_best : 28.993318479064907
not_improved_count: 0
Train Epoch: 18 [1/125 256/32000 (1%)] Loss: 2.25817 (QuantReg: 13.45359) QuantErr: 13.45359 batch_time=40.69848
Train Epoch: 18 [17/125 4352/32000 (14%)] Loss: 2.09223 (QuantReg: 13.42003) QuantErr: 13.42003 batch_time=0.83575
Train Epoch: 18 [33/125 8448/32000 (26%)] Loss: 2.03725 (QuantReg: 13.58267) QuantErr: 13.58267 batch_time=0.73863
Train Epoch: 18 [49/125 12544/32000 (39%)] Loss: 2.08691 (QuantReg: 13.68881) QuantErr: 13.68881 batch_time=0.75846
Train Epoch: 18 [65/125 16640/32000 (52%)] Loss: 1.92127 (QuantReg: 13.76549) QuantErr: 13.76549 batch_time=2.27340
Train Epoch: 18 [81/125 20736/32000 (65%)] Loss: 2.04262 (QuantReg: 13.65627) QuantErr: 13.65627 batch_time=0.90319
Train Epoch: 18 [97/125 24832/32000 (78%)] Loss: 1.92782 (QuantReg: 13.77163) QuantErr: 13.77163 batch_time=0.74914
Train Epoch: 18 [113/125 28928/32000 (90%)] Loss: 1.95159 (QuantReg: 13.86889) QuantErr: 13.86889 batch_time=0.89859
Train Epoch: 18 codebook_update_time=1.67084
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch18.pth ...
Done in 3.573s
removing stale ckpt [epoch 17] [took 0.00s]
epoch : 18
loss : 1.9642371282577515
quant_reg : 13.664888031005859
quant_err : 13.664888031005859
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 34.20523138832998
MSRVTT_full_val/t2v_metrics/R5: 69.01408450704226
MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.812877263581489
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.67909179559059
MSRVTT_full_val/v2t_metrics/R1: 37.42454728370221
MSRVTT_full_val/v2t_metrics/R5: 76.05633802816901
MSRVTT_full_val/v2t_metrics/R10: 88.12877263581488
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.084507042253521
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 63.067153449339706
MSRVTT_full_test/t2v_metrics/R1: 13.34448160535117
MSRVTT_full_test/t2v_metrics/R5: 36.52173913043478
MSRVTT_full_test/t2v_metrics/R10: 49.264214046822744
MSRVTT_full_test/t2v_metrics/R50: 81.57190635451505
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 40.72809364548495
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.848832268002198
MSRVTT_full_test/v2t_metrics/R1: 16.354515050167223
MSRVTT_full_test/v2t_metrics/R5: 43.41137123745819
MSRVTT_full_test/v2t_metrics/R10: 57.65886287625418
MSRVTT_full_test/v2t_metrics/R50: 87.22408026755853
MSRVTT_full_test/v2t_metrics/MedR: 7.0
MSRVTT_full_test/v2t_metrics/MeanR: 28.896989966555186
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.46426989841355
mnt_best : 28.993318479064907
not_improved_count: 1
Train Epoch: 19 [1/125 256/32000 (1%)] Loss: 1.85557 (QuantReg: 13.43442) QuantErr: 13.43442 batch_time=46.73158
Train Epoch: 19 [17/125 4352/32000 (14%)] Loss: 1.87466 (QuantReg: 13.64847) QuantErr: 13.64847 batch_time=0.73304
Train Epoch: 19 [33/125 8448/32000 (26%)] Loss: 1.73319 (QuantReg: 13.57274) QuantErr: 13.57274 batch_time=0.73731
Train Epoch: 19 [49/125 12544/32000 (39%)] Loss: 1.72682 (QuantReg: 13.59343) QuantErr: 13.59343 batch_time=0.75036
Train Epoch: 19 [65/125 16640/32000 (52%)] Loss: 2.13074 (QuantReg: 13.47211) QuantErr: 13.47211 batch_time=6.92884
Train Epoch: 19 [81/125 20736/32000 (65%)] Loss: 2.01865 (QuantReg: 13.81827) QuantErr: 13.81827 batch_time=0.78000
Train Epoch: 19 [97/125 24832/32000 (78%)] Loss: 1.66375 (QuantReg: 13.78869) QuantErr: 13.78869 batch_time=0.86811
Train Epoch: 19 [113/125 28928/32000 (90%)] Loss: 1.95648 (QuantReg: 13.77504) QuantErr: 13.77504 batch_time=0.75411
Train Epoch: 19 codebook_update_time=1.61423
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs256/checkpoint-epoch19.pth ...
Done in 16.292s
removing stale ckpt [epoch 18] [took 0.00s]
epoch : 19
loss : 1.929776969909668
quant_reg : 13.700823638916015
quant_err : 13.700823638916015
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 2375
MSRVTT_full_val/t2v_metrics/R1: 32.99798792756539
MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
MSRVTT_full_val/t2v_metrics/R10: 81.89134808853119
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.104627766599599
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.628951326700495
MSRVTT_full_val/v2t_metrics/R1: 36.21730382293762
MSRVTT_full_val/v2t_metrics/R5: 76.65995975855131
MSRVTT_full_val/v2t_metrics/R10: 87.32394366197182
MSRVTT_full_val/v2t_metrics/R50: 98.79275653923541
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.875251509054326
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.35520069349817
MSRVTT_full_test/t2v_metrics/R1: 13.244147157190636
MSRVTT_full_test/t2v_metrics/R5: 36.187290969899664
MSRVTT_full_test/t2v_metrics/R10: 48.99665551839465
MSRVTT_full_test/t2v_metrics/R50: 81.43812709030101
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 41.52608695652174
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.636207595719526
MSRVTT_full_test/v2t_metrics/R1: 16.555183946488295
MSRVTT_full_test/v2t_metrics/R5: 43.84615384615385