-
Notifications
You must be signed in to change notification settings - Fork 4
/
HCQ_MSRVTT_1kB_M64.txt
2593 lines (2593 loc) · 192 KB
/
HCQ_MSRVTT_1kB_M64.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64
Preparing the dataloaders ...
Loading dataset MSRVTT_miech_trainval in ram ...
Finish loading dataset MSRVTT_miech_trainval in ram, taking 761.0753781795502 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 115.05409836769104 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 69.56220293045044 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch0.pth ...
Done in 1.852s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch0.pth ...
Done in 3.574s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_miech_test/t2v_metrics/R1: 0.0
MSRVTT_miech_test/t2v_metrics/R5: 0.7
MSRVTT_miech_test/t2v_metrics/R10: 1.1
MSRVTT_miech_test/t2v_metrics/R50: 4.9
MSRVTT_miech_test/t2v_metrics/MedR: 523.0
MSRVTT_miech_test/t2v_metrics/MeanR: 508.802
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_miech_test/v2t_metrics/R1: 0.0
MSRVTT_miech_test/v2t_metrics/R5: 0.5
MSRVTT_miech_test/v2t_metrics/R10: 1.0
MSRVTT_miech_test/v2t_metrics/R50: 4.8
MSRVTT_miech_test/v2t_metrics/MedR: 507.5
MSRVTT_miech_test/v2t_metrics/MeanR: 506.8005
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.80689 (QuantReg: 21.61739) QuantErr: 21.61739 batch_time=26.46206
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.63621 (QuantReg: 21.86898) QuantErr: 21.86898 batch_time=0.67441
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.38625 (QuantReg: 21.83026) QuantErr: 21.83026 batch_time=0.61406
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.53236 (QuantReg: 21.84714) QuantErr: 21.84714 batch_time=0.59176
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.49758 (QuantReg: 21.85103) QuantErr: 21.85103 batch_time=0.64003
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.22985 (QuantReg: 21.85247) QuantErr: 21.85247 batch_time=0.60288
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.98835 (QuantReg: 21.84048) QuantErr: 21.84048 batch_time=0.59065
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.65337 (QuantReg: 21.85333) QuantErr: 21.85333 batch_time=0.60853
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.18682 (QuantReg: 21.86820) QuantErr: 21.86820 batch_time=0.60672
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.30206 (QuantReg: 21.87316) QuantErr: 21.87316 batch_time=0.59585
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.14717 (QuantReg: 21.90720) QuantErr: 21.90720 batch_time=0.61306
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.30889 (QuantReg: 21.85962) QuantErr: 21.85962 batch_time=0.63732
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.63169 (QuantReg: 21.82336) QuantErr: 21.82336 batch_time=0.60350
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.69892 (QuantReg: 21.88597) QuantErr: 21.88597 batch_time=0.60292
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.65085 (QuantReg: 21.89581) QuantErr: 21.89581 batch_time=0.61411
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.37817 (QuantReg: 21.86927) QuantErr: 21.86927 batch_time=0.59028
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.24824 (QuantReg: 21.86597) QuantErr: 21.86597 batch_time=0.71168
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.61869 (QuantReg: 21.85706) QuantErr: 21.85706 batch_time=0.60458
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.52333 (QuantReg: 21.86589) QuantErr: 21.86589 batch_time=0.60844
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.60473 (QuantReg: 21.84974) QuantErr: 21.84974 batch_time=0.61137
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.43792 (QuantReg: 21.83267) QuantErr: 21.83267 batch_time=0.61476
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.84030 (QuantReg: 21.86100) QuantErr: 21.86100 batch_time=0.59293
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.03878 (QuantReg: 21.88445) QuantErr: 21.88445 batch_time=0.61460
Train Epoch: 1 codebook_update_time=3.78691
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch1.pth ...
Done in 4.174s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch1.pth ...
Done in 8.277s
epoch : 1
loss : 5.4425521602630615
quant_reg : 21.86653443145752
quant_err : 21.86653443145752
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_miech_test/t2v_metrics/R1: 11.2
MSRVTT_miech_test/t2v_metrics/R5: 34.0
MSRVTT_miech_test/t2v_metrics/R10: 46.8
MSRVTT_miech_test/t2v_metrics/R50: 80.7
MSRVTT_miech_test/t2v_metrics/MedR: 13.0
MSRVTT_miech_test/t2v_metrics/MeanR: 41.417
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.120466617212212
MSRVTT_miech_test/v2t_metrics/R1: 11.2
MSRVTT_miech_test/v2t_metrics/R5: 33.5
MSRVTT_miech_test/v2t_metrics/R10: 48.2
MSRVTT_miech_test/v2t_metrics/R50: 79.8
MSRVTT_miech_test/v2t_metrics/MedR: 11.0
MSRVTT_miech_test/v2t_metrics/MeanR: 43.439
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.248427419019453
mnt_best : 26.120466617212212
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 3.89990 (QuantReg: 13.75116) QuantErr: 13.75116 batch_time=29.41421
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.93455 (QuantReg: 14.15730) QuantErr: 14.15730 batch_time=0.62893
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.10007 (QuantReg: 14.36181) QuantErr: 14.36181 batch_time=0.59603
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.01424 (QuantReg: 14.47066) QuantErr: 14.47066 batch_time=0.60231
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.66827 (QuantReg: 14.46198) QuantErr: 14.46198 batch_time=0.60120
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.45136 (QuantReg: 14.81817) QuantErr: 14.81817 batch_time=0.61145
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.84292 (QuantReg: 15.02799) QuantErr: 15.02799 batch_time=0.63649
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.59676 (QuantReg: 14.93729) QuantErr: 14.93729 batch_time=0.59250
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.54894 (QuantReg: 15.20818) QuantErr: 15.20818 batch_time=3.90026
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.94475 (QuantReg: 15.76972) QuantErr: 15.76972 batch_time=0.61459
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.67448 (QuantReg: 15.38854) QuantErr: 15.38854 batch_time=0.60056
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.21125 (QuantReg: 15.60380) QuantErr: 15.60380 batch_time=0.60204
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.57731 (QuantReg: 15.64485) QuantErr: 15.64485 batch_time=0.59932
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.79208 (QuantReg: 15.86265) QuantErr: 15.86265 batch_time=0.61421
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.48406 (QuantReg: 15.73951) QuantErr: 15.73951 batch_time=0.67458
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 4.06994 (QuantReg: 16.00794) QuantErr: 16.00794 batch_time=0.60284
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.68119 (QuantReg: 15.97679) QuantErr: 15.97679 batch_time=0.59880
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 2.91067 (QuantReg: 16.40505) QuantErr: 16.40505 batch_time=0.59838
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 4.00543 (QuantReg: 16.03250) QuantErr: 16.03250 batch_time=0.60166
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.66699 (QuantReg: 16.38460) QuantErr: 16.38460 batch_time=0.60380
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.59671 (QuantReg: 16.77670) QuantErr: 16.77670 batch_time=0.64985
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.38904 (QuantReg: 17.00577) QuantErr: 17.00577 batch_time=0.60214
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.09327 (QuantReg: 16.83291) QuantErr: 16.83291 batch_time=0.65506
Train Epoch: 2 codebook_update_time=4.02926
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch2.pth ...
Done in 4.087s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch2.pth ...
Done in 8.277s
removing stale ckpt [epoch 1] [took 0.02s]
removing stale ckpt [epoch 0] [took 0.04s]
epoch : 2
loss : 3.6996427068710327
quant_reg : 15.547572895050049
quant_err : 15.547572895050049
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_miech_test/t2v_metrics/R1: 14.2
MSRVTT_miech_test/t2v_metrics/R5: 39.3
MSRVTT_miech_test/t2v_metrics/R10: 53.5
MSRVTT_miech_test/t2v_metrics/R50: 85.3
MSRVTT_miech_test/t2v_metrics/MedR: 9.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.103
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.022602316349555
MSRVTT_miech_test/v2t_metrics/R1: 14.5
MSRVTT_miech_test/v2t_metrics/R5: 38.3
MSRVTT_miech_test/v2t_metrics/R10: 53.9
MSRVTT_miech_test/v2t_metrics/R50: 84.5
MSRVTT_miech_test/v2t_metrics/MedR: 9.0
MSRVTT_miech_test/v2t_metrics/MeanR: 36.7725
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.04930240116911
mnt_best : 31.022602316349555
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.88093 (QuantReg: 14.23176) QuantErr: 14.23176 batch_time=28.88914
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.22671 (QuantReg: 13.97453) QuantErr: 13.97453 batch_time=0.60104
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.29557 (QuantReg: 14.26321) QuantErr: 14.26321 batch_time=0.60241
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.90507 (QuantReg: 14.10314) QuantErr: 14.10314 batch_time=0.60632
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.15321 (QuantReg: 14.64002) QuantErr: 14.64002 batch_time=0.60905
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.22352 (QuantReg: 14.49047) QuantErr: 14.49047 batch_time=0.60314
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.96679 (QuantReg: 14.35007) QuantErr: 14.35007 batch_time=3.35035
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 2.83874 (QuantReg: 14.39777) QuantErr: 14.39777 batch_time=0.59596
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.07546 (QuantReg: 14.75193) QuantErr: 14.75193 batch_time=0.62116
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.03370 (QuantReg: 14.81681) QuantErr: 14.81681 batch_time=0.60404
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.97703 (QuantReg: 14.68224) QuantErr: 14.68224 batch_time=0.60515
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.09059 (QuantReg: 14.56673) QuantErr: 14.56673 batch_time=0.59836
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.30012 (QuantReg: 14.91190) QuantErr: 14.91190 batch_time=0.58825
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.09099 (QuantReg: 14.99559) QuantErr: 14.99559 batch_time=0.61771
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.06231 (QuantReg: 14.69029) QuantErr: 14.69029 batch_time=0.84077
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.81020 (QuantReg: 15.28367) QuantErr: 15.28367 batch_time=0.60316
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.65409 (QuantReg: 15.10687) QuantErr: 15.10687 batch_time=0.60858
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.18628 (QuantReg: 15.19263) QuantErr: 15.19263 batch_time=0.59571
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.26269 (QuantReg: 15.28325) QuantErr: 15.28325 batch_time=0.62217
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.85857 (QuantReg: 15.09597) QuantErr: 15.09597 batch_time=0.71451
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.36129 (QuantReg: 15.19753) QuantErr: 15.19753 batch_time=0.62229
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.52098 (QuantReg: 15.51673) QuantErr: 15.51673 batch_time=0.64714
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.52727 (QuantReg: 15.45298) QuantErr: 15.45298 batch_time=0.64194
Train Epoch: 3 codebook_update_time=4.05538
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch3.pth ...
Done in 4.107s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch3.pth ...
Done in 8.122s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.1380692586898804
quant_reg : 14.818924987792968
quant_err : 14.818924987792968
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_miech_test/t2v_metrics/R1: 17.6
MSRVTT_miech_test/t2v_metrics/R5: 44.3
MSRVTT_miech_test/t2v_metrics/R10: 56.2
MSRVTT_miech_test/t2v_metrics/R50: 86.5
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.029
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.25474436083248
MSRVTT_miech_test/v2t_metrics/R1: 17.1
MSRVTT_miech_test/v2t_metrics/R5: 43.2
MSRVTT_miech_test/v2t_metrics/R10: 57.8
MSRVTT_miech_test/v2t_metrics/R50: 85.9
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 32.200500000000005
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.95177467088659
mnt_best : 35.25474436083248
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.81359 (QuantReg: 14.24006) QuantErr: 14.24006 batch_time=27.64650
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.79335 (QuantReg: 14.42676) QuantErr: 14.42676 batch_time=0.58773
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.61062 (QuantReg: 14.27746) QuantErr: 14.27746 batch_time=0.59084
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.03724 (QuantReg: 14.29408) QuantErr: 14.29408 batch_time=0.62253
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.81409 (QuantReg: 14.42271) QuantErr: 14.42271 batch_time=0.58268
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.48836 (QuantReg: 14.55369) QuantErr: 14.55369 batch_time=0.60205
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.04981 (QuantReg: 14.43386) QuantErr: 14.43386 batch_time=1.42923
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.31980 (QuantReg: 14.45170) QuantErr: 14.45170 batch_time=1.13993
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.69964 (QuantReg: 14.51608) QuantErr: 14.51608 batch_time=0.58227
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.63789 (QuantReg: 14.75267) QuantErr: 14.75267 batch_time=0.58155
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.16937 (QuantReg: 14.72878) QuantErr: 14.72878 batch_time=0.71950
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.99498 (QuantReg: 14.69505) QuantErr: 14.69505 batch_time=0.59201
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.48542 (QuantReg: 14.76765) QuantErr: 14.76765 batch_time=0.65011
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.60725 (QuantReg: 14.55004) QuantErr: 14.55004 batch_time=0.59048
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.66184 (QuantReg: 14.89371) QuantErr: 14.89371 batch_time=0.58754
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.97683 (QuantReg: 14.87852) QuantErr: 14.87852 batch_time=1.24729
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.60383 (QuantReg: 14.95671) QuantErr: 14.95671 batch_time=0.59736
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.88287 (QuantReg: 14.64297) QuantErr: 14.64297 batch_time=0.58750
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.87830 (QuantReg: 15.09397) QuantErr: 15.09397 batch_time=0.66564
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.51652 (QuantReg: 14.83739) QuantErr: 14.83739 batch_time=0.60038
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.58007 (QuantReg: 15.28480) QuantErr: 15.28480 batch_time=0.59838
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.44115 (QuantReg: 14.96461) QuantErr: 14.96461 batch_time=0.59712
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.43351 (QuantReg: 14.94267) QuantErr: 14.94267 batch_time=0.62443
Train Epoch: 4 codebook_update_time=5.06289
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch4.pth ...
Done in 4.033s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch4.pth ...
Done in 7.981s
removing stale ckpt [epoch 3] [took 0.03s]
epoch : 4
loss : 2.739317794799805
quant_reg : 14.717924499511719
quant_err : 14.717924499511719
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_miech_test/t2v_metrics/R1: 16.6
MSRVTT_miech_test/t2v_metrics/R5: 44.9
MSRVTT_miech_test/t2v_metrics/R10: 58.9
MSRVTT_miech_test/t2v_metrics/R50: 87.1
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.086
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.276858895105335
MSRVTT_miech_test/v2t_metrics/R1: 16.9
MSRVTT_miech_test/v2t_metrics/R5: 45.9
MSRVTT_miech_test/v2t_metrics/R10: 60.0
MSRVTT_miech_test/v2t_metrics/R50: 87.8
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 29.553
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.970809670971974
mnt_best : 35.276858895105335
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.14766 (QuantReg: 14.13375) QuantErr: 14.13375 batch_time=31.17822
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.18346 (QuantReg: 14.47845) QuantErr: 14.47845 batch_time=3.69818
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.40199 (QuantReg: 14.29872) QuantErr: 14.29872 batch_time=0.60734
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.50159 (QuantReg: 14.38083) QuantErr: 14.38083 batch_time=0.58046
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.79801 (QuantReg: 14.37269) QuantErr: 14.37269 batch_time=0.58109
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.07261 (QuantReg: 14.52336) QuantErr: 14.52336 batch_time=0.58827
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.52705 (QuantReg: 14.61381) QuantErr: 14.61381 batch_time=0.60775
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.50507 (QuantReg: 14.56839) QuantErr: 14.56839 batch_time=0.60349
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.28595 (QuantReg: 14.39405) QuantErr: 14.39405 batch_time=0.60118
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.31982 (QuantReg: 14.79505) QuantErr: 14.79505 batch_time=0.63819
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.38531 (QuantReg: 14.67904) QuantErr: 14.67904 batch_time=0.60226
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.70224 (QuantReg: 14.91478) QuantErr: 14.91478 batch_time=0.58448
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.71676 (QuantReg: 14.78054) QuantErr: 14.78054 batch_time=0.59168
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.48618 (QuantReg: 14.68156) QuantErr: 14.68156 batch_time=0.60771
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.43500 (QuantReg: 14.84494) QuantErr: 14.84494 batch_time=0.58888
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 1.91214 (QuantReg: 15.20708) QuantErr: 15.20708 batch_time=0.62013
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.48913 (QuantReg: 14.96337) QuantErr: 14.96337 batch_time=0.59070
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.44651 (QuantReg: 14.75393) QuantErr: 14.75393 batch_time=0.60013
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.35652 (QuantReg: 14.78498) QuantErr: 14.78498 batch_time=0.59027
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.74777 (QuantReg: 14.80246) QuantErr: 14.80246 batch_time=0.58762
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.41953 (QuantReg: 14.85865) QuantErr: 14.85865 batch_time=0.59308
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 1.74494 (QuantReg: 14.79816) QuantErr: 14.79816 batch_time=0.60555
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.13861 (QuantReg: 15.17292) QuantErr: 15.17292 batch_time=0.60242
Train Epoch: 5 codebook_update_time=3.97649
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch5.pth ...
Done in 3.927s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch5.pth ...
Done in 8.017s
removing stale ckpt [epoch 4] [took 0.02s]
epoch : 5
loss : 2.502525712490082
quant_reg : 14.681871486663818
quant_err : 14.681871486663818
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_miech_test/t2v_metrics/R1: 17.8
MSRVTT_miech_test/t2v_metrics/R5: 47.2
MSRVTT_miech_test/t2v_metrics/R10: 60.3
MSRVTT_miech_test/t2v_metrics/R50: 87.4
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.242999999999995
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.00210555341811
MSRVTT_miech_test/v2t_metrics/R1: 19.4
MSRVTT_miech_test/v2t_metrics/R5: 47.3
MSRVTT_miech_test/v2t_metrics/R10: 62.0
MSRVTT_miech_test/v2t_metrics/R50: 87.9
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 29.7655
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.460788773079535
mnt_best : 37.00210555341811
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.32374 (QuantReg: 14.44416) QuantErr: 14.44416 batch_time=28.36378
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.12438 (QuantReg: 14.76052) QuantErr: 14.76052 batch_time=0.60007
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.45501 (QuantReg: 14.35480) QuantErr: 14.35480 batch_time=0.58862
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.59534 (QuantReg: 14.58358) QuantErr: 14.58358 batch_time=0.62195
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.16850 (QuantReg: 14.43482) QuantErr: 14.43482 batch_time=0.59893
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.34369 (QuantReg: 14.80093) QuantErr: 14.80093 batch_time=0.60113
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.50257 (QuantReg: 14.64541) QuantErr: 14.64541 batch_time=0.59462
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.45166 (QuantReg: 14.50851) QuantErr: 14.50851 batch_time=0.62514
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.39054 (QuantReg: 14.53029) QuantErr: 14.53029 batch_time=0.58800
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.22052 (QuantReg: 14.60806) QuantErr: 14.60806 batch_time=0.59201
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.03445 (QuantReg: 14.75852) QuantErr: 14.75852 batch_time=0.59767
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.52390 (QuantReg: 15.02098) QuantErr: 15.02098 batch_time=0.62776
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.30012 (QuantReg: 14.76322) QuantErr: 14.76322 batch_time=0.59045
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.58755 (QuantReg: 14.81835) QuantErr: 14.81835 batch_time=0.59305
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.45380 (QuantReg: 14.67504) QuantErr: 14.67504 batch_time=0.61162
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.35978 (QuantReg: 14.83429) QuantErr: 14.83429 batch_time=0.62800
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.50642 (QuantReg: 14.76739) QuantErr: 14.76739 batch_time=0.66178
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.56398 (QuantReg: 14.92986) QuantErr: 14.92986 batch_time=0.58156
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 1.86697 (QuantReg: 14.82586) QuantErr: 14.82586 batch_time=0.59959
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.37944 (QuantReg: 14.84487) QuantErr: 14.84487 batch_time=0.59557
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.35273 (QuantReg: 14.80443) QuantErr: 14.80443 batch_time=0.59853
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.23898 (QuantReg: 15.06628) QuantErr: 15.06628 batch_time=0.61757
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.05128 (QuantReg: 14.69686) QuantErr: 14.69686 batch_time=0.60692
Train Epoch: 6 codebook_update_time=4.37571
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch6.pth ...
Done in 3.818s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch6.pth ...
Done in 7.671s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.2851373558044434
quant_reg : 14.754763664245605
quant_err : 14.754763664245605
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_miech_test/t2v_metrics/R1: 19.2
MSRVTT_miech_test/t2v_metrics/R5: 48.4
MSRVTT_miech_test/t2v_metrics/R10: 61.8
MSRVTT_miech_test/t2v_metrics/R50: 88.4
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.664
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.58143307797925
MSRVTT_miech_test/v2t_metrics/R1: 19.4
MSRVTT_miech_test/v2t_metrics/R5: 48.6
MSRVTT_miech_test/v2t_metrics/R10: 62.4
MSRVTT_miech_test/v2t_metrics/R50: 87.9
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 29.888
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.89324651517254
mnt_best : 38.58143307797925
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.23000 (QuantReg: 14.48473) QuantErr: 14.48473 batch_time=31.21400
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.22544 (QuantReg: 14.32977) QuantErr: 14.32977 batch_time=0.59828
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.37330 (QuantReg: 14.99013) QuantErr: 14.99013 batch_time=0.59859
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.30142 (QuantReg: 14.36585) QuantErr: 14.36585 batch_time=0.59879
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.03155 (QuantReg: 14.61094) QuantErr: 14.61094 batch_time=0.60142
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.26770 (QuantReg: 14.60660) QuantErr: 14.60660 batch_time=0.60232
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.21418 (QuantReg: 14.77715) QuantErr: 14.77715 batch_time=0.59506
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.15818 (QuantReg: 14.57628) QuantErr: 14.57628 batch_time=0.59534
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 1.97917 (QuantReg: 14.84319) QuantErr: 14.84319 batch_time=0.59263
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.09012 (QuantReg: 14.86288) QuantErr: 14.86288 batch_time=0.62714
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.21107 (QuantReg: 14.76042) QuantErr: 14.76042 batch_time=0.59086
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.32426 (QuantReg: 14.91878) QuantErr: 14.91878 batch_time=0.62398
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.85240 (QuantReg: 14.75039) QuantErr: 14.75039 batch_time=0.59703
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.04355 (QuantReg: 14.99522) QuantErr: 14.99522 batch_time=3.60285
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.04333 (QuantReg: 15.03597) QuantErr: 15.03597 batch_time=0.58817
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.17871 (QuantReg: 15.03648) QuantErr: 15.03648 batch_time=0.57921
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.81740 (QuantReg: 14.85234) QuantErr: 14.85234 batch_time=1.12717
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.16148 (QuantReg: 14.75379) QuantErr: 14.75379 batch_time=0.61411
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.04853 (QuantReg: 14.95308) QuantErr: 14.95308 batch_time=0.59076
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.88731 (QuantReg: 15.01915) QuantErr: 15.01915 batch_time=0.58621
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.10361 (QuantReg: 15.01926) QuantErr: 15.01926 batch_time=0.59362
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.02650 (QuantReg: 14.94788) QuantErr: 14.94788 batch_time=0.61071
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 1.91864 (QuantReg: 14.89502) QuantErr: 14.89502 batch_time=0.60484
Train Epoch: 7 codebook_update_time=4.54340
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch7.pth ...
Done in 4.005s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch7.pth ...
Done in 8.001s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 2.120884237766266
quant_reg : 14.791919498443603
quant_err : 14.791919498443603
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_miech_test/t2v_metrics/R1: 20.0
MSRVTT_miech_test/t2v_metrics/R5: 48.4
MSRVTT_miech_test/t2v_metrics/R10: 63.0
MSRVTT_miech_test/t2v_metrics/R50: 88.1
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.612
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.36152978388916
MSRVTT_miech_test/v2t_metrics/R1: 18.6
MSRVTT_miech_test/v2t_metrics/R5: 49.8
MSRVTT_miech_test/v2t_metrics/R10: 63.1
MSRVTT_miech_test/v2t_metrics/R50: 88.8
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.69
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.808234161298884
mnt_best : 39.36152978388916
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.07529 (QuantReg: 14.60986) QuantErr: 14.60986 batch_time=30.49760
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.05213 (QuantReg: 14.79212) QuantErr: 14.79212 batch_time=0.58325
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.37250 (QuantReg: 14.62865) QuantErr: 14.62865 batch_time=1.47749
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.02796 (QuantReg: 14.64637) QuantErr: 14.64637 batch_time=1.39266
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.12715 (QuantReg: 14.98652) QuantErr: 14.98652 batch_time=0.59706
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.17683 (QuantReg: 14.75323) QuantErr: 14.75323 batch_time=0.66495
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.77324 (QuantReg: 14.67477) QuantErr: 14.67477 batch_time=0.58630
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.18399 (QuantReg: 14.55612) QuantErr: 14.55612 batch_time=0.59108
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.84133 (QuantReg: 14.66089) QuantErr: 14.66089 batch_time=0.61099
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.62530 (QuantReg: 15.06955) QuantErr: 15.06955 batch_time=0.61286
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.33189 (QuantReg: 15.24539) QuantErr: 15.24539 batch_time=0.58033
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 1.77556 (QuantReg: 14.69968) QuantErr: 14.69968 batch_time=0.58636
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.87359 (QuantReg: 14.95053) QuantErr: 14.95053 batch_time=0.60721
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.93535 (QuantReg: 14.90720) QuantErr: 14.90720 batch_time=1.64786
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 1.63299 (QuantReg: 14.78950) QuantErr: 14.78950 batch_time=0.59508
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.00994 (QuantReg: 14.74234) QuantErr: 14.74234 batch_time=0.58600
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.15415 (QuantReg: 14.93643) QuantErr: 14.93643 batch_time=0.57970
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 1.95618 (QuantReg: 15.17391) QuantErr: 15.17391 batch_time=0.59781
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.76858 (QuantReg: 14.70622) QuantErr: 14.70622 batch_time=0.59710
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.74370 (QuantReg: 15.05123) QuantErr: 15.05123 batch_time=0.58998
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.81226 (QuantReg: 14.76632) QuantErr: 14.76632 batch_time=0.58620
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 2.05034 (QuantReg: 14.84537) QuantErr: 14.84537 batch_time=0.60318
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.24107 (QuantReg: 15.07847) QuantErr: 15.07847 batch_time=0.59417
Train Epoch: 8 codebook_update_time=4.57266
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch8.pth ...
Done in 3.997s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 1.9887174010276794
quant_reg : 14.822891399383545
quant_err : 14.822891399383545
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_miech_test/t2v_metrics/R1: 19.4
MSRVTT_miech_test/t2v_metrics/R5: 48.7
MSRVTT_miech_test/t2v_metrics/R10: 63.4
MSRVTT_miech_test/t2v_metrics/R50: 89.1
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.3185
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.126708639361375
MSRVTT_miech_test/v2t_metrics/R1: 19.1
MSRVTT_miech_test/v2t_metrics/R5: 49.0
MSRVTT_miech_test/v2t_metrics/R10: 62.8
MSRVTT_miech_test/v2t_metrics/R50: 89.0
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.9375
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.88030803269421
mnt_best : 39.36152978388916
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.20204 (QuantReg: 14.75788) QuantErr: 14.75788 batch_time=27.35525
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.88458 (QuantReg: 14.37605) QuantErr: 14.37605 batch_time=0.59479
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.77332 (QuantReg: 14.40466) QuantErr: 14.40466 batch_time=0.58501
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 1.86775 (QuantReg: 14.78692) QuantErr: 14.78692 batch_time=0.59581
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.79040 (QuantReg: 14.70059) QuantErr: 14.70059 batch_time=0.58675
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.79373 (QuantReg: 14.73088) QuantErr: 14.73088 batch_time=0.61420
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.87072 (QuantReg: 14.88590) QuantErr: 14.88590 batch_time=1.26190
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.71292 (QuantReg: 14.72692) QuantErr: 14.72692 batch_time=0.64398
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.82437 (QuantReg: 14.49024) QuantErr: 14.49024 batch_time=0.58484
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.82553 (QuantReg: 14.69635) QuantErr: 14.69635 batch_time=0.60064
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.90921 (QuantReg: 14.73846) QuantErr: 14.73846 batch_time=0.59494
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.71639 (QuantReg: 15.28911) QuantErr: 15.28911 batch_time=0.59808
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.56768 (QuantReg: 14.95299) QuantErr: 14.95299 batch_time=0.58931
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.72648 (QuantReg: 15.09751) QuantErr: 15.09751 batch_time=1.46599
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.47037 (QuantReg: 14.87379) QuantErr: 14.87379 batch_time=0.58097
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.90998 (QuantReg: 14.97886) QuantErr: 14.97886 batch_time=0.61588
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.77107 (QuantReg: 14.86889) QuantErr: 14.86889 batch_time=0.59391
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.38728 (QuantReg: 14.94245) QuantErr: 14.94245 batch_time=0.59832
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.59003 (QuantReg: 14.93627) QuantErr: 14.93627 batch_time=0.59554
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.10246 (QuantReg: 14.89772) QuantErr: 14.89772 batch_time=0.60214
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.05000 (QuantReg: 14.91180) QuantErr: 14.91180 batch_time=0.60778
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.74204 (QuantReg: 14.90206) QuantErr: 14.90206 batch_time=0.58430
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.66911 (QuantReg: 15.04640) QuantErr: 15.04640 batch_time=0.59288
Train Epoch: 9 codebook_update_time=3.93517
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch9.pth ...
Done in 4.307s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch9.pth ...
Done in 8.376s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 1.8756747884750367
quant_reg : 14.861548473358154
quant_err : 14.861548473358154
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_miech_test/t2v_metrics/R1: 19.8
MSRVTT_miech_test/t2v_metrics/R5: 50.1
MSRVTT_miech_test/t2v_metrics/R10: 63.1
MSRVTT_miech_test/t2v_metrics/R50: 89.2
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.211
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.70489865042954
MSRVTT_miech_test/v2t_metrics/R1: 20.4
MSRVTT_miech_test/v2t_metrics/R5: 50.3
MSRVTT_miech_test/v2t_metrics/R10: 64.1
MSRVTT_miech_test/v2t_metrics/R50: 89.6
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.208
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.366279904851936
mnt_best : 39.70489865042954
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.91570 (QuantReg: 14.53606) QuantErr: 14.53606 batch_time=27.29238
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.77271 (QuantReg: 15.00795) QuantErr: 15.00795 batch_time=0.58064
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.17559 (QuantReg: 14.57781) QuantErr: 14.57781 batch_time=0.58546
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.90110 (QuantReg: 14.85778) QuantErr: 14.85778 batch_time=0.62118
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.51394 (QuantReg: 14.90159) QuantErr: 14.90159 batch_time=0.88402
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.87547 (QuantReg: 14.74935) QuantErr: 14.74935 batch_time=0.64089
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.60216 (QuantReg: 14.96391) QuantErr: 14.96391 batch_time=0.60904
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.72815 (QuantReg: 14.73330) QuantErr: 14.73330 batch_time=0.62333
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.39166 (QuantReg: 15.03193) QuantErr: 15.03193 batch_time=0.58409
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.92677 (QuantReg: 15.05977) QuantErr: 15.05977 batch_time=0.58821
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.05776 (QuantReg: 14.98540) QuantErr: 14.98540 batch_time=0.61157
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.08982 (QuantReg: 14.87157) QuantErr: 14.87157 batch_time=0.59504
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.46213 (QuantReg: 15.02699) QuantErr: 15.02699 batch_time=0.60545
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.16663 (QuantReg: 14.90733) QuantErr: 14.90733 batch_time=0.59196
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.82410 (QuantReg: 15.09748) QuantErr: 15.09748 batch_time=0.60577
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.83924 (QuantReg: 14.91338) QuantErr: 14.91338 batch_time=0.60559
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.00632 (QuantReg: 14.78418) QuantErr: 14.78418 batch_time=0.60809
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.61730 (QuantReg: 15.10086) QuantErr: 15.10086 batch_time=0.60555
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.04486 (QuantReg: 15.01629) QuantErr: 15.01629 batch_time=0.59840
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.75028 (QuantReg: 15.22220) QuantErr: 15.22220 batch_time=1.20320
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.90776 (QuantReg: 15.15424) QuantErr: 15.15424 batch_time=0.59918
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.68707 (QuantReg: 14.97731) QuantErr: 14.97731 batch_time=2.20291
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.84623 (QuantReg: 14.81515) QuantErr: 14.81515 batch_time=0.60092
Train Epoch: 10 codebook_update_time=3.99600
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch10.pth ...
Done in 6.351s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch10.pth ...
Done in 10.983s
removing stale ckpt [epoch 9] [took 0.18s]
epoch : 10
loss : 1.7755509071350097
quant_reg : 14.902017719268798
quant_err : 14.902017719268798
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_miech_test/t2v_metrics/R1: 20.7
MSRVTT_miech_test/t2v_metrics/R5: 51.5
MSRVTT_miech_test/t2v_metrics/R10: 64.1
MSRVTT_miech_test/t2v_metrics/R50: 88.5
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.949
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.88323011338325
MSRVTT_miech_test/v2t_metrics/R1: 21.6
MSRVTT_miech_test/v2t_metrics/R5: 51.2
MSRVTT_miech_test/v2t_metrics/R10: 63.8
MSRVTT_miech_test/v2t_metrics/R50: 88.8
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.479
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.322012392669066
mnt_best : 40.88323011338325
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.87073 (QuantReg: 14.87761) QuantErr: 14.87761 batch_time=28.03240
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.47764 (QuantReg: 15.02681) QuantErr: 15.02681 batch_time=0.60799
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.88779 (QuantReg: 14.91629) QuantErr: 14.91629 batch_time=3.87776
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.16635 (QuantReg: 14.48271) QuantErr: 14.48271 batch_time=0.58156
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.64767 (QuantReg: 15.02463) QuantErr: 15.02463 batch_time=0.58358
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.76157 (QuantReg: 14.81841) QuantErr: 14.81841 batch_time=0.58121
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.66805 (QuantReg: 14.84058) QuantErr: 14.84058 batch_time=0.58755
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.80563 (QuantReg: 14.77112) QuantErr: 14.77112 batch_time=0.61058
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.84067 (QuantReg: 15.07886) QuantErr: 15.07886 batch_time=0.58906
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.66570 (QuantReg: 14.81942) QuantErr: 14.81942 batch_time=0.59604
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.92369 (QuantReg: 14.88154) QuantErr: 14.88154 batch_time=0.58386
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.66675 (QuantReg: 14.95312) QuantErr: 14.95312 batch_time=0.63625
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.97882 (QuantReg: 14.89911) QuantErr: 14.89911 batch_time=0.60183
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.81198 (QuantReg: 14.90165) QuantErr: 14.90165 batch_time=0.58343
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.76888 (QuantReg: 14.70356) QuantErr: 14.70356 batch_time=0.64455
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.77463 (QuantReg: 14.82151) QuantErr: 14.82151 batch_time=0.62045
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.67488 (QuantReg: 14.84206) QuantErr: 14.84206 batch_time=0.58581
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.50100 (QuantReg: 14.82099) QuantErr: 14.82099 batch_time=0.59945
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.88559 (QuantReg: 14.78494) QuantErr: 14.78494 batch_time=0.58162
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.28966 (QuantReg: 15.11722) QuantErr: 15.11722 batch_time=2.45468
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.65822 (QuantReg: 14.96110) QuantErr: 14.96110 batch_time=0.58773
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.89026 (QuantReg: 15.04074) QuantErr: 15.04074 batch_time=0.60610
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.61706 (QuantReg: 15.10446) QuantErr: 15.10446 batch_time=0.58712
Train Epoch: 11 codebook_update_time=3.95836
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch11.pth ...
Done in 6.076s
removing stale ckpt [epoch 10] [took 0.01s]
epoch : 11
loss : 1.704303783416748
quant_reg : 14.916944732666016
quant_err : 14.916944732666016
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_miech_test/t2v_metrics/R1: 21.0
MSRVTT_miech_test/t2v_metrics/R5: 48.7
MSRVTT_miech_test/t2v_metrics/R10: 63.8
MSRVTT_miech_test/t2v_metrics/R50: 88.3
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.083
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.25838154735452
MSRVTT_miech_test/v2t_metrics/R1: 20.7
MSRVTT_miech_test/v2t_metrics/R5: 50.9
MSRVTT_miech_test/v2t_metrics/R10: 64.1
MSRVTT_miech_test/v2t_metrics/R50: 88.9
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.551
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.72383968376797
mnt_best : 40.88323011338325
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.79476 (QuantReg: 14.71056) QuantErr: 14.71056 batch_time=27.00870
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.46514 (QuantReg: 14.90872) QuantErr: 14.90872 batch_time=0.60286
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.68585 (QuantReg: 14.83876) QuantErr: 14.83876 batch_time=0.58072
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.65234 (QuantReg: 14.75306) QuantErr: 14.75306 batch_time=0.60509
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.44074 (QuantReg: 14.94470) QuantErr: 14.94470 batch_time=0.59517
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.66884 (QuantReg: 14.87776) QuantErr: 14.87776 batch_time=0.59031
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.75889 (QuantReg: 14.79162) QuantErr: 14.79162 batch_time=0.59069
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.35869 (QuantReg: 14.99246) QuantErr: 14.99246 batch_time=0.61964
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.65757 (QuantReg: 14.65726) QuantErr: 14.65726 batch_time=0.64138
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.44226 (QuantReg: 15.01692) QuantErr: 15.01692 batch_time=0.59628
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.61004 (QuantReg: 14.98089) QuantErr: 14.98089 batch_time=0.58466
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.75814 (QuantReg: 14.83425) QuantErr: 14.83425 batch_time=0.59056
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.53582 (QuantReg: 14.85676) QuantErr: 14.85676 batch_time=0.59677
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.59292 (QuantReg: 14.79603) QuantErr: 14.79603 batch_time=0.68309
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.63811 (QuantReg: 14.85679) QuantErr: 14.85679 batch_time=0.61421
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.66887 (QuantReg: 14.76665) QuantErr: 14.76665 batch_time=3.15442
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.84464 (QuantReg: 14.95652) QuantErr: 14.95652 batch_time=0.59467
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.73867 (QuantReg: 14.99094) QuantErr: 14.99094 batch_time=0.61648
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.41002 (QuantReg: 15.15757) QuantErr: 15.15757 batch_time=0.60738
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.73414 (QuantReg: 15.07189) QuantErr: 15.07189 batch_time=0.59163
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.75580 (QuantReg: 14.73665) QuantErr: 14.73665 batch_time=0.84304
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.83515 (QuantReg: 15.05284) QuantErr: 15.05284 batch_time=0.59462
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.68741 (QuantReg: 15.01076) QuantErr: 15.01076 batch_time=0.61346
Train Epoch: 12 codebook_update_time=4.28407
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch12.pth ...
Done in 5.050s
removing stale ckpt [epoch 11] [took 0.52s]
epoch : 12
loss : 1.6181392946243287
quant_reg : 14.923102195739746
quant_err : 14.923102195739746
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_miech_test/t2v_metrics/R1: 20.1
MSRVTT_miech_test/t2v_metrics/R5: 51.0
MSRVTT_miech_test/t2v_metrics/R10: 63.2
MSRVTT_miech_test/t2v_metrics/R50: 88.7
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.832
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.16315031132725
MSRVTT_miech_test/v2t_metrics/R1: 21.1
MSRVTT_miech_test/v2t_metrics/R5: 50.6
MSRVTT_miech_test/v2t_metrics/R10: 65.7
MSRVTT_miech_test/v2t_metrics/R50: 88.7
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.131
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.241341212864285
mnt_best : 40.88323011338325
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.48088 (QuantReg: 15.03139) QuantErr: 15.03139 batch_time=25.58642
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.58797 (QuantReg: 14.86885) QuantErr: 14.86885 batch_time=0.61018
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.45709 (QuantReg: 14.98426) QuantErr: 14.98426 batch_time=0.69796
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.85536 (QuantReg: 14.98138) QuantErr: 14.98138 batch_time=0.58745
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.63545 (QuantReg: 15.07418) QuantErr: 15.07418 batch_time=0.59345
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.82663 (QuantReg: 14.62825) QuantErr: 14.62825 batch_time=0.58544
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.41688 (QuantReg: 14.97281) QuantErr: 14.97281 batch_time=0.58651
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.56578 (QuantReg: 15.02171) QuantErr: 15.02171 batch_time=0.58031
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.22943 (QuantReg: 14.94725) QuantErr: 14.94725 batch_time=0.58364
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.64404 (QuantReg: 14.72751) QuantErr: 14.72751 batch_time=0.57979
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.64707 (QuantReg: 14.96174) QuantErr: 14.96174 batch_time=0.58144
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.54173 (QuantReg: 15.05859) QuantErr: 15.05859 batch_time=0.60923
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.91032 (QuantReg: 14.66211) QuantErr: 14.66211 batch_time=0.59043
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.92543 (QuantReg: 14.95377) QuantErr: 14.95377 batch_time=0.60216
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.56291 (QuantReg: 15.14025) QuantErr: 15.14025 batch_time=0.59141
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.63190 (QuantReg: 14.80783) QuantErr: 14.80783 batch_time=0.59174
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.47127 (QuantReg: 15.12434) QuantErr: 15.12434 batch_time=0.58026
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.71726 (QuantReg: 14.81800) QuantErr: 14.81800 batch_time=0.72495
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.22657 (QuantReg: 15.04511) QuantErr: 15.04511 batch_time=0.61819
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.43647 (QuantReg: 15.25904) QuantErr: 15.25904 batch_time=0.58326
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.28538 (QuantReg: 15.04717) QuantErr: 15.04717 batch_time=0.58473
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.21931 (QuantReg: 14.82852) QuantErr: 14.82852 batch_time=0.58149
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.00363 (QuantReg: 14.93215) QuantErr: 14.93215 batch_time=0.60854
Train Epoch: 13 codebook_update_time=3.82519
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch13.pth ...
Done in 4.935s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 1.5625547106266022
quant_reg : 14.95269144821167
quant_err : 14.95269144821167
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_miech_test/t2v_metrics/R1: 21.1
MSRVTT_miech_test/t2v_metrics/R5: 50.1
MSRVTT_miech_test/t2v_metrics/R10: 63.1
MSRVTT_miech_test/t2v_metrics/R50: 88.5
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.792
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.55550809721795
MSRVTT_miech_test/v2t_metrics/R1: 20.6
MSRVTT_miech_test/v2t_metrics/R5: 53.3
MSRVTT_miech_test/v2t_metrics/R10: 66.2
MSRVTT_miech_test/v2t_metrics/R50: 89.3
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.103
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.733435718076976
mnt_best : 40.88323011338325
not_improved_count: 3
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.79954 (QuantReg: 14.67043) QuantErr: 14.67043 batch_time=32.13250
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.52694 (QuantReg: 15.01705) QuantErr: 15.01705 batch_time=0.58783
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.31142 (QuantReg: 14.72173) QuantErr: 14.72173 batch_time=0.58622
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.51369 (QuantReg: 15.13167) QuantErr: 15.13167 batch_time=0.58959
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.63620 (QuantReg: 14.90555) QuantErr: 14.90555 batch_time=0.64999
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.31699 (QuantReg: 15.01900) QuantErr: 15.01900 batch_time=0.59354
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.53027 (QuantReg: 14.84800) QuantErr: 14.84800 batch_time=0.64887
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.99117 (QuantReg: 14.76680) QuantErr: 14.76680 batch_time=0.58415
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.60011 (QuantReg: 15.00377) QuantErr: 15.00377 batch_time=0.61258
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.50149 (QuantReg: 14.86047) QuantErr: 14.86047 batch_time=0.60188
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.56418 (QuantReg: 14.98243) QuantErr: 14.98243 batch_time=0.60115
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.32227 (QuantReg: 14.85061) QuantErr: 14.85061 batch_time=0.59044
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.59012 (QuantReg: 14.97316) QuantErr: 14.97316 batch_time=0.60846
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.48864 (QuantReg: 14.76500) QuantErr: 14.76500 batch_time=0.59458
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.28862 (QuantReg: 14.88475) QuantErr: 14.88475 batch_time=1.13809
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.26367 (QuantReg: 15.08341) QuantErr: 15.08341 batch_time=0.59368
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.59434 (QuantReg: 14.82238) QuantErr: 14.82238 batch_time=0.60244
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.52125 (QuantReg: 15.08193) QuantErr: 15.08193 batch_time=0.58944
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.16813 (QuantReg: 15.12085) QuantErr: 15.12085 batch_time=0.63927
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.83290 (QuantReg: 15.12806) QuantErr: 15.12806 batch_time=0.58658
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.18235 (QuantReg: 15.05613) QuantErr: 15.05613 batch_time=0.58824
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.42552 (QuantReg: 14.91464) QuantErr: 14.91464 batch_time=0.59873
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.49971 (QuantReg: 15.06602) QuantErr: 15.06602 batch_time=0.58875
Train Epoch: 14 codebook_update_time=4.83323
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch14.pth ...
Done in 4.104s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 1.4972545604705811
quant_reg : 14.973214366912842
quant_err : 14.973214366912842
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_miech_test/t2v_metrics/R1: 21.1
MSRVTT_miech_test/t2v_metrics/R5: 50.7
MSRVTT_miech_test/t2v_metrics/R10: 63.8
MSRVTT_miech_test/t2v_metrics/R50: 89.4
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.697
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.86677478540526
MSRVTT_miech_test/v2t_metrics/R1: 21.2
MSRVTT_miech_test/v2t_metrics/R5: 52.5
MSRVTT_miech_test/v2t_metrics/R10: 65.7
MSRVTT_miech_test/v2t_metrics/R50: 89.3
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 25.359
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.81706147826304
mnt_best : 40.88323011338325
not_improved_count: 4
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.60924 (QuantReg: 14.97306) QuantErr: 14.97306 batch_time=30.96895
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.58828 (QuantReg: 14.77601) QuantErr: 14.77601 batch_time=5.21170
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.68382 (QuantReg: 14.82917) QuantErr: 14.82917 batch_time=0.58661
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.84489 (QuantReg: 14.76738) QuantErr: 14.76738 batch_time=0.58702
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.69298 (QuantReg: 15.00822) QuantErr: 15.00822 batch_time=0.60047
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.01173 (QuantReg: 15.09413) QuantErr: 15.09413 batch_time=0.61769
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.59814 (QuantReg: 14.78447) QuantErr: 14.78447 batch_time=0.58361
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.07489 (QuantReg: 15.21733) QuantErr: 15.21733 batch_time=0.61451
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.87068 (QuantReg: 15.04266) QuantErr: 15.04266 batch_time=0.61830
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.33621 (QuantReg: 14.90651) QuantErr: 14.90651 batch_time=0.59232
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.29571 (QuantReg: 14.96764) QuantErr: 14.96764 batch_time=0.59455
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.41800 (QuantReg: 14.82168) QuantErr: 14.82168 batch_time=0.59166
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.85977 (QuantReg: 14.99657) QuantErr: 14.99657 batch_time=0.59103
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.44411 (QuantReg: 15.04309) QuantErr: 15.04309 batch_time=0.58185
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.59672 (QuantReg: 14.99314) QuantErr: 14.99314 batch_time=0.60163
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.40903 (QuantReg: 14.92078) QuantErr: 14.92078 batch_time=0.64410
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.33778 (QuantReg: 14.86923) QuantErr: 14.86923 batch_time=0.62315
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.73862 (QuantReg: 14.99345) QuantErr: 14.99345 batch_time=0.64760
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.12227 (QuantReg: 14.95768) QuantErr: 14.95768 batch_time=0.59861
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.22871 (QuantReg: 15.29039) QuantErr: 15.29039 batch_time=0.62548
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.32941 (QuantReg: 15.32749) QuantErr: 15.32749 batch_time=0.59705
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.51597 (QuantReg: 15.17968) QuantErr: 15.17968 batch_time=1.43712
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.56702 (QuantReg: 15.34806) QuantErr: 15.34806 batch_time=0.58505
Train Epoch: 15 codebook_update_time=3.83117
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch15.pth ...
Done in 6.494s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 1.46888685464859
quant_reg : 15.018942111968995
quant_err : 15.018942111968995
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_miech_test/t2v_metrics/R1: 20.8
MSRVTT_miech_test/t2v_metrics/R5: 50.8
MSRVTT_miech_test/t2v_metrics/R10: 62.4
MSRVTT_miech_test/t2v_metrics/R50: 89.0
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.471
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.398993537649375
MSRVTT_miech_test/v2t_metrics/R1: 21.3
MSRVTT_miech_test/v2t_metrics/R5: 50.9
MSRVTT_miech_test/v2t_metrics/R10: 65.1
MSRVTT_miech_test/v2t_metrics/R50: 89.8
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.264
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.32626200714906
mnt_best : 40.88323011338325
not_improved_count: 5
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.40531 (QuantReg: 14.78522) QuantErr: 14.78522 batch_time=41.80970
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.63326 (QuantReg: 14.68407) QuantErr: 14.68407 batch_time=0.60053
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.05641 (QuantReg: 14.75184) QuantErr: 14.75184 batch_time=0.60448
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.73260 (QuantReg: 14.79456) QuantErr: 14.79456 batch_time=0.60439
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.48031 (QuantReg: 15.04746) QuantErr: 15.04746 batch_time=0.60849
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.63315 (QuantReg: 15.07280) QuantErr: 15.07280 batch_time=0.60511
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.75473 (QuantReg: 15.02059) QuantErr: 15.02059 batch_time=0.60622
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.36014 (QuantReg: 15.22608) QuantErr: 15.22608 batch_time=0.60406
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.44671 (QuantReg: 14.86797) QuantErr: 14.86797 batch_time=0.59539
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.14321 (QuantReg: 15.03613) QuantErr: 15.03613 batch_time=0.63637
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.24077 (QuantReg: 15.03502) QuantErr: 15.03502 batch_time=0.64234
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.27267 (QuantReg: 14.86297) QuantErr: 14.86297 batch_time=0.59723
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.45755 (QuantReg: 14.95639) QuantErr: 14.95639 batch_time=0.60079
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.50251 (QuantReg: 14.96703) QuantErr: 14.96703 batch_time=0.61092
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.47747 (QuantReg: 14.99046) QuantErr: 14.99046 batch_time=0.60244
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.42477 (QuantReg: 15.08455) QuantErr: 15.08455 batch_time=0.59770
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.42356 (QuantReg: 15.17256) QuantErr: 15.17256 batch_time=0.66108
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.56667 (QuantReg: 14.81076) QuantErr: 14.81076 batch_time=0.63307
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.37171 (QuantReg: 15.15080) QuantErr: 15.15080 batch_time=0.58131
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.37154 (QuantReg: 15.20924) QuantErr: 15.20924 batch_time=0.59547
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.43586 (QuantReg: 15.11053) QuantErr: 15.11053 batch_time=0.60510
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.21236 (QuantReg: 15.24982) QuantErr: 15.24982 batch_time=0.65925
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.48421 (QuantReg: 15.04006) QuantErr: 15.04006 batch_time=0.59817
Train Epoch: 16 codebook_update_time=4.16849
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch16.pth ...
Done in 6.055s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch16.pth ...
Done in 11.042s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 1.394637505054474
quant_reg : 15.0484598197937
quant_err : 15.0484598197937
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_miech_test/t2v_metrics/R1: 22.4
MSRVTT_miech_test/t2v_metrics/R5: 51.1
MSRVTT_miech_test/t2v_metrics/R10: 63.9
MSRVTT_miech_test/t2v_metrics/R50: 88.9
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.476
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.82056785606564
MSRVTT_miech_test/v2t_metrics/R1: 21.2
MSRVTT_miech_test/v2t_metrics/R5: 51.0
MSRVTT_miech_test/v2t_metrics/R10: 66.7
MSRVTT_miech_test/v2t_metrics/R50: 89.8
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 25.642
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.62401381028862
mnt_best : 41.82056785606564
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.28667 (QuantReg: 14.82531) QuantErr: 14.82531 batch_time=32.82306
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.22542 (QuantReg: 14.95837) QuantErr: 14.95837 batch_time=0.58546
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.39614 (QuantReg: 15.01343) QuantErr: 15.01343 batch_time=0.59996
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.53850 (QuantReg: 14.86019) QuantErr: 14.86019 batch_time=0.58591
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.47920 (QuantReg: 15.16425) QuantErr: 15.16425 batch_time=0.59263
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.36363 (QuantReg: 15.05816) QuantErr: 15.05816 batch_time=0.60880
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.65688 (QuantReg: 14.80828) QuantErr: 14.80828 batch_time=0.61428
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.32026 (QuantReg: 15.21958) QuantErr: 15.21958 batch_time=0.61383
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.36813 (QuantReg: 15.09281) QuantErr: 15.09281 batch_time=0.60265
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.08919 (QuantReg: 15.36279) QuantErr: 15.36279 batch_time=0.68596
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.08816 (QuantReg: 15.14781) QuantErr: 15.14781 batch_time=0.60010
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.22922 (QuantReg: 15.01267) QuantErr: 15.01267 batch_time=0.61059
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.68522 (QuantReg: 15.04473) QuantErr: 15.04473 batch_time=0.65389
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.45692 (QuantReg: 15.04794) QuantErr: 15.04794 batch_time=0.58770
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.58719 (QuantReg: 14.94646) QuantErr: 14.94646 batch_time=0.60849
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.48048 (QuantReg: 14.92911) QuantErr: 14.92911 batch_time=1.35850
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.06769 (QuantReg: 15.01617) QuantErr: 15.01617 batch_time=0.59131
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.30291 (QuantReg: 15.11493) QuantErr: 15.11493 batch_time=0.59127
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.55929 (QuantReg: 15.10015) QuantErr: 15.10015 batch_time=0.61441
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.43607 (QuantReg: 15.07042) QuantErr: 15.07042 batch_time=3.36455
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.08301 (QuantReg: 15.16074) QuantErr: 15.16074 batch_time=0.58967
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.37211 (QuantReg: 15.14136) QuantErr: 15.14136 batch_time=0.59725
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.39420 (QuantReg: 15.10663) QuantErr: 15.10663 batch_time=0.60051
Train Epoch: 17 codebook_update_time=4.41207
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch17.pth ...
Done in 21.336s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch17.pth ...
Done in 27.129s
removing stale ckpt [epoch 16] [took 0.42s]
epoch : 17
loss : 1.364722718000412
quant_reg : 15.048282806396484
quant_err : 15.048282806396484
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_miech_test/t2v_metrics/R1: 22.8
MSRVTT_miech_test/t2v_metrics/R5: 50.0
MSRVTT_miech_test/t2v_metrics/R10: 64.6
MSRVTT_miech_test/t2v_metrics/R50: 88.7
MSRVTT_miech_test/t2v_metrics/MedR: 5.5
MSRVTT_miech_test/t2v_metrics/MeanR: 31.267
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.915931611852464
MSRVTT_miech_test/v2t_metrics/R1: 21.8
MSRVTT_miech_test/v2t_metrics/R5: 53.0
MSRVTT_miech_test/v2t_metrics/R10: 66.8
MSRVTT_miech_test/v2t_metrics/R50: 89.0
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.108
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.57646580078803
mnt_best : 41.915931611852464
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.38784 (QuantReg: 14.82677) QuantErr: 14.82677 batch_time=28.01805
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.42922 (QuantReg: 14.88500) QuantErr: 14.88500 batch_time=0.58955
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.56567 (QuantReg: 14.79388) QuantErr: 14.79388 batch_time=0.59269
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.72490 (QuantReg: 14.93766) QuantErr: 14.93766 batch_time=0.59553
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.30130 (QuantReg: 15.21107) QuantErr: 15.21107 batch_time=0.63852
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.33237 (QuantReg: 14.97310) QuantErr: 14.97310 batch_time=0.60568
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.07273 (QuantReg: 15.15819) QuantErr: 15.15819 batch_time=0.59213
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.37005 (QuantReg: 15.10697) QuantErr: 15.10697 batch_time=0.59966
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.42650 (QuantReg: 15.26754) QuantErr: 15.26754 batch_time=0.61452
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.55784 (QuantReg: 15.02503) QuantErr: 15.02503 batch_time=0.62583
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.10377 (QuantReg: 15.21280) QuantErr: 15.21280 batch_time=0.59972
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.22227 (QuantReg: 15.32084) QuantErr: 15.32084 batch_time=0.59050
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.45484 (QuantReg: 15.02665) QuantErr: 15.02665 batch_time=0.60602
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.28094 (QuantReg: 15.03056) QuantErr: 15.03056 batch_time=0.58210
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.27203 (QuantReg: 15.14125) QuantErr: 15.14125 batch_time=0.59244
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.37398 (QuantReg: 15.18405) QuantErr: 15.18405 batch_time=0.59672
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.66609 (QuantReg: 15.15868) QuantErr: 15.15868 batch_time=0.61739
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.30932 (QuantReg: 14.95886) QuantErr: 14.95886 batch_time=0.58448
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.39633 (QuantReg: 15.12797) QuantErr: 15.12797 batch_time=0.61232
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.37317 (QuantReg: 15.18517) QuantErr: 15.18517 batch_time=0.59273
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.33318 (QuantReg: 15.03626) QuantErr: 15.03626 batch_time=0.68786
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.50244 (QuantReg: 15.27082) QuantErr: 15.27082 batch_time=0.63709
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.31869 (QuantReg: 15.15734) QuantErr: 15.15734 batch_time=0.59494
Train Epoch: 18 codebook_update_time=4.68018
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch18.pth ...
Done in 4.415s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch18.pth ...
Done in 8.469s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 1.3287771589756012
quant_reg : 15.104014556884765
quant_err : 15.104014556884765
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_miech_test/t2v_metrics/R1: 22.9
MSRVTT_miech_test/t2v_metrics/R5: 51.0
MSRVTT_miech_test/t2v_metrics/R10: 64.1
MSRVTT_miech_test/t2v_metrics/R50: 89.0
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.504
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.14582530359166
MSRVTT_miech_test/v2t_metrics/R1: 21.5
MSRVTT_miech_test/v2t_metrics/R5: 52.0
MSRVTT_miech_test/v2t_metrics/R10: 67.6
MSRVTT_miech_test/v2t_metrics/R50: 88.9
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.842
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.27946662365837
mnt_best : 42.14582530359166
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.53359 (QuantReg: 14.90680) QuantErr: 14.90680 batch_time=26.96639
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.11390 (QuantReg: 15.18784) QuantErr: 15.18784 batch_time=0.68044
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.39982 (QuantReg: 14.74543) QuantErr: 14.74543 batch_time=3.63673
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.42961 (QuantReg: 14.91082) QuantErr: 14.91082 batch_time=0.58481
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.38294 (QuantReg: 15.12658) QuantErr: 15.12658 batch_time=0.58944
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.46887 (QuantReg: 15.03526) QuantErr: 15.03526 batch_time=0.59914
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.43805 (QuantReg: 14.89114) QuantErr: 14.89114 batch_time=0.59534
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.15378 (QuantReg: 15.16932) QuantErr: 15.16932 batch_time=0.61526
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.54851 (QuantReg: 15.19043) QuantErr: 15.19043 batch_time=0.59404
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.10812 (QuantReg: 14.83402) QuantErr: 14.83402 batch_time=0.59746
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.22908 (QuantReg: 15.17006) QuantErr: 15.17006 batch_time=0.61380
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.27127 (QuantReg: 15.03478) QuantErr: 15.03478 batch_time=0.59847
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.18570 (QuantReg: 15.24300) QuantErr: 15.24300 batch_time=0.64033
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.26790 (QuantReg: 15.20965) QuantErr: 15.20965 batch_time=0.59996
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.56426 (QuantReg: 15.07755) QuantErr: 15.07755 batch_time=0.59144
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.00017 (QuantReg: 15.38506) QuantErr: 15.38506 batch_time=0.62799
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.23373 (QuantReg: 15.19951) QuantErr: 15.19951 batch_time=0.60168
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.25022 (QuantReg: 15.35436) QuantErr: 15.35436 batch_time=0.62504
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.47030 (QuantReg: 15.15724) QuantErr: 15.15724 batch_time=0.61800
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.25111 (QuantReg: 15.03663) QuantErr: 15.03663 batch_time=0.59507
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.29712 (QuantReg: 15.08626) QuantErr: 15.08626 batch_time=0.61693
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.17981 (QuantReg: 15.20477) QuantErr: 15.20477 batch_time=0.59947
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.45606 (QuantReg: 14.79532) QuantErr: 14.79532 batch_time=0.60145
Train Epoch: 19 codebook_update_time=4.38755
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M64/checkpoint-epoch19.pth ...
Done in 4.238s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 1.2955259954929352
quant_reg : 15.140022521972655
quant_err : 15.140022521972655
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_miech_test/t2v_metrics/R1: 22.8
MSRVTT_miech_test/t2v_metrics/R5: 50.7
MSRVTT_miech_test/t2v_metrics/R10: 63.5
MSRVTT_miech_test/t2v_metrics/R50: 88.6
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.757
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.870245810601475
MSRVTT_miech_test/v2t_metrics/R1: 20.9