How to load backbone pretrained weight of hybrid vision transformer?

[haoweiz23]

Thanks for your work. I find the pretrained weight you converted from jax to pytorch has some incompatible key in backbone.
You deifined vit_small_resnet50d_s3_224 here. Howerver, the resnet50d has so many incompatible key that I cant load pretrained checkpoint of jx_vit_base_resnet50_224_in21k-6f7c7740.pth.

Further, I also try resnetv2 your provided in timm.models.resnetv2, the layer name is close to pretrained weight, after I change their key name to be consistent, it notices that their weights has different shape.

@register_model
def resnetv2_50x1_vit_hybrid(pretrained=False, **kwargs):
    return resnetv2._create_resnetv2(
        'resnetv2_50x1_bitm', pretrained=pretrained,
        layers=[3, 4, 9, 3], width_factor=1, stem_type='fixed', **kwargs)

@register_model
def vit_base_resnet50d_s3_224(pretrained=False, **kwargs):
    # pretrained_backbone = kwargs.get('pretrained_backbone', True)  # default to True for now, for testing
    # backbone = resnet50d(pretrained=False, features_only=True, out_indices=[3], output_stride=16)
    backbone = resnetv2_50x1_vit_hybrid(pretrained=False, features_only=True, out_indices=[3])
    model = VisionTransformer(
        img_size=224, embed_dim=768, depth=12, num_heads=12, mlp_ratio=4, hybrid_backbone=backbone, **kwargs)
    model.default_cfg = _cfg()
    model.default_cfg['input_size'] = (3, 224, 224)
    return model

`size mismatch for patch_embed.backbone.stages_0.blocks.1.norm1.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([256]).
        size mismatch for patch_embed.backbone.stages_0.blocks.1.norm3.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]).
        size mismatch for patch_embed.backbone.stages_0.blocks.1.norm3.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]).
        size mismatch for patch_embed.backbone.stages_0.blocks.2.norm1.weight: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([256]).`

Can your provide some advice or example for how to load the hybrid model's pretrained weight, especially backbone?