Experiment grid on weights and ndkl's ratios on imdb

[Rounique]

*In the initial round of experiments on female_bias, an error was identified, necessitating the repetition of all experiments.

The results table encompasses various experiments employing cross-entropy, unigram, unigram_b, inverse_unigram_b, uniform, and female_bias in which diverse weights for the female_bias objective function were explored. Specifically, experiments were conducted to calculate the utility and fairness. NDKL; the fairness metric has been calculated with both males as the minority and females as the minority at various ratios. The desired distributions for female_bias, incrementing in ratios of 0.1 provide a comprehensive analysis of the impact of varying factors on the outcomes.

The results indicate a noteworthy pattern: when male experts dominate, NDKL values are superior compared to the objective functions other than female_bias. Nevertheless, as the proportion of female experts increases, NDKL exhibits improvement, even when men constitute the majority. Conversely, when females are in the majority, NDKL consistently performs significantly better across all experiments conducted with the female_bias objective function.
This implies that if the intended distribution of experts leans towards having more females, deemed as a fairer representation, utilizing objective functions other than female_bias may not effectively recommend more female experts. In contrast, employing the female_bias objective function explicitly encourages the model to prioritize and recommend more female experts, thereby aligning with the desired distribution and fostering increased female representation in the recommendation.

The experimentation involved training the model and evaluating the outcomes with varied weights for the female bias. The results unveiled a distinct trend: when men are in the majority, weights below and above 2 adversely impact NDKL. In contrast, when women are in the majority, escalating the weight correlates with an improvement in NDKL. Importantly, it is worth noting that as NDKL improves with increasing weights, there is either a minimal drop or no significant decrease in utility. This observation shows that enhancing female representation through increased weights in female bias does not compromise overall utility or may even marginally decrease it.

Results