Order of the features in autoregressive transformations

[CaioDaumann]

Hello!

I wanted to ask about how does Zuko handle the orders of the variables in auto-regressive flows. Is it the same way as described in the MAF paper?

MAF uses the default order for the first autoregressive layer (i.e. the layer that directly models the data) and reverses the order for each successive layer [...]

Or it does something different? Also, if it follows what is in the MAF paper, is there a way to check the order of the variables in each layer?

Also, if we have three inputs $(x_1,x_2,x_3)$ and then we have $(z_1,z_2,z_3)$. Suppose now that I want to invert this from $(z_1,z_2,z_3)$ to $(x_1,x_2,x_3)$ but the MADE NNs takes as input $(x_1,x_2,x_3)$ (MAF).

In my understanding the MADE NN learns the best "conditions". For example, $p(x_1,x_2,x_3) = p(x_2|x_1,x_3) p(x_1|x_3) p(x_3)$. But when we invert it, we must obtain first $x_1$ and for this we evaluate the MADE NN, but since the $p(x_1|x_3)$ has this dependency in $x_3$, what does it do?

Cheers,
Caio

[francois-rozet]

Hello @CaioDaumann 👋

The order of the features/variables in autoregressive transformations is not learned but fixed at initialization. In MaskedAutoregressiveTransform there is an argument (order) to specify that order. In MAF, by default, the order is $(x_1, x_2, \dots, x_n)$ in even transformations and $(x_n, \dots, x_2, x_1)$ in odd transformations.

In my understanding the MADE NN learns the best "conditions". For example, $p(x_1,x_2,x_3) = p(x_2|x_1,x_3) p(x_1|x_3) p(x_3)$. But when we invert it, we must obtain first $x_1$ and for this we evaluate the MADE NN, but since the $p(x_1|x_3)$ has this dependency in $x_3$, what does it do?

In your example, if the order would be $(x_3, x_1, x_2)$. At inversion, we would first compute $x_3$ as a function of $z_3$, then $x_1$ as a function of $(z_1, x_3)$, and finally $x_2$ as a function of $(z_2, x_1, x_3)$. I invite you to take a look at #16 for more details.

Also, if it follows what is in the MAF paper, is there a way to check the order of the variables in each layer?

The order of the first transformation can be retrieved with flow.transform.transforms[0].order if flow is a MAF instance.