Enhance sensitivity analysis with covariates R^2 measurements

[DarioSimonato]

Current sensitivity analysis shows the values of RV and RVa that are not interpretable (i.e. it is not easy to say if they represent a strong inference or a weak one) without comparing them to other variable quantities.

I'd like to have a tool that includes comparisons as the one suggested by Cinelli et al. in "Making sense of sensitivity: extending omitted variable bias" that would allow to reason about the magnitude of these values as he did in this video at minute 20. What is not clear to me is how to calculate the values in the second row (i.e. those to which compare the RV and RVa) in the doubleML scenario, and I think it should be a valuable and easy addition that may enhance greatly the utility of doubleML sensitivity analysis tool.

[SvenKlaassen]

Thanks for the suggestion. We are currently discussing to add a feature of this type.
Basically this would require to copy the complete DoubleML object and refit it again (without some set of confounders).
Afterwards one would have to access the sensitivity_elements attributes of both objects to obtain "long" and "short" forms of the representations.
I will keep this discussion updated when we have some implementations (or notebooks).

[SvenKlaassen]

We are currently working on a benchmarking method on the branch
https://github.com/DoubleML/doubleml-for-py/tree/s-sensitivity-benchmark-2

This includes the sensitivity_benchmark() method to compare the confounding factors to known confounders

doubleml-for-py/doubleml/double_ml.py

Line 1872 in 4e43587

def sensitivity_benchmark(self, benchmarking_set):

You can find a first documentation and a corresponding example on docs repository:
https://github.com/DoubleML/doubleml-docs/blob/s-sensitivity-benchmark/doc/examples/py_double_ml_sensitivity.ipynb
https://github.com/DoubleML/doubleml-docs/blob/s-sensitivity-benchmark/doc/guide/sensitivity.rst#benchmarking

Feedback is always welcome!
I will update the discussion if the feature is on the main branch or released.

[SvenKlaassen]

The changes are now on the main branch:
https://github.com/DoubleML/doubleml-for-py

You can find a development version of the documentation at:
https://docs.doubleml.org/dev/guide/sensitivity.html#benchmarking

[SvenKlaassen]

The benchmarking features are included in the lastest release (0.7.0) and the documentation is updated.

[DarioSimonato]

Thanks a lot, great work!
Just one point: sensitivity_benchmark does not include the possibility to correct for multiple hypothesis testing as sensitivity_analysis via setup of level. It would be nice to have that setting in here as well without having to call sensitivity_benchmark and passing the newly discovered RV, RVa and theta to sensitivity_analysis.

[SvenKlaassen]

Thanks for the feedback!
Generally, the level should be independent from the other input parameters cf_d, cf_y and rho. You should be able to use any level you like for your subsequent analysis.
Further, the values RV, RVa and theta can not be passed as input into sensitivity_analysis since they are determined by the confounding strength parameters cf_d, cf_y and rho.
Our idea of the workflow was the following:

• Fit your DoubleML model
• Determine some known confounders, where you assume similar or larger confounding strength as for your unkown confounding
• Call sensitivity_benchmark to determine values for cf_d, cf_y and rho
• Call sensitivity_analysis using input values cf_d, cf_y and rho based on the previous benchmarking (you can choose any level which seems required)

[DarioSimonato]

Yes, I wrote the answer fast and I changed cf_d, cf_y and rhowith RV, RVa and theta. That's exactly what I've done, thanks for the info!

In my opinion the same workflow would be easier by just calling sensitivity_benchmark(cfnd_list, level).

[SvenKlaassen]

Ah, I get your point.
My reasoning, to keep sensitivity_benchmark and sensitivity_analysis seperated, was to make it very clear that the benchmark is seperate from the benchmark procedure and requires further assumptions (the similar unkown confounding).