ENH: Performance optimization for pd.merge introducing fillna= argument

[anatolix]

Is your feature request related to a problem?

Very often we merge very data sets(100Gb) with 'left'/'right'/'outer' which introduces NAs.
Basically, if we have integer field, we have two options:

1. Allow pandas autoconvert it to float which could be nan (default behaviour)
2. Use Int64 type, which is actually quite slow to use it in large datasets spending ages in _PyEval_EvalFrameDefault

Describe the solution you'd like

The obvious option is to use fill value, like -1 for NaN, it currently could be done after 1st or 2nd step, with fillna and conversion to integer type back. It will be quite simple to introduce new parameter inmerge fillna={ "column1":-1, "column2":-1} which do the same task without addition computations.

API breaking implications

By default fillna should be None, which will keep current behaviour.

p.s. the same thing is reasonable for pd.concat

[mzeitlin11]

Thanks for looking into this @anatolix! Do you have the profile results or a nice example to run for the Int64 slowness you mention? While the enhancement you propose might be a suitable workaround, seems like the ideal scenario would be to make Int64 as fast as int64 (since the whole point of Int64 is to represent integer data which might include nans)

[mroeschke]

Thanks for the issue, but it appears this hasn't gotten traction in a while so closing