PERF: Performance Improvement on DataFrame.to_csv() when index=False

[KevsterAmp]

• closes PERF: Significant Performance Difference in DataFrame.to_csv() with and without Index Reset #59312 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

I added an alternative ndarray with the same length on _get_values_for_csv's output, used on write_csv_rows crude testing

pandas/pandas/_libs/writers.pyx

Line 42 in 642d244

Py_ssize_t i, j = 0, k = len(data_index), N = 100, ncols = len(cols)

Tests

Using the same code as the referenced issue:

import pandas as pd
import pyarrow as pa
import pyarrow.csv as csv
import time

NUM_ROWS = 10000000
NUM_COLS = 20

# Example Multi-Index DataFrame
df = pd.DataFrame(
    {
        f"col_{col_idx}": range(col_idx * NUM_ROWS, (col_idx + 1) * NUM_ROWS)
        for col_idx in range(NUM_COLS)
    }
)
df = df.set_index(["col_0", "col_1"], drop=False)

# Timing Operation A
start_time = time.time()
df.to_csv("file_A.csv", index=False)
end_time = time.time()
print(f"Operation A time: {end_time - start_time} seconds")

# Timing Operation B
start_time = time.time()
df_reset = df.reset_index(drop=True)
df_reset.to_csv("file_B.csv", index=False)
end_time = time.time()
print(f"Operation B time: {end_time - start_time} seconds")

Output before performance improvement

Operation A time: 869.2354643344879 seconds
Operation B time: 42.1906418800354 seconds

Output after performance improvement

Operation A time: 51.408071756362915 seconds
Operation B time: 45.78637385368347 seconds

Operation B is used for time comparison when resetting index, the change improves the performance on Operation A

[mroeschke]

Can you use np.empty instead?

[mroeschke]

For you benchmark could you show before and after timings

[KevsterAmp]

Output before performance improvement

Operation A time: 869.2354643344879 seconds
Operation B time: 42.1906418800354 seconds

Output (after performance improvement)

Operation A time: 51.408071756362915 seconds
Operation B time: 45.78637385368347 seconds

Operation B is used for time comparison when resetting index, the change improves the performance on Operation A

Added output times to the description as well

[mroeschke]

Thanks @KevsterAmp