Require PyArrow >=16

[crusaderky]

• Closes Pandas 3 + PyArrow 14 breaks DataFrame shuffle: RuntimeError: P2P failed during barrier phase #9183
• Twin of Require PyArrow >=16 dask#12258

PyArrow 14 and 15 are incompatible with Pandas 3 when it comes to distributed shuffle.
I must confess I was lazy, did not investigate the actual issue, and instead simply bumped up the minimum version. This also gave the opportunity to clean up a lot of special cases in the codebase.

An alternative option was to pin pandas<3 in the 3.11 environment and add a runtime check for the specific combination of the two dependencies. I discarded it for the sake of simplicity.

This change is respectful of SPEC 0.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    31 files  ±0      31 suites  ±0   11h 22m 10s ⏱️ - 19m 55s
 4 113 tests ±0   4 005 ✅ +257    104 💤 ±0   4 ❌  - 254 
59 636 runs  ±0  57 157 ✅ +509  2 465 💤 ±0  14 ❌  - 505 

For more details on these failures, see this check.

Results for commit eac6217. ± Comparison against base commit 7649d2f.

[crusaderky]

Failure in test_handle_null_partitions_2 is caused by Pandas 3 (#9184) but out of scope for this PR.
Other test failures are unrelated.

[jrbourbeau]

Thanks @crusaderky. LGTM overall. Looks like #9184 is the only consistent CI failure (exists independently of the changes here)