[SYTEMDS-3902] Transfer scipy compressed matrices to java runtime#2379
[SYTEMDS-3902] Transfer scipy compressed matrices to java runtime#2379e-strauss wants to merge 1 commit intoapache:mainfrom
Conversation
|
@Baunsgaard @mboehm7 https://github.com/apache/systemds/blob/8de93a1f996348c260d48c1c29340d5a88005e6f/src/main/python/systemds/context/systemds_context.py#L772C4-L772C20 the naming "from_numpy" is misleading since we support now scipy compressed matrices as well We could also use something like "from_pydata" which merges the two separate methods from_numpy and from_pandas to single method, internally we could route to the corresponding handler based on the input instance type. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2379 +/- ##
============================================
+ Coverage 71.51% 71.54% +0.02%
- Complexity 47441 47464 +23
============================================
Files 1539 1539
Lines 182605 182631 +26
Branches 35916 35919 +3
============================================
+ Hits 130585 130655 +70
+ Misses 42028 41972 -56
- Partials 9992 10004 +12 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This sounds good to me, unifying the API would be fine.
|
|
sure, I agree |
bc56a18 to
cce93be
Compare
Baunsgaard
left a comment
There was a problem hiding this comment.
There are a few edge cases not covered.
cce93be to
97a2fbe
Compare
This commit implements optimized data transfer for Scipy sparse matrices from Python to the Java runtime. Key changes include the addition of `convertSciPyCSRToMB` and `convertSciPyCOOToMB` in the Java utility layer to directly handle compressed sparse row and coordinate formats. On the Python side, the `SystemDSContext` now supports a `sparse_data_transfer` flag and a new `from_py` method to unify data ingestion. These updates allow sparse data to be transferred without being converted to dense arrays, improving efficiency. Additionally, several data conversion methods were refactored for better maintenance.
97a2fbe to
3184b9a
Compare
This commit implements direct transfer support for Scipy sparse matrices (CSR and COO formats) between Python and the Java runtime.
MatrixBlockrepresentations.SystemDSContextto support asparse_data_transferflag and extendedfrom_numpyto acceptscipy.sparsematrices.Benchmark results:
