Add multi-threading for pack/unpack#1029
Add multi-threading for pack/unpack#1029cr3ativ3cod3r wants to merge 4 commits intokitops-ml:mainfrom
Conversation
Signed-off-by: Keerthan KK <tkthulasimandiram@gmail.com>
…rresponding goroutine Signed-off-by: Keerthan KK <tkthulasimandiram@gmail.com>
Signed-off-by: Keerthan KK <tkthulasimandiram@gmail.com>
…ches cant be called concurrently Signed-off-by: Keerthan KK <tkthulasimandiram@gmail.com>
gorkem
left a comment
gorkem
left a comment
There was a problem hiding this comment.
Some critical issues to address and also missing
- Race detector tests (go test -race)
- Concurrent layer ordering tests
- Performance benchmarks
|
|
||
| func (pm *ignorePaths) Matches(path, layerPath string) (bool, error) { | ||
| pm.mu.Lock() | ||
| defer pm.mu.Unlock() |
There was a problem hiding this comment.
saveContentLayer -> packLayerToTar -> writeLayerToTar
writeLayerToTar calls ignore.Matches` for each file With the mutex, all file traversal becomes serialized. Parallel implementation may be SLOWER than sequential due to lock contention.
| if err != nil { | ||
| return err | ||
| } | ||
| kitfile.Model.Parts[index].LayerInfo = layerInfo |
There was a problem hiding this comment.
This is introducing concurrent writes to shared Kitfile fields without synchronization, causing potential data races and corruption. The same pattern is repeated on other layers too.
| } | ||
|
|
||
| if err = eg.Wait(); err != nil { | ||
| close(results) |
There was a problem hiding this comment.
This closes the results channel and returns. However, some goroutines might still be running and will panic when trying to send to the closed channel.
|
Closing this PR as it is stale. Feel free to reopen if issues mentioned in review are addressed. |
3 participants
Description
This PR introduces parallel processing of pack and unpack operations. Previously the layers were packed and unpacked sequentially.
Linked issues
closes #254
AI-Assisted Code