Modern packages use optional dependecies / extras for specific functionality. I assume this is very well possible for the sagemaker sdk.
A similar request was posted in discussions: #5441
I couldn't find any actual "issue" and hope this gets more visibility.
The main and most urgent topic is the dependency of pytorch which requires 354/620mb size in a clean venv. I would assume that most use cases don't need pytorch at all.