fix: fix runtime type introspection by moving public API types out of TYPE_CHECKING

[junjzhang]

Problem

Classes like S3Path, GCSPath etc. have type annotations in their __init__
that reference types only imported under if TYPE_CHECKING:. This breaks runtime
type introspection tools that call typing.get_type_hints(), such as:

• tyro (CLI generation)
• pydantic (validation)
• dataclasses with type validation

here is a quick reproduce:

from typing import get_type_hints

from upath import UPath

s3_path = UPath("s3://bucket/path")
S3Path = type(s3_path)

hints = get_type_hints(S3Path.__init__)
# will raise error

Root Cause

When from __future__ import annotations is used, all annotations become strings.
get_type_hints() evaluates these strings in the module's namespace. Types under
TYPE_CHECKING don't exist at runtime, causing NameError.

Solution

Move types used in public API signatures out of TYPE_CHECKING block and import
them unconditionally. These are all lightweight imports with no performance impact:

• Literal - typing primitive
• FSSpecChainParser - already imported elsewhere
• Unpack, Self - typing utilities
• Storage options - TypedDict definitions

[junjzhang]

@ap-- Hi, could you have a look on this one?

[ap--]

Hi @junjzhang,

Thank you for your contribution! Can you provide some more context?

• Where did you run into the issue?
• To solve your issue is runtime typing only needed for __init__ or for the entire class interface?

[junjzhang]

Hi @junjzhang,

Thank you for your contribution! Can you provide some more context?

• Where did you run into the issue?
• To solve your issue is runtime typing only needed for __init__ or for the entire class interface?

Hi,

1. I met this issue while use a cli tool called tyro, below is a simple use case, but I guess this issue is common for any usage of get_type_hints as suggested above.

import tyro
from upath import UPath
from pydantic import BaseModel


class Config(BaseModel):
    path: UPath = UPath("s3://bucket/output/")

config = tyro.cli(Config)

2. Yes, here is a simple patch you could try

from typing import Literal

import upath.implementations.cloud as cloud_mod
from upath._chain import FSSpecChainParser
from typing_extensions import Unpack
from upath.types.storage_options import (
    HfStorageOptions,
    S3StorageOptions,
    GCSStorageOptions,
    AzureStorageOptions,
)

cloud_mod.Literal = Literal
cloud_mod.FSSpecChainParser = FSSpecChainParser
cloud_mod.Unpack = Unpack
cloud_mod.S3StorageOptions = S3StorageOptions
cloud_mod.GCSStorageOptions = GCSStorageOptions
cloud_mod.AzureStorageOptions = AzureStorageOptions
cloud_mod.HfStorageOptions = HfStorageOptions
cloud_mod.Literal = Literal

import tyro
from upath import UPath
from pydantic import BaseModel


class Config(BaseModel):
    path: UPath = UPath("s3://bucket/output/")

config = tyro.cli(Config)

# OR you could try to use get_type_hints like in the description

[ap--]

Thank you for the additional context 🙏

Given my quick tests, I don't think your solution solves your problem. Can you show me an example how you would provide a path to your tyro cli?

❯ python tyro_example.py --help
usage: tyro_example.py [-h] [--path.kwargs {fixed}]

╭─ options ───────────────────────────────────────────────╮
│ -h, --help              show this help message and exit │
╰─────────────────────────────────────────────────────────╯
╭─ path options ──────────────────────────────────────────╮
│ Default: s3://bucket/output                             │
│ ──────────────────────────────────────                  │
│ --path.kwargs {fixed}   (fixed to: {})                  │
╰─────────────────────────────────────────────────────────╯

❯ python tyro_example.py gcs://bucket/key
╭─ Parsing error ───────────────────────────────╮
│ Unrecognized arguments: gcs://bucket/key      │
│ ───────────────────────────────────────────── │
│ For full helptext, run tyro_example.py --help │
╰───────────────────────────────────────────────╯

Also tyro seems to have two bugs/missing-features? It does not seem to understand the Unpack[TypedDict] annotations for keyword arguments. And it seems to ignore the path: UPath type annotation and actually instantiates the detected subclass S3Path of the default value.

Given these limitations in tyro, IMO you'd be much better served with writing a custom constructor: https://brentyi.github.io/tyro/examples/custom_constructors/

Would you be interested in giving that a shot to see if we can then find a better way to make this more convenient with universal-pathlib?

[junjzhang]

Thank you for the additional context 🙏

Given my quick tests, I don't think your solution solves your problem. Can you show me an example how you would provide a path to your tyro cli?

❯ python tyro_example.py --help
usage: tyro_example.py [-h] [--path.kwargs {fixed}]

╭─ options ───────────────────────────────────────────────╮
│ -h, --help              show this help message and exit │
╰─────────────────────────────────────────────────────────╯
╭─ path options ──────────────────────────────────────────╮
│ Default: s3://bucket/output                             │
│ ──────────────────────────────────────                  │
│ --path.kwargs {fixed}   (fixed to: {})                  │
╰─────────────────────────────────────────────────────────╯

❯ python tyro_example.py gcs://bucket/key
╭─ Parsing error ───────────────────────────────╮
│ Unrecognized arguments: gcs://bucket/key      │
│ ───────────────────────────────────────────── │
│ For full helptext, run tyro_example.py --help │
╰───────────────────────────────────────────────╯

Also tyro seems to have two bugs/missing-features? It does not seem to understand the Unpack[TypedDict] annotations for keyword arguments. And it seems to ignore the path: UPath type annotation and actually instantiates the detected subclass S3Path of the default value.

Given these limitations in tyro, IMO you'd be much better served with writing a custom constructor: https://brentyi.github.io/tyro/examples/custom_constructors/

Would you be interested in giving that a shot to see if we can then find a better way to make this more convenient with universal-pathlib?

Oh, that's just a minimal reproduce, my daily use case is :

# ******* PATCH *******, remove will cause error

from typing import Literal

import upath.implementations.cloud as cloud_mod
from upath._chain import FSSpecChainParser
from typing_extensions import Unpack
from upath.types.storage_options import (
    HfStorageOptions,
    S3StorageOptions,
    GCSStorageOptions,
    AzureStorageOptions,
)

cloud_mod.Literal = Literal
cloud_mod.FSSpecChainParser = FSSpecChainParser
cloud_mod.Unpack = Unpack
cloud_mod.S3StorageOptions = S3StorageOptions
cloud_mod.GCSStorageOptions = GCSStorageOptions
cloud_mod.AzureStorageOptions = AzureStorageOptions
cloud_mod.HfStorageOptions = HfStorageOptions
cloud_mod.Literal = Literal
# ******* PATCH END *******
from typing import Annotated

import tyro
from upath import UPath
from pydantic import BaseModel, PlainValidator, PlainSerializer


def serialize_path(value: UPath | str) -> str:
    """Serialize UPath/str to plain str to avoid dict-like JSON output."""
    return str(value)


def validate_path_str(value) -> UPath:
    """Validate and convert to UPath."""
    if isinstance(value, str):
        return UPath(value)
    elif isinstance(value, UPath):
        return value
    elif hasattr(value, "__fspath__"):  # Support PathLike objects
        return UPath(value)
    else:
        # Try to convert to string first
        try:
            return UPath(str(value))
        except Exception as e:
            raise ValueError(f"Expected str or path-like object, got {type(value)}") from e

PathStr = Annotated[
    str | UPath,
    PlainValidator(validate_path_str),
    PlainSerializer(serialize_path, return_type=str),
]


class Config(BaseModel):
    path: PathStr = UPath("s3://bucket/output/")

config = tyro.cli(Config)

And the output should be

python dev/test_upath.py --help
usage: test_upath.py [-h] [{path:str,path:u-path}]

╭─ options ───────────────────────────────────────────────────────────────────────╮
│ -h, --help        show this help message and exit                               │
╰─────────────────────────────────────────────────────────────────────────────────╯
╭─ optional subcommands ──────────────────────────────────────────────────────────╮
│ (default: path:str)                                                             │
│ ─────────────────────────────────────────────────────────────────────────────── │
│ [{path:str,path:u-path}]                                                        │
│     path:str                                                                    │
│     path:u-path   Base class for pathlike paths backed by an fsspec filesystem. │
╰─────────────────────────────────────────────────────────────────────────────────╯




python dev/test_upath.py path:str gcs://bucket/key # Will be fine

[junjzhang]

Thank you for the additional context 🙏

Given my quick tests, I don't think your solution solves your problem. Can you show me an example how you would provide a path to your tyro cli?

❯ python tyro_example.py --help
usage: tyro_example.py [-h] [--path.kwargs {fixed}]

╭─ options ───────────────────────────────────────────────╮
│ -h, --help              show this help message and exit │
╰─────────────────────────────────────────────────────────╯
╭─ path options ──────────────────────────────────────────╮
│ Default: s3://bucket/output                             │
│ ──────────────────────────────────────                  │
│ --path.kwargs {fixed}   (fixed to: {})                  │
╰─────────────────────────────────────────────────────────╯

❯ python tyro_example.py gcs://bucket/key
╭─ Parsing error ───────────────────────────────╮
│ Unrecognized arguments: gcs://bucket/key      │
│ ───────────────────────────────────────────── │
│ For full helptext, run tyro_example.py --help │
╰───────────────────────────────────────────────╯

Also tyro seems to have two bugs/missing-features? It does not seem to understand the Unpack[TypedDict] annotations for keyword arguments. And it seems to ignore the path: UPath type annotation and actually instantiates the detected subclass S3Path of the default value.

Given these limitations in tyro, IMO you'd be much better served with writing a custom constructor: https://brentyi.github.io/tyro/examples/custom_constructors/

Would you be interested in giving that a shot to see if we can then find a better way to make this more convenient with universal-pathlib?

But I tried use custom_constructor, I guess it would be a better walk-around

# # ******* PATCH *******, remove will cause error

# from typing import Literal

# import upath.implementations.cloud as cloud_mod
# from upath._chain import FSSpecChainParser
# from typing_extensions import Unpack
# from upath.types.storage_options import (
#     HfStorageOptions,
#     S3StorageOptions,
#     GCSStorageOptions,
#     AzureStorageOptions,
# )

# cloud_mod.Literal = Literal
# cloud_mod.FSSpecChainParser = FSSpecChainParser
# cloud_mod.Unpack = Unpack
# cloud_mod.S3StorageOptions = S3StorageOptions
# cloud_mod.GCSStorageOptions = GCSStorageOptions
# cloud_mod.AzureStorageOptions = AzureStorageOptions
# cloud_mod.HfStorageOptions = HfStorageOptions
# cloud_mod.Literal = Literal
# # ******* PATCH END *******



from typing import Annotated

import tyro
from upath import UPath
from pydantic import BaseModel


def construct_UPath(
    path_str: str
) -> UPath:
    """A custom constructor for UPath."""
    return UPath(path_str)

class Config(BaseModel):
    path: Annotated[UPath, tyro.conf.arg(constructor=construct_UPath)] = UPath("s3://bucket/output/")


config = tyro.cli(Config)

But is this case with built-in method get_type_hint raise error is expected? Or these type are not supposed to be exposed?

from typing import get_type_hints

from upath.implementations.cloud import S3Path

hints = get_type_hints(S3Path.__init__)
# will raise error

[ap--]

Awesome 🎉 Well with this example it seems you got your use case working correctly:

# cli_tyro.py
from typing import Annotated
import tyro
from upath import UPath
from pydantic import BaseModel

def construct_UPath(path_str: str) -> UPath:
    """A custom constructor for UPath."""
    return UPath(path_str)

class Config(BaseModel):
    path: Annotated[UPath, tyro.conf.arg(constructor=construct_UPath)] = UPath("s3://bucket/output/")

config = tyro.cli(Config)
print(repr(config.path))

universal_pathlib on  cut-release-0.3.7 [$!?] via 🐍 v3.13.7 (universal-pathlib) via 🅒 base 
❯ python cli_tyro.py --path.path-str 'gcs://bucket/key'
GCSPath('bucket/key', protocol='gcs')

universal_pathlib on  cut-release-0.3.7 [$!?] via 🐍 v3.13.7 (universal-pathlib) via 🅒 base 
❯ python cli_tyro.py --path.path-str 's3://bucket/key'
S3Path('bucket/key', protocol='s3')

universal_pathlib on  cut-release-0.3.7 [$!?] via 🐍 v3.13.7 (universal-pathlib) via 🅒 base 
❯ python cli_tyro.py --path.path-str 'http://bucket.com/path'
HTTPPath('http://bucket.com/path', protocol='http')

The problem with your fix is, that tyro seems to do something strange with path: UPath = S3Path(...) We would need to figure out why that is happening. Maybe the pydantic v2 serialization of the base UPath class needs tweaking. I would recommend reporting this case to tyro, and maybe we can find the correct solution. Because even if we enable inspection of the __init__ signature at runtime, your example seems broken.

[junjzhang]

Awesome 🎉 Well with this example it seems you got your use case working correctly:

# cli_tyro.py
from typing import Annotated
import tyro
from upath import UPath
from pydantic import BaseModel

def construct_UPath(path_str: str) -> UPath:
    """A custom constructor for UPath."""
    return UPath(path_str)

class Config(BaseModel):
    path: Annotated[UPath, tyro.conf.arg(constructor=construct_UPath)] = UPath("s3://bucket/output/")

config = tyro.cli(Config)
print(repr(config.path))

universal_pathlib on  cut-release-0.3.7 [$!?] via 🐍 v3.13.7 (universal-pathlib) via 🅒 base 
❯ python cli_tyro.py --path.path-str 'gcs://bucket/key'
GCSPath('bucket/key', protocol='gcs')

universal_pathlib on  cut-release-0.3.7 [$!?] via 🐍 v3.13.7 (universal-pathlib) via 🅒 base 
❯ python cli_tyro.py --path.path-str 's3://bucket/key'
S3Path('bucket/key', protocol='s3')

universal_pathlib on  cut-release-0.3.7 [$!?] via 🐍 v3.13.7 (universal-pathlib) via 🅒 base 
❯ python cli_tyro.py --path.path-str 'http://bucket.com/path'
HTTPPath('http://bucket.com/path', protocol='http')

The problem with your fix is, that tyro seems to do something strange with path: UPath = S3Path(...) We would need to figure out why that is happening. Maybe the pydantic v2 serialization of the base UPath class needs tweaking. I would recommend reporting this case to tyro, and maybe we can find the correct solution. Because even if we enable inspection of the __init__ signature at runtime, your example seems broken.

Great!, let me change it into a draft pr and put a issue on tyro