ADR suggestion ModelCollection for List-Based Model Management

[rozyczko]

ModelCollection for List-Based Model Management

Status

Proposed

Context

The EasyScience corelib provides base classes for building scientific models, including ModelBase which extends from NewBase. However, there was no dedicated collection class that combines the list-like functionality of Python's built-in containers with the EasyScience features such as:

• Interface propagation to contained items
• Graph-based dependency tracking
• Serialization support
• Parameter management across collections

Users who needed to manage collections of model objects had to either:

1. Use Python's built-in list types, losing EasyScience features
2. Use the existing CollectionBase which is based on BasedBase (older architecture)
3. Create custom collection implementations for each use case

This gap in the architecture made it difficult to create clean, maintainable code when working with multiple related model objects that need to share interfaces and participate in the EasyScience ecosystem.

Decision

We have decided to introduce a new ModelCollection class that:

1. Extends ModelBase: Inherits all EasyScience features from ModelBase
2. Implements MutableSequence: Provides full list-like functionality through Python's collections.abc.MutableSequence interface
3. Supports Interface Propagation: Automatically propagates interface bindings to all items in the collection
4. Maintains Graph Edges: Properly manages graph relationships through the global object map
5. Enables Type Safety: Uses Python's typing system with generics (TypeVar) for better IDE support
6. Provides Multiple Access Methods: Supports indexing by integer, slice, name, or unique_name

Key Design Decisions

1. Inheritance from ModelBase and MutableSequence

class ModelCollection(ModelBase, MutableSequence[T]):

This dual inheritance provides:

• ModelBase features: serialization, parameter management, graph tracking
• MutableSequence interface: standard Python list operations (append, insert, pop, etc.)

2. Interface Propagation

The collection automatically propagates its interface to all contained items:

@interface.setter
def interface(self, new_interface: InterfaceType) -> None:
    self._interface = new_interface
    for item in self._data:
        if hasattr(item, 'interface'):
            setattr(item, 'interface', new_interface)

This ensures that when a collection's interface changes, all items are updated automatically.

3. Graph Edge Management

All collection operations (add, remove, replace) properly maintain graph edges:

def _add_item(self, item: Any) -> None:
    if not isinstance(item, NewBase):
        raise TypeError(f'Items must be NewBase objects, got {type(item)}')
    if item in self._data:
        return  # Skip duplicates
    self._data.append(item)
    self._global_object.map.add_edge(self, item)
    self._global_object.map.reset_type(item, 'created_internal')

4. Multiple Access Patterns

Support for accessing items by:

• Integer index: collection[0]
• Slice: collection[0:2]
• Name: collection['item_name']
• Unique name: collection[unique_id]

5. Type Safety with Generics

Using TypeVar and @overload decorators for precise type hints:

T = TypeVar('T', bound=NewBase)

@overload
def __getitem__(self, idx: int) -> T: ...
@overload
def __getitem__(self, idx: slice) -> 'ModelCollection[T]': ...
@overload
def __getitem__(self, idx: str) -> T: ...

Consequences

Positive

1. Unified API: Users get familiar list operations while maintaining EasyScience features
2. Type Safety: Better IDE support and static analysis through comprehensive type hints
3. Automatic Interface Management: Interface changes propagate automatically to all items
4. Clean Architecture: Separation of concerns with _add_item and _remove_item helpers
5. Backward Compatible: Doesn't break existing code, adds new functionality
6. Well-Tested: Comprehensive test suite (102 tests covering all functionality)
7. Flexible Access: Multiple ways to access items (by index, name, or unique_name)

Negative

1. Additional Complexity: Users need to understand when to use ModelCollection vs CollectionBase
2. Memory Overhead: Graph edge tracking adds some memory overhead
3. Learning Curve: New users need to understand the interface propagation mechanism
4. Type Checking Limitations: Some dynamic operations may not be fully type-checked

Neutral

1. Migration Path: Existing code using plain lists can be migrated gradually
2. Documentation Needs: Requires documentation and examples for effective use
3. CollectionBase Relationship: Need to clarify relationship with existing CollectionBase

Implementation Details

File Changes

Based on PR #180, the implementation includes:

1. New File: src/easyscience/base_classes/model_collection.py (278 lines)

   • Core ModelCollection class implementation
   • Complete MutableSequence interface
   • Interface propagation logic
   • Serialization support

2. Modified: src/easyscience/base_classes/__init__.py

   • Export ModelCollection from base_classes module

3. Modified: src/easyscience/base_classes/collection_base.py

   • Update to accept NewBase objects (not just BasedBase)

4. New File: tests/unit_tests/base_classes/test_model_collection.py (953 lines)

   • 102 comprehensive unit tests
   • Tests for all MutableSequence operations
   • Interface propagation tests
   • Graph edge management tests

Key Methods

• __init__: Initialize collection with optional items and interface
• __getitem__: Support int, slice, and string (name) indexing
• __setitem__: Replace items while maintaining graph edges
• __delitem__: Remove items and clean up graph edges
• insert: Add items at specific positions
• interface property: Get/set interface with propagation
• get_all_variables: Collect variables from all items
• sort: Sort collection by custom mapping function
• _convert_to_dict: Serialize collection to dictionary

Alternatives Considered

1. Extending CollectionBase:

   • Rejected because CollectionBase is based on BasedBase (older architecture)
   • Would not provide the clean MutableSequence interface

2. Composition over Inheritance:

   • Could wrap a list instead of inheriting from MutableSequence
   • Rejected because it would require manually implementing all list methods

3. Using Python's Built-in list:

   • Simplest approach but loses all EasyScience features
   • No interface propagation or graph tracking

4. Generic Container Class:

   • Could create a more generic container for any object type
   • Rejected to maintain type safety and EasyScience feature integration

References

• Pull Request New collection base #180
• Related Issue: New collection base functionality
• Python MutableSequence: https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableSequence
• Type Hints PEP 484: https://www.python.org/dev/peps/pep-0484/

Notes

This ADR documents the design decisions made in PR #180 (New collection base). The implementation is currently in draft/review stage. Once merged, this ADR should be updated to "Accepted" status.

Future Considerations

1. Consider adding more specialized collection types (e.g., ordered, sorted)
2. Evaluate performance with large collections (1000+ items)
3. Consider adding collection-level operations (filter, map, reduce)
4. Potential for async/lazy loading of items for very large datasets
5. Integration with numpy arrays or pandas DataFrames for numerical operations

[damskii9992]

Can you please explain to me WHY exactly we need a ModelCollection class?
Why not just have a list as an attribute to a ModelBase class like what @henrikjacobsenfys did in EasyDynamics for ComponentCollection?
I'm open for the idea but we need a valid reason to implement this (not just that it existed before).

[henrikjacobsenfys]

It would save me some code if I didn't have to write the add_component and similar methods. I'm also unsure if my code interacts correctly with the global object, but I don't know enough about it to be sure.
Finally, there's the indexing issue - it's very neat to be able to index by name as well as int as mentioned today and yesterday, but I'm unsure what the best solution is

[rozyczko]

Can you please explain to me WHY exactly we need a ModelCollection class?

• We avoid polluting every ModelBase subclass with list management logic, like add_sample, remove_sample etc.
• When you change the interface bound to a collection, you want ALL items in that collection to automatically receive the new interface. With a plain list attribute, you'd need to manually iterate and set the interface every time.
• Graph dependency tracking: With a plain list, every append/remove/insert requires manual graph management
• A ModelCollection is a ModelBase object itself so it can have its own parameters, can be serialized/deserialized as a unit, can be nested in other collections or models potentially.
• ModelCollection can be accessed in many wyas (as Henrik noted) - by index, unique_name and properly sliced#

[damskii9992]

• We avoid polluting every ModelBase subclass with list management logic, like add_sample, remove_sample etc.

Not all ModelBase subclasses will be list/require list logic, and many of those that do, will require unique list-logic anyways. Like checking that the added component is a valid component for the list.

• When you change the interface bound to a collection, you want ALL items in that collection to automatically receive the new interface. With a plain list attribute, you'd need to manually iterate and set the interface every time.

This is mute as we are moving to get rid of the interface attribute.

• Graph dependency tracking: With a plain list, every append/remove/insert requires manual graph management

This is also mute as we aim to get rid of the graph completely.

• A ModelCollection is a ModelBase object itself so it can have its own parameters, can be serialized/deserialized as a unit, can be nested in other collections or models potentially.

This is also true for a ModelBase with a simple list attribute, as @henrikjacobsenfys has shown in EasyDynamics.

• ModelCollection can be accessed in many wyas (as Henrik noted) - by index, unique_name and properly sliced#

This is so far the only valid reasoning I've found, having a list that can also be accessed by its components unique_names is definitely something I can see a use for. But such a class would be much simpler than what we have here. And we might have many places where we overwrite the setters/getters anyways.

It would save me some code if I didn't have to write the add_component and similar methods. I'm also unsure if my code interacts correctly with the global object, but I don't know enough about it to be sure.
Finally, there's the indexing issue - it's very neat to be able to index by name as well as int as mentioned today and yesterday, but I'm unsure what the best solution is

You'd have to overwrite the add_component anyways because you only allow ModelComponents in your list. And you might overwrite the unique_name getter anyways too, to solve one of the issues you've mentioned (you can make the getter be a regex search, so that a user writing "Lorentzian" would get any objects whose unique_names start with "Lorentzian", that way they don't have to know that the true unique_name is "Lorentzian_0").

The way I see it, it might make sense to have a class, lets call it EasyCollection, which only allows members inheriting from NewBase and which is a list with the additional unique_name getter and inspection. The might part here depends on how much we would have to overwrite these methods in classes inheriting from it.

[seventil]

The issue I found trying implementing a colleciton like that that was derived from built-ins is that we can't use list and dict as parents at the same time, at least half the functionality still has to be explicitely implemented. Then, if we want to use narrower typeguards, than just NewBase typevar, all the list/dict methods have to be implemented anyway with the only difference being either calling super().method() or self._items.method(). There's almost no code repetition avoided if we inherit from a built-in.

I think ModelCollection makes sense to change it to a more general collection class without interface/graph maintenance, but with a potential regex name lookup. Or maybe regex name lookup should be part of a NewBase?

[rozyczko]

regex name lookup for querying with the unique_name? As in

  model_backgrounds = model_collection['background_*']

Yes, this could potentially be useful.

[damskii9992]

Looking at my own requirements in EasyImaging, I believe a first implementation would be a much simpler implementation. I propose the EasyList:

*** EasyList ***
Will inherit from MutableSequence. Will be a small class, essentially just extending the regular list with 2 new features:

** Member type-protection **
The EasyList will take an optional parameter of allowed_types which by default is None. If provided, any added or inserted elements into the list will be checked against these types. If None, they will still be checked against NewBase to ensure they have a unique_name which is critical for the 2nd feature:

** unique_name lookup **
Beside the regular slicing and indexing, the getitem will also accept a string corresponding to a unique_name which will be queried to find the corresponding member, if it exists, in the list.

Obviously, the EasyList should also be serializable.

Currently working on an example implementation.

for discussion:
Should it also inherit from NewBase? That would essentialy give it a unique_name and a display_name. Does a list need that?

[henrikjacobsenfys]

Sorry for the wall of text.

Inheriting from NewBase means it can easily be used in other EasyScience things such as fitting, right? I think not inherting from NewBase could come with some annoying unexpected downsides. It maye in fact be better to inherit from ModelBase and implement get_all_variables to get all variables in the list, since at least in my case, I'd use EasyList to contain a list of ModelComponents for fitting.

TL;DR of the rest: I will need a lookup using a name or something similar that is not globally unique, but unique to each instance of the list.

This is because I will be working with multiple copies of the same model (list of components). It'll be a (Easy)List of (Easy)List. I do need to keep them as separate lists, because users will want to do something like this with their parameters (made up example, you get the gist), and because I will need to make global fits where all the Parameters must be different objects.
(here, the first index is Q, which could also be looked up in a smarter way, but that's less of a problem).

my_lists[0]['my_gaussian'].area.fixed=False
my_lists[1]['my_lorentzian'].area.value=2
my_lists[2]['my_gaussian'].width.fixed=True`

Right now, users have to figure out/remember the order in which components was added (i.e. was it first the Gaussian, then the Lorentzian, or the other way around?), which is brittle and annoying.
If they don't provide a unique_name, they'll have to first figure out what the unique_name of each component is, which is as brittle and annoying as indexing with numbers: rerunning their code will generate new Parameter's, which will have new unique_names.

You have previously suggested that the unique_name in this case could be generated from the template and be something like my_gaussian_0, my_gaussian_1 etc, and the lookup could do a partial match. This may work, but there's another issue with working directly with unique names:

If users provide the unique_name themselves, i.e. gaussian = Gaussian(unique_name='my_gaussian') rerunning parts of their code in a Notebook withour restarting it will not work.
Rerunning parts of a Notebook is extremely common behavior, especially with longer Notebooks that may load in a lot of data. One example is when fitting, where you almost always have to try a handful of start guesses to get it right. Users will not want to restart their Notebook and reload their data, just because they realised the start values of the fit Parameters need to change a bit. However, they will then either get errors or will have to make up a new unique_name for every iteration of manually adjusting their Parameters.
(To add insult to injury, what's the first unique_name you try when your code tells you my_gaussian isn't allowed because it already exists? Perhaps my_gaussian_1, which also already exists.)

[damskii9992]

A few important points to your comment:

1. The issue with rerunning notebook cells with unique_names is an inherent issue with unique_names. If this is a big issue, then we need to completely remove unique_names. There are no "fixes". But this is a question for another discussion/thread.
2. Your use-case is so hyper-specific that your implementation should probably be in your own library. This is a suggestion for a much more generic and broadly applicable class that can then possibly be inherited from or used directly in your class.
3. Inheriting from NewBase doesn't mean it can be used in fitting, it just means it can be saved/loaded and queried in the global_object. Inheriting from ModelBase means it can be used in fitting. But we might simply inherit from this class when making a SampleModelCollection class later which is then meant for fitting.

[henrikjacobsenfys]

I partly see your point, but wanting to be able to re-run fits without restarting the notebook is not specific; every single user of every single software we write will want it.

[damskii9992]

Of course you should be able to re-run fits. But running your fit shouldn't create new Parameters and thus shouldn't have any unique_name issues. But again, this is a different topic for a different discussion.