Create DDS Transport Protocol #1174
Conversation
Greptile OverviewGreptile SummaryThis PR adds DDS (Data Distribution Service) transport support using Eclipse CycloneDDS, providing a high-performance pub/sub transport option for network communication. Key Changes:
Implementation Details:
Confidence Score: 5/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant DDSTransport
participant DDS
participant DDSService
participant DomainParticipant
participant DataWriter
participant DataReader
participant Listener
User->>DDSTransport: __init__(topic, type, **kwargs)
DDSTransport->>DDS: __init__(**kwargs)
DDSTransport->>DDSTransport: Create RLock
User->>DDSTransport: publish(message)
DDSTransport->>DDSTransport: Check _started (with lock)
alt Not started
DDSTransport->>DDS: start()
DDS->>DDSService: start()
DDSService->>DDSService: Check _participants dict
alt Domain not in _participants
DDSService->>DomainParticipant: Create participant(domain_id)
DDSService->>DDSService: Store in _participants[domain_id]
end
end
DDSTransport->>DDS: publish(topic, message)
DDS->>DDS: _get_writer(topic)
alt Writer doesn't exist
DDS->>DDSTopic: Create DDSTopic
DDS->>DataWriter: Create DataWriter(participant, topic, qos)
DDS->>DDS: Store in _writers[topic]
end
DDS->>DataWriter: write(message)
User->>DDSTransport: subscribe(callback)
DDSTransport->>DDSTransport: Check _started (with lock)
alt Not started
DDSTransport->>DDS: start()
end
DDSTransport->>DDS: subscribe(topic, callback)
DDS->>DDS: _get_listener(topic)
alt Listener doesn't exist
DDS->>DDSTopic: Create DDSTopic
DDS->>Listener: Create _DDSMessageListener(topic)
DDS->>DataReader: Create DataReader(participant, topic, qos, listener)
DDS->>DDS: Store reader and listener
end
DDS->>Listener: add_callback(callback)
DDS-->>DDSTransport: Return unsubscribe lambda
DDSTransport-->>User: Return unsubscribe function
DataReader->>Listener: on_data_available(reader)
Listener->>DataReader: take()
loop For each sample
loop For each callback
Listener->>User: callback(sample, topic)
end
end
User->>DDSTransport: stop()
DDSTransport->>DDS: stop()
DDS->>DDS: Clear _readers, _listeners, _writers
DDS->>DDSService: stop()
Note over DDSService,DomainParticipant: DomainParticipant persists<br/>for potential reuse
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
@greptile |
|
When installing I get this: I think you need to add to the docs how to build it. I've tried |
This makes sense, because I use Nix to run this. You need to build build CycloneDDS from source otherwise. Where should I add this functionality? |
Since But, should it be in the root dependencies? That is, I don't think everyone needs it. Currently, running |
ugh.. are we sure we can't just install cyclonedds on ubuntu? or ros-ubuntu? this will be annoying because |
|
Got it, moved to a extra group so now it is installed via |
|
Now the build is failing because core imports actually depend on the cyclonedds. I'm not sure what the solution is here. Options are:
Separately, these are the instructions for how I got it running on Ubuntu (technically I'm running Linux Mint, but it's based on Ubuntu 24.04). You probably want to add this to the docs. ### Optional: DDS Transport Support
The `dds` extra provides DDS (Data Distribution Service) transport support via Eclipse Cyclone DDS. This requires installing system libraries before the Python package can be built.
**Ubuntu/Debian:**
```bash
# Install the CycloneDDS development library
sudo apt install cyclonedds-dev
# Create a compatibility directory structure
# (required because Ubuntu's multiarch layout doesn't match the expected CMake layout)
sudo mkdir -p /opt/cyclonedds/{lib,bin,include}
sudo ln -sf /usr/lib/x86_64-linux-gnu/libddsc.so* /opt/cyclonedds/lib/
sudo ln -sf /usr/lib/x86_64-linux-gnu/libcycloneddsidl.so* /opt/cyclonedds/lib/
sudo ln -sf /usr/bin/idlc /opt/cyclonedds/bin/
sudo ln -sf /usr/bin/ddsperf /opt/cyclonedds/bin/
sudo ln -sf /usr/include/dds /opt/cyclonedds/include/
# Install with the dds extra
CYCLONEDDS_HOME=/opt/cyclonedds uv pip install -e '.[dds]'
```
To install all extras including DDS:
```bash
CYCLONEDDS_HOME=/opt/cyclonedds uv sync --all-extras
``` |
|
@Kaweees What is |
|
My results for reference (very similar): 
|
Kaweees
force-pushed
the
miguel/dds_transport
branch
from
fd7fd0c to
0246eae
Compare
I've been trying to figure out how we would deploy locomotion polcies on the Go2 and G1 in preparation of AGI Bot. That was not supposed to be commit, just rebased it out |
Perhaps in all cases where DDS is imported, we just try except as a hacky way to keep it optional. |
👍 |
Yeah definitely can't have DDS in core deps so good removal there. Current failing tests should be fixed by just catching all of the cycloneDDS imports with try/catch importerror. |
|
@spomichter @paul-nechifor Docs have been updated and DDS has been made an optional transport. |
|
@greptile |
|
It works on my machine, I think its because the tests aren't installing the dds extra uv sync --extra dds |
# Conflicts: # pyproject.toml # uv.lock
…1' for improved compatibility
4 participants
This merge request implements DDS (Data Distribution Service) transport layer using CycloneDDS, providing a new high-performance pub/sub transport option.
Quick Start
Note
We currently use Eclipse Cyclone DDS as our DDS implementation. Its IdlStruct feature lets you define DDS topic types in pure Python, eliminating the need for separate IDL files, with automatic serialization support.
Unit Tests/Benchmarks
This builds off of #1144, which was closed due to having too many merge conflicts with
devand #1036, which was closed because the branch was renamed tomiguel/dds_transport