diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md new file mode 100644 index 00000000..99162419 --- /dev/null +++ b/.claude/CLAUDE.md @@ -0,0 +1,306 @@ +# Claude Code Universal Behavior Guidelines + +## Overview + +This document defines universal behavior guidelines for Claude Code across all commands and workflows. These principles apply regardless of the specific command being executed. + +## Core Principles + +### 1. Complete Documentation +- Document every action you take in the appropriate JSON file +- Track all files created, modified, or deleted +- Capture task progress and status changes +- Include all relevant context, decisions, and assumptions +- Never assume information is obvious - document everything explicitly + +### 2. Consistent Output Format +- Always use the unified JSON schema (see below) +- Include all required fields for the relevant status +- Use optional fields as needed to provide additional context +- Validate JSON structure before completing work +- Ensure JSON is properly formatted and parseable + +### 3. Session Management & Stateful Resumption +- Claude Code provides a session ID that maintains conversation context automatically +- Always include `session_id` in your output to enable seamless continuation +- When resuming work from a previous session, include `parent_session_id` to link sessions +- The session ID allows Claude Code to preserve full conversation history +- If you need user input, the context is preserved via session ID +- Include enough detail in `session_summary` to understand what was accomplished +- Don't make the user repeat information - session maintains context + +### 4. Task Management +- Track all tasks in JSON output files (NOT in separate markdown files) +- Use hierarchical task IDs: "1.0" for parent, "1.1", "1.2" for children +- Track task status: pending, in_progress, completed, skipped, blocked +- Include task descriptions and any relevant notes +- Update task status as you work +- Document which tasks were completed in each session +- Note any tasks that were skipped and explain why +- When blocked, document the blocker clearly + +### 5. Query Management +- Save all queries to users in the session JSON file +- When querying users, include: + - Clear, specific questions + - Query type (text, multiple_choice, boolean) + - Any relevant context needed to answer + - Query number for reference +- Save user responses in the same JSON file +- Link queries and responses with query numbers + +## File Organization Structure + +All agent-related documents and files must be organized under the `agent-io` directory: + +``` +agent-io/ +├── prds/ +│ └── / +│ ├── humanprompt.md # Original user description of PRD +│ ├── fullprompt.md # Fully fleshed PRD after completion +│ └── data.json # JSON file documenting queries, responses, tasks, etc. +└── docs/ + └── .md # Architecture docs, usage docs, etc. +``` + +### File Organization Guidelines: +- **PRD Files**: Save to `agent-io/prds//` directory + - Each PRD gets its own directory named after the PRD + - Use kebab-case for PRD names (e.g., "user-profile-editing", "payment-integration") + - Directory contains: humanprompt.md, fullprompt.md, and data.json + - The data.json file tracks all queries, responses, tasks, errors, and progress + +- **PRD Storage and Reference**: + - **When user provides a prompt without a PRD name**: + - Analyze the prompt to create a descriptive PRD name (use kebab-case) + - Create directory: `agent-io/prds//` + - Save the original user prompt to `agent-io/prds//humanprompt.md` + - Document the PRD name in your output for future reference + - This allows users to reference this PRD by name in future sessions + + - **When user references an existing PRD by name**: + - Look for the PRD directory: `agent-io/prds//` + - Read available PRD files in order of preference: + 1. `fullprompt.md` - the complete, finalized PRD (if available) + 2. `humanprompt.md` - the original user description + - Use these files as context for the requested work + - Update or create additional files as needed + + - **PRD Naming Best Practices**: + - Use descriptive, feature-focused names + - Keep names concise (2-4 words typically) + - Use kebab-case consistently + - Examples: "user-authentication", "payment-processing", "real-time-notifications" + +- **Documentation Files**: Save to `agent-io/docs/` + - Architecture documentation: `agent-io/docs/-architecture.md` + - Usage documentation: `agent-io/docs/-usage.md` + - Other documentation as appropriate + +- **Code Files**: Save to appropriate project locations + - Follow existing project structure + - Document each file in the JSON tracking file + - Include purpose and type for each file + +### JSON Documentation Files: +- Every PRD must have an associated `data.json` file in its directory +- The data.json file documents: + - Tasks and their status + - Queries to users and their responses + - Errors and problems encountered + - Files created, modified, deleted + - Session information and summaries + - Comments and context + +## Unified JSON Output Schema + +Use this schema for all JSON output files: + +```json +{ + "command_type": "string (create-prd | doc-code-for-dev | doc-code-usage | free-agent | generate-tasks)", + "status": "string (complete | incomplete | user_query | error)", + "session_id": "string - Claude Code session ID for this execution", + "parent_session_id": "string | null - Session ID of previous session when resuming work", + "session_summary": "string - Brief summary of what was accomplished", + + "tasks": [ + { + "task_id": "string (e.g., '1.0', '1.1', '2.0')", + "description": "string", + "status": "string (pending | in_progress | completed | skipped | blocked)", + "parent_task_id": "string | null", + "notes": "string (optional details about completion/issues)" + } + ], + + "files": { + "created": [ + { + "path": "string (relative to working directory)", + "purpose": "string (why this file was created)", + "type": "string (markdown | code | config | documentation)" + } + ], + "modified": [ + { + "path": "string", + "changes": "string (description of modifications)" + } + ], + "deleted": [ + { + "path": "string", + "reason": "string" + } + ] + }, + + "artifacts": { + "prd_filename": "string (for create-prd command)", + "documentation_filename": "string (for doc-code commands)" + }, + + "queries_for_user": [ + { + "query_number": "integer", + "query": "string", + "type": "string (text | multiple_choice | boolean)", + "choices": [ + { + "id": "string", + "value": "string" + } + ], + "response": "string | null - User's response (populated after query is answered)" + } + ], + + "comments": [ + "string - important notes, warnings, observations" + ], + + "context": "string - optional supplementary state details. Session ID preserves full context automatically, so this field is only needed for additional implementation-specific state not captured in the conversation.", + + "metrics": { + "duration_seconds": "number (optional)", + "files_analyzed": "number (optional)", + "lines_of_code": "number (optional)" + }, + + "errors": [ + { + "message": "string", + "type": "string", + "fatal": "boolean" + } + ] +} +``` + +## Required Fields by Status + +### Status: "complete" +- `command_type`, `status`, `session_id`, `session_summary`, `files`, `comments` +- `parent_session_id` (if this session continues work from a previous session) +- Plus any command-specific artifacts (prd_filename, documentation_filename, etc.) +- `tasks` array if the command involves tasks + +### Status: "user_query" +- `command_type`, `status`, `session_id`, `session_summary`, `queries_for_user` +- `files` (for work done so far) +- `comments` (explaining why input is needed) +- `context` (optional - session_id maintains context automatically) +- Note: When user provides answers, they'll create a new session with `parent_session_id` linking back to this one + +### Status: "incomplete" +- `command_type`, `status`, `session_id`, `session_summary`, `files`, `comments` +- Explanation in `comments` of what's incomplete and why +- `errors` array if errors caused incompleteness +- `context` (optional - session_id maintains context automatically) + +### Status: "error" +- `command_type`, `status`, `session_id`, `session_summary`, `errors`, `comments` +- `files` (if any work was done before error) +- `context` (optional - for additional recovery details beyond what session maintains) + +## Error Handling + +When errors occur: +1. Set status to "error" (or "incomplete" if partial work succeeded) +2. Document the error in the `errors` array +3. Include what failed, why it failed, and potential fixes +4. Document any work that was completed before the error +5. Provide context for potential recovery +6. Save error details to the JSON file + +## Code Development Guidelines + +### Keep Code Simple +- Prefer simple, straightforward implementations over clever or complex solutions +- Write code that is easy to read and understand +- Avoid unnecessary abstractions or over-engineering +- Use clear, descriptive variable and function names +- Comment complex logic, but prefer self-documenting code + +### Limit Complexity +- Minimize the number of classes and Python files +- Consolidate related functionality into fewer, well-organized modules +- Only create new files when there's a clear separation of concerns +- Avoid deep inheritance hierarchies +- Prefer composition over inheritance when appropriate + +### Use JSON Schema Validation +- All JSON files must have corresponding JSON schemas +- Validate JSON files against their schemas +- Document the schema in comments or separate schema files +- Use schema validation to catch errors early +- Keep schemas simple and focused + +### Keep Code Management Simple +- Don't use excessive linting rules +- Avoid complex documentation frameworks (like Sphinx) unless truly needed +- Use simple, standard tools (pytest for testing, basic linting) +- Focus on clear code over extensive tooling +- Documentation should be clear markdown files, not generated sites + +## Best Practices + +- **Be Specific**: Include file paths, line numbers, function names +- **Be Complete**: Don't leave out details assuming the user knows them +- **Be Clear**: Write for someone who wasn't watching you work +- **Be Actionable**: Comments should help the user understand next steps +- **Be Honest**: If something is incomplete or uncertain, say so +- **Be Consistent**: Follow the same patterns and conventions throughout +- **Be Thorough**: Test your work and verify it functions correctly +- **Be Organized**: Maintain clean directory structure and file organization + +## Workflow Principles + +### PRD Workflow +1. User provides initial feature description → saved as `humanprompt.md` +2. Complete PRD after workflow → saved as `fullprompt.md` +3. All progress tracked in `.json` + +### Task Workflow +1. Break work into clear, manageable tasks +2. Use hierarchical task IDs (1.0, 1.1, 1.2, 2.0, etc.) +3. Update task status as work progresses +4. Document completed work and any blockers +5. Track everything in JSON file + +### Documentation Workflow +1. Understand the codebase or feature thoroughly +2. Create clear, well-organized documentation +3. Save to appropriate location in `agent-io/docs/` +4. Track file creation and content in JSON output +5. Include examples and practical guidance + +### Query Workflow +1. Only query when genuinely needed +2. Ask clear, specific questions +3. Save query to JSON file with query_number +4. Wait for user response +5. Save response to same JSON file +6. Continue work with provided information diff --git a/.claude/commands/analyze-email.md b/.claude/commands/analyze-email.md new file mode 100644 index 00000000..967e101a --- /dev/null +++ b/.claude/commands/analyze-email.md @@ -0,0 +1,282 @@ +# Command: analyze-email + +## Purpose + +Analyze an email document to extract key information, classify its importance, assign it to relevant projects, identify action items, and prepare a draft response. All analysis results are saved to a structured JSON file for downstream processing. + +## Command Type + +`analyze-email` + +## Input + +You will receive a request file containing: +- Email content (body, subject, sender, recipients) +- Email metadata (date, time, headers) +- User preferences (optional) + +## Process + +### Phase 1: Email Content Analysis + +1. **Read Email Document** + - Parse email subject, body, sender, recipients + - Extract metadata (date, time, CC, BCC if available) + - Identify attachments mentioned or referenced + - Note email thread context if provided + +2. **Extract Key Information** + - Identify main topics and themes + - Extract specific requests or questions + - Note mentioned dates, deadlines, or time-sensitive information + - Identify key stakeholders mentioned + - Extract any reference numbers, project codes, or identifiers + +### Phase 2: Classification + +3. **Classify Email Importance** + - Analyze content and metadata to classify as one of: + - **unimportant**: Mass emails, newsletters, low-priority updates, spam-like content + - **personal**: Personal correspondence, non-work related, social invitations + - **professional**: Work-related, business correspondence, project updates, actionable items + + - Consider these factors: + - Sender relationship (colleague, client, vendor, unknown) + - Subject urgency indicators (urgent, ASAP, deadline, etc.) + - Content type (FYI, action required, question, update) + - Presence of deadlines or action items + - Email thread importance + + - Provide classification confidence score (0.0-1.0) + - Document classification reasoning in comments + - Emails classified as unimportant should not proceed to further processing + +### Phase 3: Task Extraction + +4. **Identify Action Items** + - Scan email for explicit tasks: + - Action verbs (review, approve, send, create, update, etc.) + - Questions requiring responses + - Requests for information or deliverables + - Meeting requests or scheduling needs + + - For each identified task: + - Extract task description + - Determine task type (respond, review, create, schedule, research, etc.) + - Identify task owner (you, sender, other party) + - Extract related context and requirements + +5. **Determine Urgency and Deadlines** + - Analyze for urgency indicators: + - **Critical**: Explicit urgent markers, imminent deadlines (<24 hours), blocking issues + - **High**: Near-term deadlines (1-3 days), important stakeholders, escalations + - **Medium**: Standard deadlines (4-7 days), routine requests, normal priority + - **Low**: Long-term deadlines (>7 days), FYI items, optional tasks + + - Extract deadlines: + - Explicit dates ("by Friday", "before March 15") + - Implicit timeframes ("ASAP", "end of week", "Q1") + - Recurring deadlines ("weekly report", "monthly update") + + - Convert to standardized format (ISO 8601) + - If no deadline specified, suggest reasonable deadline based on urgency + +### Phase 4: Draft Response + +6. **Analyze Response Requirements** + - Determine if response is needed + - Identify key points to address + - Note any questions to answer + - Consider required tone (formal, casual, apologetic, etc.) + - Identify if response requires attachments or follow-up actions + +7. **Generate Draft Response** + - Create draft email response including: + - Appropriate greeting based on sender relationship + - Address all questions and requests + - Confirm understanding of tasks and deadlines + - Propose next steps if applicable + - Professional closing + + - Match tone to original email and relationship + - Keep response concise and actionable + - Include placeholders for information you don't have ([YOUR_INPUT_NEEDED]) + - Add suggested subject line (Re: or continuation) + + - If no response needed, set draft_response to null and explain why + +### Phase 5: Save Structured Output + +8. **Prepare JSON Output File** + - Determine sequence number for email analysis + - Check `orchestrator/email-analysis/` directory for existing analyses + - Use next sequential number (0001, 0002, 0003, etc.) + - If directory doesn't exist, create it and start at 0001 + +9. **Save Analysis File** + - Filename format: `orchestrator/email-analysis/[NNNN]-[YYYY-MM-DD]-[sender-name].json` + - Example: `orchestrator/email-analysis/0042-2025-11-09-john-smith.json` + - Use kebab-case for sender name + - Document the filename in JSON output's `artifacts.analysis_filename` + +## JSON Output Schema + +The analysis JSON file must follow this structure: + +```json +{ + "email_metadata": { + "subject": "string", + "sender": { + "name": "string", + "email": "string" + }, + "recipients": { + "to": ["email1@example.com", "email2@example.com"], + "cc": ["email3@example.com"], + "bcc": [] + }, + "date_received": "ISO 8601 datetime", + "thread_id": "string or null", + "message_id": "string or null", + "attachments": ["filename1.pdf", "filename2.xlsx"] + }, + + "classification": { + "category": "unimportant | personal | professional", + "confidence": 0.95, + "reasoning": "Detailed explanation of classification decision", + "urgency_level": "critical | high | medium | low", + "is_actionable": true, + "sentiment": "positive | neutral | negative | mixed" + }, + + "tasks": [ + { + "task_id": "T001", + "description": "Review and approve the Q4 budget proposal", + "task_type": "review | respond | create | schedule | research | approve | other", + "owner": "self | sender | other", + "urgency": "critical | high | medium | low", + "deadline": { + "date": "ISO 8601 datetime or null", + "is_explicit": true, + "original_text": "by end of week", + "suggested_deadline": "ISO 8601 datetime - if no explicit deadline" + }, + "status": "pending", + "context": "Additional context from email about this task", + "dependencies": ["T002"], + "estimated_effort": "15 minutes | 1 hour | 2 hours | 1 day | 1 week" + } + ], + + "draft_response": { + "should_respond": true, + "response_urgency": "immediate | today | this_week | no_rush", + "suggested_subject": "Re: Q4 Budget Review Request", + "draft_body": "Full draft email body with appropriate greeting, content, and closing", + "tone": "formal | professional | casual | friendly | apologetic", + "requires_attachments": false, + "placeholders": [ + { + "placeholder": "[YOUR_INPUT_NEEDED]", + "description": "Insert your availability for the meeting", + "location": "paragraph 2" + } + ], + "key_points_to_address": [ + "Confirm receipt of budget proposal", + "Provide timeline for review", + "Ask clarifying questions about line items" + ] + }, + + "summary": { + "one_line": "Budget approval request from Finance requiring review by Friday", + "detailed": "Longer summary (2-3 sentences) of email content and required actions", + "key_entities": [ + {"type": "person", "value": "Jane Doe"}, + {"type": "project", "value": "Q4 Budget Planning"}, + {"type": "document", "value": "Budget_Proposal_Q4.xlsx"}, + {"type": "date", "value": "2025-11-15"} + ] + }, + + "analysis_metadata": { + "analyzed_at": "ISO 8601 datetime", + "analysis_version": "1.0", + "model_used": "string", + "processing_time_seconds": 3.45, + "confidence_overall": 0.89, + "requires_human_review": false, + "review_reason": "string or null - why human review is needed" + } +} +``` + +## Command JSON Output Requirements + +Your command execution JSON output must include: + +**Required Fields:** +- `command_type`: "analyze-email" +- `status`: "complete", "user_query", or "error" +- `session_summary`: Brief summary of email analysis +- `files.created`: Array with the analysis JSON file entry +- `artifacts.analysis_filename`: Path to the analysis JSON file +- `artifacts.email_data`: Copy of the email_metadata for quick reference +- `comments`: Array of notes about the analysis process + +**For user_query status:** +- `queries_for_user`: Questions needing clarification +- `context`: Save partial analysis and email content + +**Example Comments:** +- "Email classified as professional with high confidence (0.95)" +- "Identified 3 action items with deadlines ranging from 2-5 days" +- "Draft response prepared; requires user input for meeting availability" +- "No explicit deadlines found; suggested deadlines based on urgency level" + +## Tasks to Track + +Create tasks in the internal todo list: + +``` +1.0 Parse and extract email content +2.0 Classify email importance and urgency +3.0 Extract tasks and deadlines +4.0 Generate draft response +5.0 Save structured JSON file +``` + +Mark tasks as completed as you progress. + +## Quality Checklist + +Before marking complete, verify: +- ✅ Email metadata completely extracted and validated +- ✅ Classification includes confidence score and reasoning +- ✅ All action items extracted with urgency and deadlines +- ✅ Deadlines converted to ISO 8601 format +- ✅ Draft response addresses all key points (if response needed) +- ✅ JSON file saved with correct naming and structure +- ✅ All required JSON schema fields populated +- ✅ Comments include insights about classification and task extraction +- ✅ Edge cases handled (no deadline, no clear tasks, etc.) + +## Error Handling + +Handle these scenarios gracefully: + +1. **Malformed Email**: Return error status with details +2. **No Clear Tasks**: Set tasks array to empty, note in comments +3. **Ambiguous Classification**: Use most likely category, lower confidence score +4. **No Response Needed**: Set draft_response.should_respond to false with explanation + +## Privacy and Security Considerations + +- Ensure sensitive information (passwords, SSNs, credentials) is not logged in comments +- Redact sensitive data in analysis file if present in email +- Document any sensitive content detected in analysis_metadata.requires_human_review +- Do not include full email body in command output JSON, only in analysis file diff --git a/.claude/commands/claude-commands-expert.md b/.claude/commands/claude-commands-expert.md new file mode 100644 index 00000000..d7777f78 --- /dev/null +++ b/.claude/commands/claude-commands-expert.md @@ -0,0 +1,309 @@ +# ClaudeCommands Expert + +You are an expert on the ClaudeCommands repository - a system for managing Claude Code commands and skills across multiple projects. You have deep knowledge of: + +1. **The CLI Tool** - `claude-commands` for installing, updating, and managing commands +2. **Command/Skill Development** - How to create new commands and expert skills +3. **System Architecture** - The two-file input pattern, unified JSON output, session management +4. **Deployment Workflow** - How commands are deployed to projects and ~/.claude + +## Repository Purpose + +ClaudeCommands exists to solve a key problem: **managing reusable Claude Code extensions across multiple projects**. + +**What it provides:** +- A centralized repository for command and skill definitions +- A CLI tool to deploy commands to any project +- A unified output schema for consistent JSON results +- Expert skills that provide domain-specific knowledge + +**The ecosystem includes skills for:** +- ModelSEEDpy metabolic modeling (`/modelseedpy-expert`) +- MSModelUtil class (`/msmodelutl-expert`) +- FBA packages (`/fbapkg-expert`) +- KBase SDK development (`/kb-sdk-dev`) +- This repository itself (`/claude-commands-expert`) + +## Related Commands + +- `/create-skill` - **Use this for guided skill creation.** Interactively creates new skills with comprehensive content through a 4-phase workflow. + +## CLI Execution + +You can execute CLI commands when users ask you to manage projects or deploy updates. + +**Available operations:** +```bash +# List tracked projects +claude-commands list + +# Update all projects with latest commands +claude-commands update + +# Install to global ~/.claude +claude-commands install + +# Add a new project +claude-commands addproject /path/to/project + +# Remove a project from tracking +claude-commands removeproject project-name +``` + +**When to execute:** +- User asks to "deploy", "update", "install" → Run the appropriate command +- User asks "what projects are tracked" → Run `claude-commands list` +- User asks to "add this project" → Run `claude-commands addproject` + +## Knowledge Loading + +Before answering, read the relevant documentation from this repository: + +**Core Documentation:** +- `/Users/chenry/Dropbox/Projects/ClaudeCommands/README.md` - Overview and quick start +- `/Users/chenry/Dropbox/Projects/ClaudeCommands/docs/CLI.md` - CLI tool documentation +- `/Users/chenry/Dropbox/Projects/ClaudeCommands/docs/ARCHITECTURE.md` - System design + +**When needed:** +- `/Users/chenry/Dropbox/Projects/ClaudeCommands/SYSTEM-PROMPT.md` - Universal system instructions +- `/Users/chenry/Dropbox/Projects/ClaudeCommands/claude_commands.py` - CLI implementation + +## Quick Reference + +### Repository Structure +``` +ClaudeCommands/ +├── SYSTEM-PROMPT.md # Universal instructions for all commands +├── claude_commands.py # CLI tool implementation +├── setup.py # pip install configuration +├── commands/ # SOURCE command definitions +│ ├── create-prd.md # PRD generation command +│ ├── create-skill.md # Interactive skill creation +│ ├── free-agent.md # Simple task execution +│ ├── msmodelutl-expert.md # Expert skill example +│ └── msmodelutl-expert/ # Context subdirectory +│ └── context/ +│ ├── api-summary.md +│ ├── patterns.md +│ └── integration.md +├── data/ # CLI runtime data +│ └── projects.json # Tracked projects +├── docs/ # Documentation +└── .claude/ # Local installation (for testing) + ├── CLAUDE.md + └── commands/ +``` + +### CLI Commands + +| Command | Purpose | +|---------|---------| +| `claude-commands install` | Install to ~/.claude (global) | +| `claude-commands addproject ` | Add project and install commands | +| `claude-commands update` | Update all tracked projects | +| `claude-commands list` | List all tracked projects | +| `claude-commands removeproject ` | Stop tracking a project | + +### Two Types of Extensions + +| Aspect | Command | Expert Skill | +|--------|---------|--------------| +| Purpose | Execute a specific task | Answer questions, provide guidance | +| Input | Request JSON file | User question (natural language) | +| Output | JSON + artifacts | Conversational response | +| Invocation | Headless execution | `/skill-name ` | +| Examples | create-prd, generate-tasks | msmodelutl-expert, kb-sdk-dev | + +### Creating an Expert Skill + +For guided creation, use: `/create-skill` + +Manual creation structure: +``` +commands/ +├── .md # Main skill definition +└── / # Optional context directory + └── context/ + ├── api-summary.md # Quick API reference + ├── patterns.md # Common usage patterns + └── integration.md # Integration with other modules +``` + +Main skill file template: +```markdown +# Expert + +You are an expert on . You have deep knowledge of: +1. **Topic 1** - Description +2. **Topic 2** - Description + +## Knowledge Loading +Before answering, read: +- `/path/to/documentation.md` +- `/path/to/source/code.py` (when needed) + +## Quick Reference +[Embedded patterns and common info] + +## Guidelines +How to respond to questions + +## User Request +$ARGUMENTS +``` + +### Creating a Command + +Command file template: +```markdown +# Command: + +## Purpose +What this command does + +## Command Type +`` + +## Core Directive +What Claude should do + +## Input +What the request file should contain + +## Process +Step-by-step execution + +## Output Requirements +What goes in the JSON output + +## Quality Checklist +Verification steps +``` + +### Deployment Flow + +``` +commands/ (source) + │ + ├── install ──────────────► ~/.claude/commands/ + │ + └── addproject ───────────► project/.claude/commands/ + │ + └── update ───────► All tracked projects +``` + +### Unified JSON Output Schema + +All commands produce output following this schema: +```json +{ + "command_type": "string", + "status": "complete|incomplete|user_query|error", + "session_id": "string", + "session_summary": "string", + "tasks": [...], + "files": { "created": [], "modified": [], "deleted": [] }, + "artifacts": {...}, + "queries_for_user": [...], + "comments": [...], + "errors": [...] +} +``` + +## Common Tasks + +### "I want to create a new skill" +Recommend using `/create-skill` for guided creation. It will: +1. Ask about the domain and knowledge areas +2. Propose a structure +3. Create files with comprehensive content +4. Optionally deploy to all projects + +### "Deploy the latest changes" +Execute: `claude-commands update` + +### "What projects are using these commands?" +Execute: `claude-commands list` + +### "Add my current project" +Execute: `claude-commands addproject /path/to/project` + +## Troubleshooting + +### "Commands not appearing in project" +1. Check if project is tracked: `claude-commands list` +2. If missing, add it: `claude-commands addproject /path` +3. If tracked but outdated, update: `claude-commands update` + +### "Skill not loading context files" +1. Verify context files exist in `commands//context/` +2. Run `claude-commands update` to deploy latest +3. Check file paths in Knowledge Loading section are correct + +### "Changes not reflecting in tracked projects" +Run `claude-commands update` - this copies latest from source to all projects + +## Guidelines for Responding + +When helping users: + +1. **Be practical** - Provide working examples and commands +2. **Reference files** - Point to specific files in the repository +3. **Explain the flow** - Show how components connect +4. **Execute when asked** - Run CLI commands for deploy/install/list requests +5. **Recommend /create-skill** - For users wanting to create new skills + +## Response Formats + +### For "how do I" questions: +``` +### Approach + +Brief explanation + +**Step 1:** Description +```bash +command or code +``` + +**Step 2:** Description +... + +**Files involved:** List of relevant files +``` + +### For architecture questions: +``` +### Overview + +Brief explanation of the component/concept + +### How It Works + +1. First... +2. Then... + +### Key Files + +- `path/to/file.md` - Purpose +- `path/to/code.py` - Purpose + +### Example + +Working example +``` + +### For CLI requests: +Execute the command and report results: +``` +Running `claude-commands update`... + +✓ Updated 19 projects: + - ProjectA: 15 commands + 12 context files + - ProjectB: 15 commands + 12 context files + ... +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/claude-commands-expert/context/architecture.md b/.claude/commands/claude-commands-expert/context/architecture.md new file mode 100644 index 00000000..b04e140c --- /dev/null +++ b/.claude/commands/claude-commands-expert/context/architecture.md @@ -0,0 +1,282 @@ +# ClaudeCommands Architecture + +## System Overview + +ClaudeCommands is a framework for running Claude Code in headless mode with structured input/output and comprehensive documentation. + +``` +┌─────────────────────────────────────────────────────────────┐ +│ ClaudeCommands Repository │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ SYSTEM- │ │ commands/ │ │ data/ │ │ +│ │ PROMPT.md │ │ *.md │ │ projects.json│ │ +│ │ │ │ */context/ │ │ │ │ +│ │ (Universal │ │ (Command & │ │ (Tracked │ │ +│ │ instructions)│ │ Skill defs) │ │ projects) │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +│ │ │ │ +│ └────────┬────────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────────┐ │ +│ │claude_ │ │ +│ │commands.py │ ◄───── CLI Tool │ +│ │ │ │ +│ └──────────────┘ │ +│ │ │ +│ ┌───────┴───────┐ │ +│ │ │ │ +│ ▼ ▼ │ +│ ┌────────────┐ ┌────────────────┐ │ +│ │ ~/.claude/ │ │ project/.claude│ │ +│ │ (Global) │ │ (Per-project) │ │ +│ └────────────┘ └────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Core Components + +### 1. SYSTEM-PROMPT.md + +Universal instructions that apply to ALL command executions. + +**Purpose:** Define the "rules of the game" - output format, documentation requirements, session management. + +**Key sections:** +- Core Principles (documentation, output format, session management) +- Unified JSON Output Schema +- File Organization Structure +- Error Handling +- Best Practices + +**Deployed as:** `CLAUDE.md` in target directories + +### 2. Command Files (commands/*.md) + +Define WHAT to do for specific command types. + +**Categories:** +- **Task Commands:** create-prd, generate-tasks, free-agent +- **Documentation Commands:** doc-code-for-dev, doc-code-usage +- **Expert Skills:** msmodelutl-expert, claude-commands-expert + +**Structure:** +```markdown +# Command/Skill Name +## Purpose +## Command Type (for commands) or Knowledge Loading (for skills) +## Process/Guidelines +## Output Requirements/Response Format +## Quality Checklist +``` + +### 3. CLI Tool (claude_commands.py) + +Manages deployment of commands to projects. + +**Key methods:** +```python +class ClaudeCommandsCLI: + def install(self) # → ~/.claude/ + def addproject(self, dir) # → project/.claude/ + def update(self) # → All tracked projects + def list(self) # → Show tracked projects + def removeproject(self, name) +``` + +**Deployment logic:** +```python +def _copy_files_to_project(self, project_path): + # 1. Copy SYSTEM-PROMPT.md → .claude/CLAUDE.md + shutil.copy2(self.system_prompt, target_prompt) + + # 2. Copy entire commands/ → .claude/commands/ + shutil.copytree(self.commands_dir, target_commands) + # (Preserves subdirectories for skills with context) +``` + +### 4. Project Tracking (data/projects.json) + +Tracks which projects have commands installed. + +```json +{ + "project-name": "/absolute/path/to/project", + "another-project": "/path/to/another" +} +``` + +## Information Flow + +### Headless Execution + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Claude Code CLI │ +│ │ +│ Inputs: │ +│ ├─ --system-prompt .claude/CLAUDE.md │ +│ ├─ --command .claude/commands/.md │ +│ └─ --request request.json │ +│ │ +│ Execution: │ +│ ├─ Reads and follows system prompt │ +│ ├─ Follows command-specific instructions │ +│ ├─ Creates artifacts (PRDs, docs, code) │ +│ └─ Documents everything │ +│ │ +│ Outputs: │ +│ ├─ claude-output.json (complete execution record) │ +│ └─ [artifacts] (files created by command) │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Skill Invocation + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Claude Code IDE/CLI │ +│ │ +│ User types: /skill-name How do I do X? │ +│ │ +│ Claude: │ +│ ├─ Loads .claude/commands/skill-name.md │ +│ ├─ Follows Knowledge Loading instructions │ +│ ├─ Reads referenced documentation (dynamic) │ +│ ├─ Uses Quick Reference (static) │ +│ └─ Responds following Guidelines │ +│ │ +│ Output: Conversational response with examples │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Deployment Architecture + +### Global Installation (~/.claude/) + +Available in all projects (fallback when project-level not present). + +``` +~/.claude/ +├── CLAUDE.md # System prompt +└── commands/ + ├── command1.md + ├── skill1.md + └── skill1/ + └── context/ + └── *.md +``` + +### Project Installation (project/.claude/) + +Project-specific, takes precedence over global. + +``` +my-project/ +├── .claude/ +│ ├── CLAUDE.md # System prompt +│ └── commands/ +│ └── ... +└── (project files) +``` + +### Precedence + +1. Project-level (.claude/) - checked first +2. User-level (~/.claude/) - fallback + +## Unified JSON Output Schema + +All commands produce structured output: + +```json +{ + "command_type": "string", + "status": "complete|incomplete|user_query|error", + "session_id": "string", + "parent_session_id": "string|null", + "session_summary": "string", + + "tasks": [ + { + "task_id": "1.0", + "description": "string", + "status": "pending|in_progress|completed|skipped|blocked", + "parent_task_id": "string|null", + "notes": "string" + } + ], + + "files": { + "created": [{"path": "", "purpose": "", "type": ""}], + "modified": [{"path": "", "changes": ""}], + "deleted": [{"path": "", "reason": ""}] + }, + + "artifacts": { + "prd_filename": "string", + "documentation_filename": "string" + }, + + "queries_for_user": [ + { + "query_number": 1, + "query": "string", + "type": "text|multiple_choice|boolean", + "choices": [{"id": "", "value": ""}], + "response": "string|null" + } + ], + + "comments": ["string"], + "context": "string", + + "errors": [ + { + "message": "string", + "type": "string", + "fatal": true + } + ] +} +``` + +## Extension Points + +### Adding New Commands + +1. Create `commands/.md` following template +2. Deploy with `claude-commands update` + +### Adding Expert Skills + +1. Create `commands/.md` with Knowledge Loading +2. Optionally add `commands//context/` for reference docs +3. Deploy with `claude-commands update` + +### Modifying System Behavior + +1. Edit `SYSTEM-PROMPT.md` +2. Deploy with `claude-commands update` + +## Design Principles + +1. **Separation of Concerns** + - Universal rules → SYSTEM-PROMPT.md + - Command logic → commands/*.md + - User requests → request.json + +2. **Single Source of Truth** + - One system prompt for all commands + - One output schema for all outputs + +3. **Complete Documentation** + - Everything in JSON (user can't see terminal) + - All file operations tracked + - Session management for resumption + +4. **Centralized Management** + - Commands developed in one repo + - Deployed to many projects + - Single update pushes everywhere diff --git a/.claude/commands/claude-commands-expert/context/cli-reference.md b/.claude/commands/claude-commands-expert/context/cli-reference.md new file mode 100644 index 00000000..98c9fb30 --- /dev/null +++ b/.claude/commands/claude-commands-expert/context/cli-reference.md @@ -0,0 +1,187 @@ +# ClaudeCommands CLI Reference + +## Installation + +```bash +cd /path/to/ClaudeCommands +pip install -e . +``` + +This installs the `claude-commands` command globally. + +## Commands + +### install + +Install commands to user's home directory (~/.claude). + +```bash +claude-commands install +``` + +**What it does:** +1. Creates `~/.claude/` directory if missing +2. Copies `SYSTEM-PROMPT.md` to `~/.claude/CLAUDE.md` +3. Copies all commands (including subdirectories) to `~/.claude/commands/` + +**Prompts:** +- Asks before overwriting existing files + +### addproject + +Add a project to tracking and install commands. + +```bash +claude-commands addproject ~/my-project +``` + +**What it does:** +1. Validates the directory exists +2. Adds to tracking list (`data/projects.json`) +3. Creates `.claude/` directory in project +4. Copies `SYSTEM-PROMPT.md` to `.claude/CLAUDE.md` +5. Copies all commands to `.claude/commands/` + +**Name collision:** +- Projects are tracked by directory name +- Two projects with same directory name → error + +### update + +Update all tracked projects with latest commands. + +```bash +claude-commands update +``` + +**What it does:** +1. Reads `data/projects.json` +2. For each project: + - Verifies directory exists + - Re-copies SYSTEM-PROMPT.md and commands +3. Reports success/warnings + +### list + +List all tracked projects. + +```bash +claude-commands list +``` + +**Output:** +``` +Tracked projects (3): + + ✓ my-project + /Users/me/code/my-project + + ✓ another-app + /Users/me/work/another-app + + ✗ deleted-project + /Users/me/old/deleted-project +``` + +- ✓ = directory exists +- ✗ = directory missing + +### removeproject + +Remove a project from tracking. + +```bash +claude-commands removeproject my-project +``` + +**What it does:** +1. Removes from `data/projects.json` +2. Does NOT delete `.claude/` directory + +## Project Tracking + +Projects tracked in `data/projects.json`: + +```json +{ + "my-project": "/Users/me/code/my-project", + "another-app": "/Users/me/work/another-app" +} +``` + +**Key points:** +- Project name = directory name (not path) +- Paths are absolute +- File is gitignored (local to machine) +- Don't edit manually - use CLI + +## File Structure After Installation + +### User-level (~/.claude/) +``` +~/.claude/ +├── CLAUDE.md # Universal system prompt +└── commands/ + ├── create-prd.md + ├── free-agent.md + ├── msmodelutl-expert.md + └── msmodelutl-expert/ + └── context/ + ├── api-summary.md + ├── patterns.md + └── integration.md +``` + +### Project-level (project/.claude/) +``` +my-project/ +├── .claude/ +│ ├── CLAUDE.md # Universal system prompt +│ └── commands/ +│ ├── create-prd.md +│ └── ... +└── (project files) +``` + +## Workflow Examples + +### Initial Setup +```bash +# Clone repo +git clone ClaudeCommands +cd ClaudeCommands + +# Install CLI +pip install -e . + +# Install to home directory +claude-commands install + +# Add your projects +claude-commands addproject ~/project1 +claude-commands addproject ~/project2 +``` + +### After Modifying Commands +```bash +# Edit a command +vim commands/my-command.md + +# Push to all projects +claude-commands update +``` + +### Adding New Project +```bash +claude-commands addproject ~/new-project +# Commands automatically installed +``` + +### Cleaning Up +```bash +# Remove project from tracking +claude-commands removeproject old-project + +# Manually delete .claude if desired +rm -rf ~/old-project/.claude +``` diff --git a/.claude/commands/claude-commands-expert/context/skill-development.md b/.claude/commands/claude-commands-expert/context/skill-development.md new file mode 100644 index 00000000..6972cdc5 --- /dev/null +++ b/.claude/commands/claude-commands-expert/context/skill-development.md @@ -0,0 +1,377 @@ +# Skill/Command Development Guide + +## Overview + +This repository manages two types of Claude Code extensions: + +1. **Commands** - Task-oriented instructions (create-prd, free-agent, etc.) +2. **Expert Skills** - Domain-specific knowledge assistants (msmodelutl-expert, etc.) + +## Quick Start: Use /create-skill + +For guided skill creation with comprehensive content generation, use the `/create-skill` command: + +``` +/create-skill Docker container management +``` + +This interactive command will: +1. **Discovery** - Ask about knowledge areas, source files, related skills +2. **Design** - Propose structure and get your approval +3. **Creation** - Generate comprehensive skill files with real content +4. **Deployment** - Optionally deploy to all tracked projects + +**When to use /create-skill vs manual creation:** +- Use `/create-skill` for: New domains, when you want guided creation, comprehensive content +- Use manual creation for: Quick edits, copying existing skill patterns, full control + +## Command vs. Skill + +| Aspect | Command | Expert Skill | +|--------|---------|--------------| +| Purpose | Execute a specific task | Answer questions, provide guidance | +| Input | Request JSON file | User question (natural language) | +| Output | JSON + artifacts | Conversational response | +| Invocation | `claude code headless --command` | `/skill-name ` | +| Examples | create-prd, generate-tasks | msmodelutl-expert | + +## Creating a New Command + +### Step 1: Create Command File + +Create `commands/.md`: + +```markdown +# Command: + +## Purpose +Brief description of what this command does. + +## Command Type +`` + +## Core Directive +You are a [role]. Your job is to [primary responsibility]. + +**YOUR JOB:** +- ✅ Task 1 +- ✅ Task 2 + +**DO NOT:** +- ❌ Anti-pattern 1 +- ❌ Anti-pattern 2 + +## Input +You will receive a request file containing: +- `field1`: Description +- `field2`: Description + +## Process + +### 1. First Step +Description of what to do. + +### 2. Second Step +Description of what to do. + +## Output Requirements +Describe what goes in the JSON output: +- Required fields +- Artifacts to create +- Files to document + +## Quality Checklist +- ✅ Verification 1 +- ✅ Verification 2 +``` + +### Step 2: Add to Schema + +Update `unified-output-schema.json`: + +```json +"command_type": { + "enum": [..., "your-new-command"] +} +``` + +### Step 3: Create Example + +Add `examples/-example.json`: + +```json +{ + "request_type": "", + "description": "Example request", + "context": { + "relevant_field": "value" + } +} +``` + +### Step 4: Deploy + +```bash +claude-commands update +``` + +## Creating an Expert Skill + +Expert skills provide domain expertise for answering questions. + +### Step 1: Create Main Skill File + +Create `commands/.md`: + +```markdown +# Expert + +You are an expert on . You have deep knowledge of: + +1. **Area 1** - Description +2. **Area 2** - Description +3. **Area 3** - Description + +## Knowledge Loading + +Before answering, read the relevant documentation: + +**Always read:** +- `/path/to/main/documentation.md` + +**When needed:** +- `/path/to/source/code.py` +- `/path/to/additional/docs.md` + +## Quick Reference + +### Key Concept 1 +```python +# Code example +example_code() +``` + +### Key Concept 2 +Brief explanation with example. + +### Common Mistakes +1. **Mistake 1**: How to avoid it +2. **Mistake 2**: How to avoid it + +## Guidelines for Responding + +When helping users: + +1. **Be specific** - Reference exact functions, parameters +2. **Show examples** - Provide working code +3. **Explain why** - Not just what, but why +4. **Warn about pitfalls** - Common mistakes + +## Response Formats + +### For API questions: +\``` +### Method: `method_name(params)` + +**Purpose:** Description + +**Parameters:** +- `param1` (type): Description + +**Returns:** Description + +**Example:** +```python +code +``` +\``` + +### For "how do I" questions: +\``` +### Approach + +Explanation + +**Step 1:** Description +```python +code +``` +\``` + +## User Request + +$ARGUMENTS +``` + +### Step 2: Create Context Directory (Optional) + +For skills with extensive reference material: + +``` +commands/ +├── .md +└── / + └── context/ + ├── api-summary.md # Quick API reference + ├── patterns.md # Common usage patterns + └── integration.md # Integration with other systems +``` + +### Step 3: Choose Documentation Strategy + +**Static Context (embedded in skill):** +- Faster response time +- Requires manual updates when source changes +- Best for: stable APIs, patterns that rarely change + +**Dynamic Loading (read files on invocation):** +- Always current with source +- Slightly slower +- Best for: actively developed code + +**Hybrid Approach (recommended):** +- Static: patterns, common mistakes, integration info +- Dynamic: full API reference, source code + +Example hybrid: +```markdown +## Knowledge Loading + +Before answering, read the current documentation: +- `/path/to/developer-guide.md` # Dynamic - always current + +## Quick Reference +[Embedded patterns and common info] # Static - fast access +``` + +### Step 4: Deploy + +```bash +claude-commands update +``` + +## Skill Invocation + +After deployment, invoke with: + +``` +/skill-name How do I do X? +/skill-name What's the difference between A and B? +/skill-name Debug this code that's failing +``` + +## Best Practices + +### For Commands +1. Follow the standard template structure +2. Reference SYSTEM-PROMPT.md for output format (don't duplicate) +3. Include clear quality checklist +4. Document expected request format +5. Provide examples + +### For Expert Skills +1. Use dynamic loading for frequently-updated documentation +2. Embed common patterns for fast access +3. Include response format templates +4. Warn about common mistakes +5. Reference exact file paths + +### For Both +1. Keep files in `commands/` directory (source) +2. Use `claude-commands update` to deploy +3. Test locally before pushing to all projects +4. Document in comments what each file does + +## File Naming Conventions + +| Type | Pattern | Example | +|------|---------|---------| +| Command | `-.md` | `create-prd.md`, `doc-code-usage.md` | +| Expert Skill | `-expert.md` | `msmodelutl-expert.md` | +| Context Dir | `/context/` | `msmodelutl-expert/context/` | + +## Real Examples from This Repository + +### Example 1: msmodelutl-expert + +A focused skill for a specific Python class: + +``` +commands/ +├── msmodelutl-expert.md # Main skill (200 lines) +└── msmodelutl-expert/ + └── context/ + ├── api-summary.md # Method signatures, parameters + ├── patterns.md # Common usage patterns + └── integration.md # How it connects to other modules +``` + +**Key features:** +- References source code for dynamic loading +- Embeds 5 essential patterns in Quick Reference +- Common Mistakes section prevents typical errors +- Links to related skills: `/modelseedpy-expert`, `/fbapkg-expert` + +### Example 2: kb-sdk-dev + +A comprehensive skill for KBase SDK development: + +``` +commands/ +├── kb-sdk-dev.md # Main skill +└── kb-sdk-dev/ + └── context/ + ├── kidl-reference.md # KIDL specification + ├── workspace-datatypes.md # Data type reference + ├── ui-spec-reference.md # UI specification + └── kbutillib-integration.md # Integration guide +``` + +**Key features:** +- Multiple context files for different aspects +- Reference documentation for specifications +- Integration guides for related tools + +### Example 3: claude-commands-expert + +A self-referential skill for this repository: + +``` +commands/ +├── claude-commands-expert.md # Main skill +└── claude-commands-expert/ + └── context/ + ├── architecture.md # System design + ├── skill-development.md # This file + └── cli-reference.md # CLI commands +``` + +**Key features:** +- Can execute CLI commands (not just explain) +- References `/create-skill` for guided creation +- Troubleshooting section for common issues + +## Troubleshooting + +### Skill not appearing after creation +1. Verify file is in `commands/` directory +2. Check filename ends with `.md` +3. Run `claude-commands update` to deploy +4. Verify with `claude-commands list` + +### Context files not loading +1. Check directory structure: `/context/` +2. Ensure context files are `.md` format +3. Verify paths in Knowledge Loading section are absolute +4. Run `claude-commands update` after adding context files + +### Skill producing poor responses +1. Add more content to Quick Reference section +2. Include Common Mistakes section +3. Add response format templates +4. Reference more source files in Knowledge Loading + +### Changes not reflecting in projects +1. Ensure you edited files in `commands/` (source), not `.claude/commands/` (deployed) +2. Run `claude-commands update` to push changes +3. Check project is tracked: `claude-commands list` diff --git a/.claude/commands/create-new-project.md b/.claude/commands/create-new-project.md new file mode 100644 index 00000000..b637c50c --- /dev/null +++ b/.claude/commands/create-new-project.md @@ -0,0 +1,637 @@ +# Command: create-new-project + +## Purpose + +Create a new project with complete setup including Cursor workspace configuration, virtual environment management, Claude commands installation, and optional git repository initialization. This command orchestrates multiple setup steps to create a fully configured development environment. + +## Command Type + +`create-new-project` + +## Input + +You will receive a request file containing: +- Project name (required) +- Project directory path (required - can be relative or absolute) +- Project type (optional: python, javascript, typescript, jupyter, multi-language) +- Initialize git repository (optional: boolean, default true) +- Python version for venv (optional: e.g., "3.11", default to system python3) +- Additional workspace folders (optional) +- Workspace settings preferences (optional) + +## Process + +### Phase 1: Create Project Directory + +1. **Setup Project Directory** + - Create project directory at specified path if it doesn't exist + - Convert to absolute path for consistency + - Verify write permissions + - Document the project path + +2. **Validate Project Name** + - Use provided project name or derive from directory name + - Sanitize for use in filenames (remove special characters) + - Check for conflicts with existing projects + - Document the final project name + +### Phase 2: Initialize Git Repository (Optional) + +3. **Git Initialization** + - If git initialization requested (default: true): + - Run: git init + - Create .gitignore file with common patterns + - Create initial commit with project structure + - Document git initialization status + - If git initialization skipped: + - Note in comments why it was skipped + - Continue with setup + +4. **Create .gitignore** + - Add common patterns based on project type: + - Python: venv/, __pycache__/, *.pyc, .pytest_cache/, *.egg-info/ + - JavaScript/Node: node_modules/, dist/, .cache/ + - Jupyter: .ipynb_checkpoints/, notebooks/datacache/ + - General: .DS_Store, .vscode/, *.swp, *.swo + - Claude: .claude/commands/ (managed by claude-commands) + - Keep: .claude/CLAUDE.md, .claude/settings.local.json + - Document .gitignore creation + +### Phase 3: Setup Virtual Environment with venvman + +5. **Register with venvman** + - Run: venvman add PROJECT_NAME PROJECT_PATH + - If Python version specified, create venv with that version + - If not specified, use system default python3 + - Document venvman registration + - Note the virtual environment path + +6. **Activate and Setup Python Environment** + - If project type is Python or Jupyter: + - Install basic dependencies (pip, setuptools, wheel) + - Create requirements.txt if it doesn't exist + - Document Python setup + - If not Python project: + - Note that venv was created but is optional + +### Phase 4: Install Claude Commands + +7. **Register with claude-commands** + - Run: claude-commands addproject PROJECT_PATH + - This installs SYSTEM-PROMPT.md to .claude/CLAUDE.md + - Installs all command files to .claude/commands/ + - Document claude-commands registration + - Count and list installed commands + +### Phase 5: Create Cursor Workspace + +8. **Generate Workspace File** + - Create workspace file: EXCLAMATION + project-name.code-workspace + - Include current directory as primary folder + - Configure settings based on project type + - Add file exclusions (venv/, node_modules/, __pycache__, etc.) + - Add search exclusions for performance + - Document workspace creation + +9. **Configure Project-Specific Settings** + - Python projects: Black formatter, pytest, type checking + - JavaScript/TypeScript: Prettier, ESLint + - Jupyter: Notebook settings, output limits + - Add extension recommendations + - Document all settings configured + +### Phase 6: Create Project Structure + +10. **Create Standard Directories** + - ALWAYS create: agent-io/ directory for Claude command tracking files + - Based on project type, create: + - Python: src/, tests/, docs/ + - Jupyter: notebooks/, notebooks/data/, notebooks/datacache/, notebooks/genomes/, notebooks/models/, notebooks/nboutput/, notebooks/util.py + - JavaScript: src/, tests/, dist/ + - General: docs/, README.md + - Document directory structure created + +11. **Create Initial Files** + - README.md with project name and description + - requirements.txt (for Python projects) + - package.json (for JavaScript projects) + - For Jupyter: notebooks/util.py with NotebookUtil template + - Document files created + +### Phase 7: Finalize Setup + +12. **Create Initial Git Commit (if git enabled)** + - Stage all created files + - Create commit: "Initial project setup: PROJECT_NAME" + - Include setup details in commit message + - Document commit creation + +13. **Generate Setup Summary** + - List all tools registered (venvman, claude-commands) + - List all files and directories created + - Provide next steps for user + - Document complete setup status + +### Phase 8: Save Structured Output + +14. **Save JSON Tracking File** + - IMPORTANT: Save all agent-io output to the NEW project directory, NOT the current working directory + - Create agent-io/ directory in the new project if it doesn't exist + - Save tracking JSON to: NEW_PROJECT_PATH/agent-io/create-new-project-session-SESSIONID.json + - Document all setup steps completed + - List all files and directories created + - Record all command executions + - Note any errors or warnings + - Include completion status + +## JSON Output Schema + +```json +{ + "command_type": "create-new-project", + "status": "complete | incomplete | user_query | error", + "session_id": "string", + "parent_session_id": "string | null", + "session_summary": "Brief summary of project creation", + + "project": { + "name": "string - project name", + "path": "string - absolute path to project", + "type": "python | javascript | typescript | jupyter | multi-language | other" + }, + + "git": { + "initialized": true, + "initial_commit": true, + "commit_hash": "string - git commit hash", + "gitignore_created": true + }, + + "venvman": { + "registered": true, + "command_run": "venvman add PROJECT_NAME PROJECT_PATH", + "venv_path": "string - path to virtual environment", + "python_version": "3.11" + }, + + "claude_commands": { + "registered": true, + "command_run": "claude-commands addproject .", + "commands_installed": 5, + "system_prompt_installed": true, + "commands_list": ["create-prd", "doc-code-for-dev", "doc-code-usage", "jupyter-dev", "cursor-setup"] + }, + + "workspace": { + "filename": "string - workspace file with ! prefix", + "path": "string - absolute path to workspace file", + "folders_count": 1, + "settings_configured": true, + "extensions_recommended": ["ms-python.python", "ms-toolsai.jupyter"] + }, + + "directories_created": [ + "agent-io/", + "src/", + "tests/", + "docs/", + "notebooks/", + "notebooks/data/", + "notebooks/datacache/", + "notebooks/genomes/", + "notebooks/models/", + "notebooks/nboutput/" + ], + + "files": { + "created": [ + { + "path": "!ProjectName.code-workspace", + "purpose": "Cursor workspace configuration", + "type": "config" + }, + { + "path": ".gitignore", + "purpose": "Git ignore patterns", + "type": "config" + }, + { + "path": "README.md", + "purpose": "Project documentation", + "type": "documentation" + }, + { + "path": "requirements.txt", + "purpose": "Python dependencies", + "type": "config" + }, + { + "path": ".claude/CLAUDE.md", + "purpose": "Claude system prompt", + "type": "documentation" + }, + { + "path": "agent-io/create-new-project-session-SESSIONID.json", + "purpose": "Claude command execution tracking for this session", + "type": "tracking" + } + ], + "modified": [] + }, + + "artifacts": { + "project_path": "absolute path to project", + "workspace_file": "path to workspace file", + "readme_file": "path to README.md", + "tracking_file": "agent-io/create-new-project-session-SESSIONID.json" + }, + + "next_steps": [ + "Open workspace: code !ProjectName.code-workspace", + "Activate venv: venvman activate ProjectName", + "Install dependencies: pip install -r requirements.txt", + "Start developing!" + ], + + "comments": [ + "Created project directory at /path/to/project", + "Created agent-io/ directory for Claude command tracking", + "Initialized git repository with initial commit", + "Registered with venvman using Python 3.11", + "Installed 5 Claude commands to .claude/commands/", + "Created Cursor workspace with Python settings", + "Created standard Python project structure (src/, tests/, docs/)", + "Generated README.md and requirements.txt", + "Saved tracking JSON to NEW_PROJECT_PATH/agent-io/" + ], + + "queries_for_user": [], + + "errors": [] +} +``` + +## Command JSON Output Requirements + +**Required Fields:** +- `command_type`: "create-new-project" +- `status`: "complete", "user_query", or "error" +- `session_id`: Session ID for this execution +- `session_summary`: Brief summary of project creation +- `project`: Project details (name, path, type) +- `git`: Git initialization status +- `venvman`: Virtual environment registration +- `claude_commands`: Claude commands registration +- `workspace`: Cursor workspace details +- `directories_created`: List of directories created +- `files`: All files created +- `artifacts`: Key file paths +- `next_steps`: User guidance for next actions +- `comments`: Detailed notes about setup process + +**For user_query status:** +- `queries_for_user`: Questions needing clarification +- `context`: Save partial setup state + +**Example Comments:** +- "Created new project 'MetabolicModeling' at ~/Projects/MetabolicModeling" +- "Initialized git repository with initial commit (abc123f)" +- "Registered with venvman using Python 3.11 at ~/Projects/MetabolicModeling/venv" +- "Installed 5 Claude commands to .claude/commands/" +- "Created Cursor workspace: !MetabolicModeling.code-workspace" +- "Created Jupyter notebook structure with util.py template" +- "Generated .gitignore with Python and Jupyter patterns" + +## .gitignore Template + +### Python Projects +``` +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +venv/ +env/ +ENV/ +.venv +pip-log.txt +pip-delete-this-directory.txt +.pytest_cache/ +.coverage +htmlcov/ +*.egg-info/ +dist/ +build/ + +# Jupyter +.ipynb_checkpoints/ +notebooks/datacache/ + +# IDE +.vscode/ +.idea/ +*.swp +*.swo + +# OS +.DS_Store +Thumbs.db + +# Claude (commands are managed by claude-commands) +.claude/commands/ + +# Agent-IO (Claude command tracking - keep in git for project history) +# agent-io/ is intentionally tracked + +# Keep these +!.claude/CLAUDE.md +!.claude/settings.local.json +``` + +### JavaScript/Node Projects +``` +# Node +node_modules/ +npm-debug.log* +yarn-debug.log* +yarn-error.log* +.pnpm-debug.log* +dist/ +build/ +.cache/ + +# IDE +.vscode/ +.idea/ +*.swp +*.swo + +# OS +.DS_Store +Thumbs.db + +# Claude +.claude/commands/ + +# Agent-IO (keep in git) +# agent-io/ is intentionally tracked + +# Keep these +!.claude/CLAUDE.md +!.claude/settings.local.json +``` + +### Jupyter Projects +``` +# Jupyter +.ipynb_checkpoints/ +notebooks/datacache/ + +# Python +__pycache__/ +*.py[cod] +venv/ +*.egg-info/ + +# Data (keep structure, ignore large files) +notebooks/data/*.csv +notebooks/data/*.tsv +notebooks/data/*.xlsx +notebooks/genomes/*.fasta +notebooks/genomes/*.gbk +notebooks/models/*.xml +notebooks/models/*.json +notebooks/nboutput/* + +# Keep these data directory files +!notebooks/data/.gitkeep +!notebooks/genomes/.gitkeep +!notebooks/models/.gitkeep + +# IDE +.vscode/ +.DS_Store + +# Claude +.claude/commands/ + +# Agent-IO (keep in git) +# agent-io/ is intentionally tracked + +# Keep these +!.claude/CLAUDE.md +``` + +## README.md Template + +```markdown +# PROJECT_NAME + +[Brief project description] + +## Setup + +This project was created with the `create-new-project` Claude command. + +### Prerequisites + +- Python 3.11+ (or appropriate version) +- venvman for virtual environment management +- claude-commands for Claude Code integration + +### Installation + +1. Activate the virtual environment: + ```bash + venvman activate PROJECT_NAME + ``` + +2. Install dependencies: + ```bash + pip install -r requirements.txt + ``` + +### Development + +Open the Cursor workspace: +```bash +code !PROJECT_NAME.code-workspace +``` + +### Project Structure + +- `agent-io/` - Claude command execution tracking and session history +- `src/` - Source code +- `tests/` - Test files +- `docs/` - Documentation +- `notebooks/` - Jupyter notebooks (if applicable) +- `.claude/` - Claude Code configuration (commands managed by claude-commands) + +### Claude Code Integration + +This project includes Claude Code integration: +- Command tracking stored in `agent-io/` for project history +- Commands automatically installed to `.claude/commands/` (managed by claude-commands) +- Update commands: `claude-commands update` + +## License + +[Add license information] +``` + +## Jupyter util.py Template + +For Jupyter projects, create notebooks/util.py: + +```python +import sys +import os +import json +from os import path + +# Add the parent directory to the sys.path +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +script_path = os.path.abspath(__file__) +script_dir = os.path.dirname(script_path) +base_dir = os.path.dirname(os.path.dirname(script_dir)) +folder_name = os.path.basename(script_dir) + +print(base_dir+"/KBUtilLib/src") +sys.path = [base_dir+"/KBUtilLib/src",base_dir+"/cobrakbase",base_dir+"/ModelSEEDpy/"] + sys.path + +# Import utilities with error handling +from kbutillib import NotebookUtils + +import hashlib +import pandas as pd +from modelseedpy import AnnotationOntology, MSPackageManager, MSMedia, MSModelUtil, MSBuilder, MSATPCorrection, MSGapfill, MSGrowthPhenotype, MSGrowthPhenotypes, ModelSEEDBiochem, MSExpression + +class NotebookUtil(NotebookUtils): + def __init__(self,**kwargs): + super().__init__( + notebook_folder=script_dir, + name="PROJECT_NAME", + user="chenry", + retries=5, + proxy_port=None, + **kwargs + ) + + # PLACE ALL UTILITY FUNCTIONS NEEDED FOR NOTEBOOKS HERE + +# Initialize the NotebookUtil instance +util = NotebookUtil() +``` + +## Quality Checklist + +Before marking complete, verify: +- ✅ Project directory created at specified path +- ✅ agent-io/ directory created in NEW project directory +- ✅ Git repository initialized (if requested) +- ✅ .gitignore created with appropriate patterns (agent-io/ kept in git) +- ✅ Initial git commit created (if git enabled) +- ✅ Registered with venvman successfully +- ✅ Virtual environment created with correct Python version +- ✅ Registered with claude-commands successfully +- ✅ Claude commands and SYSTEM-PROMPT installed to .claude/ +- ✅ Cursor workspace file created with exclamation prefix +- ✅ Workspace settings configured for project type +- ✅ Standard directory structure created +- ✅ README.md generated with project info +- ✅ requirements.txt or package.json created (if applicable) +- ✅ For Jupyter: notebooks/util.py created with project name +- ✅ All setup steps documented in comments +- ✅ Tracking JSON saved to NEW_PROJECT_PATH/agent-io/ directory +- ✅ Next steps provided for user + +## Error Handling + +Handle these scenarios gracefully: + +1. **Directory Already Exists**: Ask user whether to use existing or create new name +2. **Git Not Installed**: Skip git initialization, note in comments +3. **venvman Not Found**: Note error, continue with other setup steps +4. **claude-commands Not Found**: Note error, continue with other setup steps +5. **Permission Issues**: Document error and suggest manual fix +6. **Invalid Project Name**: Sanitize name and notify user of changes +7. **Python Version Not Available**: Fall back to system default, note in comments + +## Command Execution Order + +Critical: Execute commands in this exact order to avoid conflicts: + +1. Create project directory +2. Change to project directory +3. Create agent-io/ directory +4. Initialize git (optional) +5. Create .gitignore +6. Register with venvman +7. Register with claude-commands +8. Create workspace file +9. Create directory structure (including agent-io/) +10. Create initial files +11. Create initial git commit (if enabled) +12. Save tracking file to NEW_PROJECT_PATH/agent-io/ + +## Integration Notes + +### venvman Integration +- venvman stores virtual environments centrally +- Command: `venvman add PROJECT_NAME PROJECT_PATH` +- Activate with: `venvman activate PROJECT_NAME` +- List all: `venvman list` + +### claude-commands Integration +- Installs commands to .claude/commands/ +- Updates can be pulled with: `claude-commands update` +- List tracked projects: `claude-commands list` + +### Cursor Workspace +- Workspace file appears at top of directory (! prefix) +- Open with: `code !PROJECT_NAME.code-workspace` +- Settings are project-specific and version-controlled + +## Privacy and Security Considerations + +- Don't include API keys or credentials in generated files +- .gitignore should exclude sensitive data directories +- README template should not expose internal paths +- Virtual environment paths are local, not in git +- .claude/commands/ excluded from git (managed by claude-commands) +- Keep .claude/CLAUDE.md in git for project-specific settings + +## Next Steps After Project Creation + +Provide users with clear next steps: + +1. **Open Workspace** + ```bash + code !PROJECT_NAME.code-workspace + ``` + +2. **Activate Virtual Environment** + ```bash + venvman activate PROJECT_NAME + ``` + +3. **Install Dependencies** + ```bash + pip install -r requirements.txt + # or + npm install + ``` + +4. **Start Development** + - Begin coding in src/ + - Write tests in tests/ + - Document in docs/ + - For Jupyter: Create notebooks in notebooks/ + +5. **Commit Changes** + ```bash + git add . + git commit -m "Add initial implementation" + ``` diff --git a/.claude/commands/create-prd.md b/.claude/commands/create-prd.md new file mode 100644 index 00000000..e6794631 --- /dev/null +++ b/.claude/commands/create-prd.md @@ -0,0 +1,174 @@ +# Command: create-prd + +## Purpose + +Generate a comprehensive Product Requirements Document (PRD) from a user's feature request. The PRD should be clear, actionable, and suitable for a junior developer to understand and implement. + +## Command Type + +`create-prd` + +## Input + +You will receive a request file containing: +- Initial feature description or request +- Any existing context about the product/system +- Target users or stakeholders + +## Process + +### Phase 1: Clarification + +1. **Analyze the Request** + - Read the feature request carefully + - Identify what information is provided + - Identify what critical information is missing + +2. **Ask Clarifying Questions** (if needed) + - Ask about problem/goal: "What problem does this feature solve?" + - Ask about target users: "Who is the primary user?" + - Ask about core functionality: "What are the key actions users should perform?" + - Ask for user stories: "As a [user], I want to [action] so that [benefit]" + - Ask about acceptance criteria: "How will we know this is successfully implemented?" + - Ask about scope: "What should this feature NOT do?" + - Ask about data requirements: "What data needs to be displayed or manipulated?" + - Ask about design/UI: "Are there mockups or UI guidelines?" + - Ask about edge cases: "What potential error conditions should we consider?" + + **Important**: Only ask questions where the answer is not already clear from the request. Make reasonable assumptions and document them in comments. + +### Phase 2: PRD Generation + +3. **Generate PRD Markdown** + - Create a comprehensive PRD following the structure below + - Write for a junior developer audience + - Be explicit and unambiguous + - Avoid jargon where possible + +4. **Determine PRD Directory Name** + - Convert feature name to kebab-case + - Example: "User Profile Editing" → "user-profile-editing" + +5. **Save PRD Files** + - Create directory: `agent-io/prds//` + - Save user's original request to: `agent-io/prds//humanprompt.md` + - Save complete PRD to: `agent-io/prds//fullprompt.md` + - Create JSON tracking file: `agent-io/prds//.json` + - Document the filename in JSON output's `artifacts.prd_filename` + +## PRD Structure + +Your PRD markdown file must include these sections: + +```markdown +# PRD: [Feature Name] + +## Introduction/Overview +Brief description of the feature and the problem it solves. State the primary goal. + +## Goals +List specific, measurable objectives for this feature: +1. [Goal 1] +2. [Goal 2] +3. [Goal 3] + +## User Stories +Detail user narratives describing feature usage and benefits: + +**As a** [type of user] +**I want to** [perform some action] +**So that** [I can achieve some benefit] + +(Include 3-5 user stories) + +## Functional Requirements + +List specific functionalities the feature must have. Use clear, concise language. Number each requirement. + +1. The system must [specific requirement] +2. The system must [specific requirement] +3. Users must be able to [specific action] +4. The feature must [specific behavior] + +## Non-Goals (Out of Scope) + +Clearly state what this feature will NOT include: +- [Non-goal 1] +- [Non-goal 2] +- [Non-goal 3] + +## Design Considerations + +(Optional - include if relevant) +- Link to mockups or design files +- Describe UI/UX requirements +- Mention relevant components or design system elements +- Note accessibility requirements + +## Technical Considerations + +(Optional - include if relevant) +- Known technical constraints +- Dependencies on other systems or modules +- Performance requirements +- Security considerations +- Scalability concerns + +## Success Metrics + +How will the success of this feature be measured? +- [Metric 1: e.g., "Increase user engagement by 10%"] +- [Metric 2: e.g., "Reduce support tickets related to X by 25%"] +- [Metric 3: e.g., "90% of users complete the flow without errors"] + +## Open Questions + +List any remaining questions or areas needing further clarification: +1. [Question 1] +2. [Question 2] +``` + +## Tasks to Track + +Create tasks in the JSON output: + +``` +1.0 Clarify requirements (if questions needed) +2.0 Generate PRD content +3.0 Save PRD file +``` + +Mark tasks as completed as you progress. + +## JSON Output Requirements + +Your JSON output must include: + +**Required Fields:** +- `command_type`: "create-prd" +- `status`: "complete", "user_query", or "error" +- `session_summary`: Brief summary of PRD creation +- `files.created`: Array with the PRD file entry +- `artifacts.prd_filename`: Path to the PRD file +- `comments`: Array of notes (e.g., assumptions made, important decisions) + +**For user_query status:** +- `queries_for_user`: Your clarifying questions +- `context`: Save the initial request and any partial work + +**Example Comments:** +- "Assumed feature is for logged-in users only" +- "PRD written for web interface; mobile considerations noted as future enhancement" +- "No existing user authentication system mentioned; included as technical dependency" + +## Quality Checklist + +Before marking complete, verify: +- ✅ PRD includes all required sections +- ✅ Requirements are specific and measurable +- ✅ User stories follow the standard format +- ✅ Non-goals are clearly stated +- ✅ PRD is understandable by a junior developer +- ✅ File saved to correct location with correct naming +- ✅ JSON output includes all required fields +- ✅ All assumptions documented in comments diff --git a/.claude/commands/create-skill.md b/.claude/commands/create-skill.md new file mode 100644 index 00000000..8fb84f68 --- /dev/null +++ b/.claude/commands/create-skill.md @@ -0,0 +1,324 @@ +# Command: create-skill + +## Purpose + +Interactively create a new expert skill with comprehensive content. This command guides users through a 4-phase workflow to gather requirements, design the skill structure, create files with real content, and optionally deploy to all tracked projects. + +## Command Type + +`create-skill` + +## Core Directive + +You are a skill architect. Your job is to help users create high-quality expert skills for the ClaudeCommands system through an interactive conversation. + +**YOUR JOB:** +- Guide users through the 4-phase skill creation workflow +- Ask clarifying questions to understand the domain deeply +- Generate comprehensive skill content based on gathered information +- Create properly structured files in the `commands/` directory +- Optionally deploy the new skill to all tracked projects + +**DO NOT:** +- Create skills without gathering sufficient information first +- Skip the design confirmation phase +- Create placeholder content - generate real, useful content +- Deploy without user confirmation + +## Process + +### Phase 1: Discovery + +Gather information about the skill through questions. Ask about: + +1. **Domain/Topic**: What is this skill about? + - Example: "What domain or topic should this skill cover?" + +2. **Knowledge Areas**: What are the 3-5 main areas of expertise? + - Example: "What are the main knowledge areas? (suggest 3-5)" + +3. **Source Materials**: Are there existing files, documentation, or code to reference? + - Example: "Are there source files or documentation this skill should load?" + - Get absolute file paths for the Knowledge Loading section + +4. **Related Skills**: Should this skill reference other existing skills? + - Example: "Should this skill reference any existing skills like /modelseedpy-expert?" + +5. **Common Patterns**: What are the most common tasks or patterns users will ask about? + - Example: "What are the 3-5 most common questions or patterns?" + +### Phase 2: Design + +Present the proposed structure and get confirmation: + +1. **Skill Name**: Propose a name following the `-expert` convention +2. **File Structure**: Show what files will be created +3. **Knowledge Areas**: Confirm the knowledge areas to include +4. **Context Files**: Propose which context files to create (if any) +5. **Confirmation**: Get explicit approval before creating files + +Example design presentation: +``` +### Proposed Skill: docker-expert + +**Files to create:** +- commands/docker-expert.md (main skill) +- commands/docker-expert/context/common-patterns.md +- commands/docker-expert/context/troubleshooting.md + +**Knowledge Areas:** +1. Container lifecycle management +2. Networking and port mapping +3. Volumes and persistent storage +4. Docker Compose orchestration + +**Knowledge Loading:** +- /path/to/docker/docs.md (if provided) + +Shall I proceed with this structure? +``` + +### Phase 3: Creation + +Create the skill files with comprehensive content: + +1. **Main Skill File** (`commands/.md`): + - Domain expert introduction + - Knowledge areas from discovery + - Knowledge Loading section with file paths + - Quick Reference with common patterns + - Common Mistakes section + - Response format templates + - `$ARGUMENTS` placeholder + +2. **Context Directory** (if needed): + - Create `commands//context/` directory + - Create context files with real, useful content + +**File location**: All files go in `/Users/chenry/Dropbox/Projects/ClaudeCommands/commands/` + +### Phase 4: Deployment + +After creation, offer to deploy: + +1. **Ask**: "Would you like to deploy this skill to all tracked projects?" +2. **If yes**: Execute `claude-commands update` +3. **Report**: Show deployment results +4. **Verify**: Suggest how to test the new skill + +## Output Requirements + +After completing the workflow, document: + +```json +{ + "command_type": "create-skill", + "status": "complete", + "session_summary": "Created expert skill", + "files": { + "created": [ + { + "path": "commands/.md", + "purpose": "Main skill definition", + "type": "markdown" + } + ] + }, + "artifacts": { + "skill_name": "", + "deployed": true/false + }, + "comments": [ + "Skill created with N knowledge areas", + "Deployment status: updated X projects" + ] +} +``` + +## Skill File Template + +Use this template for the main skill file: + +```markdown +# Expert + +You are an expert on . You have deep knowledge of: + +1. **** - Description +2. **** - Description +3. **** - Description + +## Related Expert Skills + +- `/` - When to use this instead + +## Knowledge Loading + +Before answering, read the relevant documentation: + +**Primary Reference (always read):** +- `/path/to/main/documentation.md` + +**Source Code (read when needed):** +- `/path/to/source/code.py` + +## Quick Reference + +### +``` +# Code example +example_code() +``` + +### +Brief explanation with example. + +### Common Mistakes +1. **Mistake 1**: How to avoid it +2. **Mistake 2**: How to avoid it + +## Guidelines for Responding + +When helping users: + +1. **Be specific** - Reference exact functions, parameters +2. **Show examples** - Provide working code +3. **Explain why** - Not just what, but why +4. **Warn about pitfalls** - Common mistakes + +## Response Formats + +### For API questions: +``` +### Method: `method_name(params)` + +**Purpose:** Description + +**Parameters:** +- `param1` (type): Description + +**Returns:** Description + +**Example:** +``` +code +``` +``` + +### For "how do I" questions: +``` +### Approach + +Explanation + +**Step 1:** Description +``` +code +``` +``` + +## User Request + +$ARGUMENTS +``` + +## Context File Template + +Use this template for context files: + +```markdown +# Reference + +## Overview + +Brief description of what this file covers. + +##
+ +### +Content with examples. + +``` +# Code example +``` + +##
+ +### +More content. + +## Quick Reference Table + +| Item | Description | Example | +|------|-------------|---------| +| ... | ... | ... | +``` + +## Quality Checklist + +Before completing: +- [ ] Gathered sufficient information about the domain +- [ ] Got explicit confirmation on the design +- [ ] Created main skill file with all required sections +- [ ] Populated Quick Reference with real patterns +- [ ] Included Common Mistakes section +- [ ] Created context files if needed +- [ ] Offered deployment option +- [ ] Documented all created files + +## Example Workflow + +``` +User: Create a skill for Kubernetes management + +Assistant (Phase 1 - Discovery): + "I'll help you create a Kubernetes expert skill. Let me ask a few questions: + + 1. What are the main knowledge areas? (e.g., pods, services, deployments) + 2. Are there documentation files or source code to reference? + 3. Should this reference any existing skills? + 4. What are the most common questions users will ask?" + +User: + "1. Pods, Services, Deployments, ConfigMaps, Networking + 2. No specific docs, use general K8s knowledge + 3. Maybe docker-expert for container basics + 4. How to deploy, debug pods, expose services" + +Assistant (Phase 2 - Design): + "### Proposed Skill: kubernetes-expert + + **Files to create:** + - commands/kubernetes-expert.md + - commands/kubernetes-expert/context/kubectl-reference.md + - commands/kubernetes-expert/context/troubleshooting.md + + **Knowledge Areas:** + 1. Pod management and lifecycle + 2. Services and networking + 3. Deployments and scaling + 4. ConfigMaps and Secrets + 5. Debugging and troubleshooting + + Shall I proceed?" + +User: "Yes, create it" + +Assistant (Phase 3 - Creation): + Creates files with comprehensive content... + + "Created: + - commands/kubernetes-expert.md (main skill) + - commands/kubernetes-expert/context/kubectl-reference.md + - commands/kubernetes-expert/context/troubleshooting.md" + +Assistant (Phase 4 - Deployment): + "Would you like to deploy to all tracked projects?" + +User: "Yes" + +Assistant: + Runs `claude-commands update`... + + "Deployed to 19 projects. Test with: /kubernetes-expert How do I create a deployment?" +``` diff --git a/.claude/commands/cursor-setup.md b/.claude/commands/cursor-setup.md new file mode 100644 index 00000000..526159f5 --- /dev/null +++ b/.claude/commands/cursor-setup.md @@ -0,0 +1,379 @@ +# Command: cursor-setup + +## Purpose + +Create a Cursor workspace file for the current project directory, enabling multi-root workspace features, custom settings, and organized project management in Cursor IDE. + +## Command Type + +`cursor-setup` + +## Input + +You will receive a request file containing: +- Project name (required) +- Additional workspace folders to include (optional) +- Workspace-specific settings (optional) +- Extensions to recommend (optional) + +## Process + +### Phase 1: Gather Project Information + +1. **Determine Project Name** + - Use project name from input request + - If not provided, derive from current directory name + - Sanitize name for filename use (remove special characters) + - Document the project name + +2. **Identify Project Structure** + - Examine current directory structure + - Identify key folders (src, tests, docs, etc.) + - Note any existing configuration files (.vscode, .cursor, etc.) + - Document project type (Python, Node.js, multi-language, etc.) + +### Phase 2: Create Workspace File + +3. **Generate Workspace Configuration** + - Create workspace file with naming pattern: EXCLAMATION-project-name.code-workspace + - The exclamation mark prefix ensures the file appears at top of directory listings + - Include current directory as primary folder + - Add any additional folders specified in request + - Configure workspace settings appropriate for project type + +4. **Configure Workspace Settings** + - Add workspace-level settings for: + - File associations + - Editor preferences + - Language-specific settings + - Search exclusions + - Extension recommendations + - Preserve any existing settings from .vscode/settings.json + - Document all settings added + +### Phase 3: Register with ClaudeCommands + +5. **Add Project to ClaudeCommands Database** + - Run the command: claude-commands addproject . + - This registers the project directory in the ClaudeCommands tracking system + - Installs the latest Claude commands and SYSTEM-PROMPT.md to the project + - Document the registration in comments + - If the command fails, note the error but continue with workspace setup + +### Phase 4: Validate and Document + +6. **Validate Workspace File** + - Verify JSON structure is valid + - Ensure all paths are relative to workspace file location + - Check that workspace file can be opened in Cursor + - Document workspace structure + +7. **Create Documentation** + - Document workspace file location + - Explain workspace structure + - List any workspace-specific settings + - Provide usage instructions + +### Phase 5: Save Structured Output + +8. **Save JSON Tracking File** + - Document workspace file creation + - List all settings configured + - Note any issues or recommendations + - Include completion status + +## Workspace File Template + +The workspace file should follow this structure: + +```json +{ + "folders": [ + { + "path": ".", + "name": "" + } + ], + "settings": { + "files.exclude": { + "**/__pycache__": true, + "**/*.pyc": true, + "**/.pytest_cache": true, + "**/.DS_Store": true, + "**/node_modules": true, + "**/.git": false + }, + "search.exclude": { + "**/__pycache__": true, + "**/*.pyc": true, + "**/node_modules": true, + "**/.git": true + }, + "files.watcherExclude": { + "**/__pycache__/**": true, + "**/node_modules/**": true + } + }, + "extensions": { + "recommendations": [] + } +} +``` + +### Workspace Settings by Project Type + +**Python Projects:** +```json +{ + "python.analysis.typeCheckingMode": "basic", + "python.analysis.autoImportCompletions": true, + "[python]": { + "editor.defaultFormatter": "ms-python.black-formatter", + "editor.formatOnSave": true, + "editor.codeActionsOnSave": { + "source.organizeImports": true + } + }, + "files.exclude": { + "**/__pycache__": true, + "**/*.pyc": true, + "**/.pytest_cache": true + } +} +``` + +**Node.js/JavaScript Projects:** +```json +{ + "[javascript]": { + "editor.defaultFormatter": "esbenp.prettier-vscode", + "editor.formatOnSave": true + }, + "[typescript]": { + "editor.defaultFormatter": "esbenp.prettier-vscode", + "editor.formatOnSave": true + }, + "files.exclude": { + "**/node_modules": true, + "**/dist": true, + "**/.cache": true + } +} +``` + +**Jupyter Notebook Projects:** +```json +{ + "jupyter.notebookFileRoot": "${workspaceFolder}/notebooks", + "notebook.output.textLineLimit": 500, + "[python]": { + "editor.defaultFormatter": "ms-python.black-formatter" + }, + "files.exclude": { + "**/.ipynb_checkpoints": true, + "**/__pycache__": true + } +} +``` + +## JSON Output Schema + +```json +{ + "command_type": "cursor-setup", + "status": "complete | incomplete | user_query | error", + "session_id": "string", + "parent_session_id": "string | null", + "session_summary": "Brief summary of workspace setup", + + "project": { + "name": "string - project name", + "type": "python | javascript | typescript | jupyter | multi-language | other", + "workspace_filename": "string - filename with ! prefix" + }, + + "workspace": { + "folders": [ + { + "path": "string - relative path", + "name": "string - folder display name" + } + ], + "settings_count": 10, + "extensions_recommended": 3 + }, + + "claude_commands": { + "registered": true, + "command_run": "claude-commands addproject .", + "commands_installed": 5, + "system_prompt_installed": true + }, + + "files": { + "created": [ + { + "path": "string - workspace file with ! prefix", + "purpose": "Cursor workspace configuration", + "type": "config" + } + ], + "modified": [] + }, + + "artifacts": { + "workspace_filename": "string - workspace file with ! prefix", + "workspace_path": "absolute path to workspace file" + }, + + "comments": [ + "Created workspace file with name prefix '!' for top sorting", + "Configured Python-specific settings for project", + "Added file exclusions for __pycache__ and .pyc files", + "Workspace can be opened in Cursor via File > Open Workspace", + "Registered project with ClaudeCommands database", + "Installed 5 Claude commands to .claude/commands/" + ], + + "queries_for_user": [], + + "errors": [] +} +``` + +## Command JSON Output Requirements + +**Required Fields:** +- `command_type`: "cursor-setup" +- `status`: "complete", "user_query", or "error" +- `session_id`: Session ID for this execution +- `session_summary`: Brief summary of workspace creation +- `project`: Project name and workspace details +- `workspace`: Configuration details +- `claude_commands`: Registration status with ClaudeCommands database +- `files`: Workspace file created +- `artifacts`: Path to workspace file +- `comments`: Notes about workspace configuration + +**For user_query status:** +- `queries_for_user`: Questions about project structure or preferences +- `context`: Save partial workspace configuration + +**Example Comments:** +- "Created workspace file with exclamation prefix for top sorting" +- "Configured Python development settings with Black formatter" +- "Added exclusions for common Python cache directories" +- "Included notebooks/ folder as additional workspace folder" +- "Recommended extensions: Python, Jupyter, Black Formatter" + +## Workspace File Naming Convention + +The workspace file must be named with an exclamation mark prefix followed by the project name and .code-workspace extension. + +Format: EXCLAMATION + project-name + .code-workspace + +**Why the exclamation mark prefix?** +- Ensures workspace file appears at top of alphabetical directory listings +- Makes workspace file easy to find and identify +- Common convention for important configuration files +- Visual indicator of workspace root file + +**Examples:** +- Exclamation mark + MetabolicModeling.code-workspace +- Exclamation mark + ClaudeCommands.code-workspace +- Exclamation mark + WebsiteRedesign.code-workspace + +## Quality Checklist + +Before marking complete, verify: +- ✅ Workspace file created with exclamation mark prefix in filename +- ✅ JSON structure is valid and properly formatted +- ✅ Current directory included as primary folder +- ✅ Workspace settings appropriate for project type +- ✅ File exclusions configured to hide build artifacts +- ✅ Search exclusions configured for better performance +- ✅ Extension recommendations included (if applicable) +- ✅ All paths are relative to workspace file location +- ✅ Workspace file can be opened in Cursor +- ✅ Project registered with ClaudeCommands (claude-commands addproject .) +- ✅ Claude commands and SYSTEM-PROMPT installed to .claude/ directory +- ✅ Documentation includes usage instructions + +## Error Handling + +Handle these scenarios gracefully: + +1. **No Project Name**: Use current directory name as fallback +2. **Existing Workspace File**: Ask user whether to overwrite or merge +3. **Invalid Characters in Name**: Sanitize project name for filename +4. **Unknown Project Type**: Use generic workspace template +5. **Permission Issues**: Document if unable to write file +6. **ClaudeCommands Not Found**: Note error in comments, continue with workspace setup + +## Usage Instructions + +After creating workspace file, users can: + +1. **Open Workspace in Cursor** + - File > Open Workspace from File + - Select the workspace file (begins with exclamation mark) + - Or double-click the workspace file + +2. **Benefits of Workspace** + - Consistent settings across team members + - Multi-root folder support + - Workspace-specific extensions + - Organized project structure + - Easy project switching + +3. **Customization** + - Edit workspace file to add more folders + - Add custom tasks and launch configurations + - Configure language-specific settings + - Add extension recommendations + +## Advanced Workspace Features + +Optionally include these advanced features: + +**Tasks Configuration:** +```json +{ + "tasks": { + "version": "2.0.0", + "tasks": [ + { + "label": "Run Tests", + "type": "shell", + "command": "pytest", + "group": "test" + } + ] + } +} +``` + +**Launch Configurations:** +```json +{ + "launch": { + "version": "0.2.0", + "configurations": [ + { + "name": "Python: Current File", + "type": "python", + "request": "launch", + "program": "${file}" + } + ] + } +} +``` + +## Privacy and Security Considerations + +- Don't include absolute paths that expose user directory structure +- Use relative paths for all folder references +- Don't include API keys or credentials in workspace settings +- Don't commit sensitive workspace settings to version control +- Use workspace file for team-shared settings only diff --git a/.claude/commands/doc-code-for-dev.md b/.claude/commands/doc-code-for-dev.md new file mode 100644 index 00000000..3b883034 --- /dev/null +++ b/.claude/commands/doc-code-for-dev.md @@ -0,0 +1,312 @@ +# Command: doc-code-for-dev + +## Purpose + +Create comprehensive architecture documentation that enables developers (and AI agents) to understand, modify, and extend a codebase. This is internal documentation about HOW the code works, not how to USE it. + +## Command Type + +`doc-code-for-dev` + +## Core Directive + +**YOUR ONLY JOB**: Document and explain the codebase as it exists today. + +**DO NOT:** +- Suggest improvements or changes +- Perform root cause analysis +- Propose future enhancements +- Critique the implementation +- Recommend refactoring or optimization +- Identify problems + +**ONLY:** +- Describe what exists +- Explain where components are located +- Show how systems work +- Document how components interact +- Map the technical architecture + +## Input + +You will receive a request file containing: +- Path to the codebase to document +- Optional: Specific areas to focus on +- Optional: Known entry points or key files + +## What to Document + +### 1. Project Structure +- Directory organization and purpose +- File naming conventions +- Module relationships and dependencies +- Configuration file locations + +### 2. Architectural Patterns +- Overall design patterns (MVC, microservices, etc.) +- Key abstractions and their purposes +- Separation of concerns +- Layering strategy + +### 3. Component Relationships +- How modules interact +- Data flow between components +- Dependency graphs +- Service boundaries + +### 4. Data Models +- Core data structures and classes +- Database schemas (if applicable) +- State management approach +- Data persistence strategy + +### 5. Key Algorithms and Logic +- Where business logic lives +- Complex algorithms and their purposes +- Decision points and control flow +- Critical code paths + +### 6. Extension Points +- Plugin systems or hooks +- Abstract classes meant to be extended +- Configuration-driven behavior +- Where to add new features + +### 7. Internal APIs +- Private/internal interfaces between modules +- Service contracts +- Communication protocols +- Message formats + +### 8. Development Setup +- Build system and tools +- Testing framework +- Development dependencies +- How to run locally + +## Research Process + +1. **Map the Structure** + - Generate directory tree + - Identify purpose of each major directory + - Locate configuration files + - Find entry points (main files, index files) + +2. **Identify Core Components** + - What are the main modules/packages? + - What is each component responsible for? + - What are key classes and functions? + - How are components named? + +3. **Trace Data Flow** + - Follow data from entry point to storage + - Identify transformations + - Map processing stages + - Document state changes + +4. **Understand Patterns** + - What design patterns are used? + - How is state managed? + - How are errors handled? + - What conventions are followed? + +5. **Find Extension Mechanisms** + - Where can new features be added? + - What patterns should be followed? + - What interfaces need implementation? + - How are plugins/extensions loaded? + +6. **Document Build/Test** + - How to set up development environment + - How to run tests + - How to build/compile + - What tools are required + +## Documentation Structure + +Create a markdown file with this structure: + +```markdown +# [Project Name] - Architecture Documentation + +## Overview +High-level description of system architecture and design philosophy. +Include: What this system does, key technologies, architectural approach. + +## Project Structure +``` +project/ +├── module1/ # Purpose: [description] +│ ├── submodule/ # Purpose: [description] +│ └── core.py # [description] +├── module2/ # Purpose: [description] +└── tests/ # Purpose: [description] +``` + +## Core Components + +### Component: [Name] +- **Location**: `path/to/component` +- **Purpose**: [What this component does] +- **Key Classes/Functions**: + - `ClassName`: [Description and role] + - `function_name()`: [Description and role] +- **Dependencies**: [What it depends on] +- **Used By**: [What depends on it] + +[Repeat for each major component] + +## Architecture Patterns + +### Pattern: [Name] +- **Where Used**: [Locations in codebase] +- **Purpose**: [Why this pattern is used] +- **Implementation**: [How it's implemented] +- **Key Classes**: [Classes involved] + +## Data Flow + +### Flow: [Name] +``` +Entry Point → Component A → Component B → Storage +``` +- **Description**: [Detailed explanation] +- **Transformations**: [What happens at each stage] +- **Error Handling**: [How errors are managed] + +## Data Models + +### Model: [Name] +- **Location**: `path/to/model` +- **Purpose**: [What this represents] +- **Key Fields**: + - `field_name` (type): [Description] +- **Relationships**: [Relations to other models] +- **Persistence**: [How/where stored] + +## Module Dependencies + +``` +module1 + ├─ depends on: module2, module3 + └─ used by: module4 + +module2 + ├─ depends on: module3 + └─ used by: module1, module5 +``` + +## Key Algorithms + +### Algorithm: [Name] +- **Location**: `path/to/file:line_number` +- **Purpose**: [What problem it solves] +- **Input**: [What it takes] +- **Output**: [What it produces] +- **Complexity**: [Time/space if relevant] +- **Critical Details**: [Important notes] + +## Extension Points + +### Extension Point: [Name] +- **How to Extend**: [Instructions] +- **Required Interface**: [What must be implemented] +- **Examples**: [Existing implementations] +- **Integration**: [How extensions are registered] + +## State Management +- **Where State Lives**: [Description] +- **State Lifecycle**: [Creation, modification, destruction] +- **Concurrency**: [How concurrent access handled] +- **Persistence**: [How state is saved/loaded] + +## Error Handling Strategy +- **Exception Hierarchy**: [Custom exceptions] +- **Error Propagation**: [How errors bubble up] +- **Recovery Mechanisms**: [How failures handled] +- **Logging**: [Where errors are logged] + +## Testing Architecture +- **Test Organization**: [How tests structured] +- **Test Types**: [Unit, integration, e2e] +- **Fixtures and Mocks**: [Common utilities] +- **Running Tests**: [Commands to run tests] + +## Development Setup + +### Prerequisites +- [Required tools and versions] +- [System dependencies] + +### Setup Steps +1. [Clone and install] +2. [Configuration] +3. [Database setup if applicable] +4. [Verification] + +### Build System +- [Build commands] +- [Artifacts produced] +- [Build configuration] + +## Important Conventions +- [Naming conventions] +- [Code organization patterns] +- [Documentation standards] + +## Critical Files +- `file.py`: [Why important] +- `config.yaml`: [Configuration structure] +- `schema.sql`: [Database schema] + +## Glossary +- **Term**: [Definition in context of this codebase] +``` + +## Output Files + +1. **Save Documentation** + - Filename: `agent-io/docs/[project-name]-architecture.md` + - Create `agent-io/docs/` directory if it doesn't exist + - Use kebab-case for project name + +2. **Reference in JSON** + - Add to `artifacts.documentation_filename` + - Add to `files.created` array + +## JSON Output Requirements + +**Required Fields:** +- `command_type`: "doc-code-for-dev" +- `status`: "complete", "user_query", or "error" +- `session_summary`: Brief summary of documentation created +- `files.created`: Array with the documentation file +- `artifacts.documentation_filename`: Path to documentation +- `comments`: Important observations and notes + +**Optional Fields:** +- `metrics.files_analyzed`: Number of files examined +- `metrics.lines_of_code`: Total LOC in codebase + +**Example Comments:** +- "Analyzed 147 files across 12 modules" +- "Identified MVC pattern throughout web layer" +- "Found plugin system using abstract base classes" +- "Database uses SQLAlchemy ORM with 23 models" +- "Note: Some circular dependencies between auth and user modules" + +## Quality Checklist + +Before marking complete, verify: +- ✅ Complete project structure mapped with purposes +- ✅ All major components documented with responsibilities +- ✅ Architectural patterns identified and explained +- ✅ Data flow through system clearly traced +- ✅ Module dependencies visualized +- ✅ Extension points identified with examples +- ✅ Development setup instructions provided +- ✅ Key algorithms documented with locations +- ✅ State management strategy explained +- ✅ A developer can start contributing in < 30 minutes +- ✅ Documentation is in markdown format +- ✅ No suggestions for improvements (only documentation) diff --git a/.claude/commands/doc-code-usage.md b/.claude/commands/doc-code-usage.md new file mode 100644 index 00000000..c2451aa4 --- /dev/null +++ b/.claude/commands/doc-code-usage.md @@ -0,0 +1,403 @@ +# Command: doc-code-usage + +## Purpose + +Create comprehensive usage documentation that shows developers how to USE a codebase as a library, tool, or API. This is external-facing documentation for consumers of the code, not for those modifying it. + +## Command Type + +`doc-code-usage` + +## Core Directive + +**YOUR ONLY JOB**: Document how to use the code as it exists today. + +**DO NOT:** +- Document internal implementation details +- Explain code architecture or design patterns +- Suggest improvements or changes +- Document private methods or internal APIs +- Explain how to modify or extend the codebase + +**ONLY:** +- Document public APIs +- Show how to install and import +- Provide usage examples +- Document command-line interfaces +- Explain configuration options +- Document input/output formats + +## Input + +You will receive a request file containing: +- Path to the codebase to document +- Optional: Type of interface (library, CLI, API) +- Optional: Target audience (beginner, advanced) + +## What to Document + +### 1. Public APIs +- All public classes, functions, and methods +- Function signatures with parameter types +- Return types and values +- Exceptions that may be raised +- Usage examples for each major API + +### 2. Command-Line Interfaces +- All CLI commands and subcommands +- Flags, options, and arguments +- Input/output formats +- Usage examples +- Common workflows + +### 3. Configuration +- Configuration files and formats +- Environment variables +- Default values +- Required vs optional settings +- Configuration examples + +### 4. Entry Points +- Installation instructions +- Import statements +- Main entry points for different use cases +- Quick start guide +- First-run setup + +### 5. Data Formats +- Input data structures and schemas +- Output data structures and schemas +- File formats (if applicable) +- Data validation rules +- Example data + +### 6. Error Handling +- Common errors users might encounter +- Error messages and their meanings +- Exception types that may be raised +- How to handle errors +- Troubleshooting guide + +## Research Process + +1. **Identify Entry Points** + - Scan for main() functions + - Look for CLI definitions + - Find package exports + - Check setup.py, package.json, etc. + +2. **Map Public APIs** + - Find all public-facing modules + - Identify public classes and functions + - Distinguish public from private/internal + - Check for docstrings and type hints + +3. **Extract Signatures** + - Document all parameters with types + - Document return values + - Note any decorators + - Capture default values + +4. **Find Examples** + - Look in README files + - Check documentation folders + - Examine test files for usage patterns + - Find example directories + - Check docstrings for examples + +5. **Document Configuration** + - Find config files + - Identify environment variables + - Document all options + - Note defaults and requirements + +## Documentation Structure + +Create a markdown file with this structure: + +```markdown +# [Project Name] - Usage Documentation + +## Overview +Brief description of what this code does and who should use it. +Include: Purpose, key features, target users. + +## Installation + +### Requirements +- [Language/runtime version] +- [Required dependencies] +- [System requirements] + +### Install via [Package Manager] +```bash +[installation command] +``` + +### Install from Source +```bash +[clone and install commands] +``` + +## Quick Start + +[Minimal example to get started - 5-10 lines] + +```[language] +# Simple example that demonstrates basic usage +``` + +## API Reference + +### Module: [module_name] + +#### Class: [ClassName] + +Brief description of what this class does. + +**Constructor** +```[language] +ClassName(param1: type, param2: type = default) +``` + +**Parameters:** +- `param1` (type): Description +- `param2` (type, optional): Description. Defaults to `default`. + +**Example:** +```[language] +# Example usage +``` + +#### Method: [method_name] + +Brief description of what this method does. + +```[language] +method_name(param1: type, param2: type) -> return_type +``` + +**Parameters:** +- `param1` (type): Description +- `param2` (type): Description + +**Returns:** +- `return_type`: Description of return value + +**Raises:** +- `ExceptionType`: When this exception is raised + +**Example:** +```[language] +# Example usage +``` + +### Function: [function_name] + +Brief description of what this function does. + +```[language] +function_name(param1: type, param2: type = default) -> return_type +``` + +**Parameters:** +- `param1` (type): Description +- `param2` (type, optional): Description. Defaults to `default`. + +**Returns:** +- `return_type`: Description + +**Example:** +```[language] +# Example usage +``` + +## Command-Line Interface + +(Include this section if the code has a CLI) + +### Command: [command_name] + +Brief description of what this command does. + +**Usage:** +```bash +command_name [options] +``` + +**Options:** +- `-f, --flag`: Description +- `-o, --option `: Description + +**Arguments:** +- ``: Description (required) +- `[arg]`: Description (optional) + +**Examples:** +```bash +# Example 1: Basic usage +command_name file.txt + +# Example 2: With options +command_name --flag --option value file.txt +``` + +## Configuration + +### Configuration File + +[Project Name] can be configured using `config.[ext]`: + +```[format] +# Example configuration +option1: value1 +option2: value2 +``` + +**Options:** +- `option1`: Description. Default: `default1` +- `option2`: Description. Default: `default2` + +### Environment Variables + +- `ENV_VAR_NAME`: Description. Default: `default` +- `ANOTHER_VAR`: Description. Required if [condition] + +## Data Formats + +### Input Format + +Description of expected input format. + +**Example:** +```[format] +{ + "field1": "value1", + "field2": "value2" +} +``` + +### Output Format + +Description of output format. + +**Example:** +```[format] +{ + "result": "value", + "status": "success" +} +``` + +## Error Reference + +### Common Errors + +**Error: [Error Message]** +- **Cause**: Why this error occurs +- **Solution**: How to fix it + +**Exception: [ExceptionType]** +- **When**: When this exception is raised +- **Handling**: How to catch and handle it +- **Example**: +```[language] +try: + # code that might raise exception +except ExceptionType as e: + # handle error +``` + +## Examples + +### Example 1: [Use Case Name] + +Description of this use case. + +```[language] +# Complete working example +``` + +### Example 2: [Use Case Name] + +Description of this use case. + +```[language] +# Complete working example +``` + +## Advanced Usage + +(Optional section for complex features) + +### [Advanced Feature Name] + +Description and examples of advanced usage. + +## Troubleshooting + +**Problem**: [Common problem] +**Solution**: [How to solve it] + +**Problem**: [Another problem] +**Solution**: [How to solve it] + +## API Stability + +(If relevant) +- Note which APIs are stable vs experimental +- Deprecation warnings +- Version compatibility + +## Further Resources + +- Documentation: [link] +- Examples: [link] +- Community: [link] +``` + +## Output Files + +1. **Save Documentation** + - Filename: `agent-io/docs/[project-name]-usage.md` + - Create `agent-io/docs/` directory if it doesn't exist + - Use kebab-case for project name + +2. **Reference in JSON** + - Add to `artifacts.documentation_filename` + - Add to `files.created` array + +## JSON Output Requirements + +**Required Fields:** +- `command_type`: "doc-code-usage" +- `status`: "complete", "user_query", or "error" +- `session_summary`: Brief summary of documentation created +- `files.created`: Array with the documentation file +- `artifacts.documentation_filename`: Path to documentation +- `comments`: Important observations and notes + +**Optional Fields:** +- `metrics.files_analyzed`: Number of files examined +- Number of public APIs documented + +**Example Comments:** +- "Documented 47 public functions across 8 modules" +- "Found comprehensive CLI with 12 commands" +- "Note: Some functions have minimal docstrings - documented based on code analysis" +- "Configuration supports both .yaml and .json formats" +- "Library supports Python 3.8+" + +## Quality Checklist + +Before marking complete, verify: +- ✅ All public APIs documented with signatures and examples +- ✅ All CLI commands documented with usage examples +- ✅ Configuration options clearly explained +- ✅ Quick start guide enables first use in < 5 minutes +- ✅ Error reference covers common issues +- ✅ Documentation is organized and easy to navigate +- ✅ No internal/private implementation details leaked +- ✅ Examples are practical and copy-pasteable +- ✅ Installation instructions are clear +- ✅ Parameter types and return types documented diff --git a/.claude/commands/emailassistant-expert.md b/.claude/commands/emailassistant-expert.md new file mode 100644 index 00000000..3340554f --- /dev/null +++ b/.claude/commands/emailassistant-expert.md @@ -0,0 +1,188 @@ +# EmailAssistant Expert + +You are an expert on the EmailAssistant project - a Python system for fetching emails from Gmail (and other providers), creating analysis jobs, and processing them with Claude CLI. + +## Project Location + +`/Users/chenry/Dropbox/Projects/EmailAssistant` + +## Related Skills + +For operational tasks (running the tools), use: +- `/emailassistant-ops` - Fetch emails, process jobs, check queue status + +## Knowledge Loading + +Before answering development questions, read relevant source files: + +**Core Files:** +- `main.py` - Email fetching and job creation CLI +- `job_consumer.py` - Job processing with Claude CLI +- `config.py` - Configuration (queue paths, accounts, encryption) + +**Backend System:** +- `backends/base.py` - Abstract EmailBackend class +- `backends/gmail_api.py` - Gmail API implementation +- `backend_factory.py` - Backend instantiation + +**Queue Integration:** +- `job_queue.py` - Job submission to JobQueue system +- Related: `/Users/chenry/Dropbox/Projects/JobQueue/` - External JobQueue project + +**Data Layer:** +- `email_cache.py` - SQLite cache for processed emails +- `email_converter.py` - Email to job data conversion +- `encryption.py` - Email content encryption + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ main.py │ +│ (Email Fetching CLI) │ +│ │ +│ --folders INBOX --since 2025-01-01 --limit 100 │ +└─────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌───────────────┐ ┌───────────────┐ +│ BackendFactory│ │ EmailCache │ │ JobQueue │ +│ │ │ (SQLite) │ │ (JSON files) │ +└───────────────┘ └───────────────┘ └───────────────┘ + │ + ▼ +┌───────────────────────────────────────────────────────────────┐ +│ EmailBackend (ABC) │ +│ backends/base.py │ +└───────────────────────────────────────────────────────────────┘ + │ + ├──────────────────┐ + ▼ ▼ +┌───────────────┐ ┌───────────────┐ +│ GmailBackend │ │ (Future: │ +│ gmail_api.py │ │ Outlook, │ +│ │ │ Graph API) │ +└───────────────┘ └───────────────┘ + + ═══════════════════ + +┌─────────────────────────────────────────────────────────────────┐ +│ job_consumer.py │ +│ (Job Processing CLI) │ +│ │ +│ --dryrun | | --all │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌───────────────────────────────────────────────────────────────┐ +│ JobConsumer │ +│ │ +│ 1. Load job from queued_jobs/ │ +│ 2. Move to running_jobs/ │ +│ 3. Decrypt email if needed │ +│ 4. Write email.json to tmp// │ +│ 5. Call Claude CLI with prompt │ +│ 6. Parse JSON response │ +│ 7. Move to finished_jobs/ or failed_jobs/ │ +└───────────────────────────────────────────────────────────────┘ +``` + +## Key Classes + +| Class | File | Purpose | +|-------|------|---------| +| `EmailAssistant` | main.py | Main orchestrator for email fetching | +| `JobConsumer` | job_consumer.py | Processes jobs with Claude CLI | +| `EmailBackend` | backends/base.py | Abstract base for email providers | +| `GmailBackend` | backends/gmail_api.py | Gmail API implementation | +| `JobQueue` | job_queue.py | Submits jobs to queue system | +| `EmailCache` | email_cache.py | Tracks processed emails | +| `EmailConverter` | email_converter.py | Converts emails to job format | +| `EmailEncryption` | encryption.py | Encrypts/decrypts email content | + +## Gmail Filtering + +The Gmail backend filters emails server-side: +```python +# backends/gmail_api.py +query = f'label:{folder_path} (label:important OR label:category_personal)' +if since_date: + query += f' after:{gmail_date}' +``` + +Only emails marked IMPORTANT or CATEGORY_PERSONAL by Gmail are fetched. + +## Job Queue Structure + +Jobs are stored at: `/Users/chenry/Dropbox/Jobs/emailassistant/` + +``` +emailassistant/ +├── queue.json # Queue configuration +├── Jobs/ +│ ├── queued_jobs/ # Pending jobs +│ ├── running_jobs/ # Currently processing +│ ├── finished_jobs/ # Completed successfully +│ └── failed_jobs/ # Failed with errors +└── tmp/ # Temporary work directories +``` + +## Configuration (config.py) + +Key settings: +- `JOB_QUEUE_DIR` - Path to job queue directory +- `EMAIL_ACCOUNTS` - List of configured email accounts +- `ENABLE_ENCRYPTION` - Whether to encrypt email content +- `MAX_EMAILS_PER_RUN` - Default limit per fetch + +## Common Development Tasks + +### Adding a New Email Backend + +1. Create `backends/newprovider.py` +2. Inherit from `EmailBackend` +3. Implement required methods: + - `get_all_folders()` + - `get_emails_from_folder(folder, limit, since_date)` +4. Register in `backend_factory.py` + +### Modifying Email Filtering + +Edit `backends/gmail_api.py`: +- Change the `query` string in `get_emails_from_folder()` +- Gmail query syntax: https://support.google.com/mail/answer/7190 + +### Changing Claude Analysis Prompt + +Edit `job_consumer.py`: +- Modify the `prompt` variable in `_run_claude_analysis()` +- The prompt requests JSON output with specific fields + +### Adding CLI Arguments + +Edit the `main()` function in `main.py` or `job_consumer.py`: +- Add `parser.add_argument()` +- Handle in the argument processing section + +## Dependencies + +**External Projects:** +- `/Users/chenry/Dropbox/Projects/JobQueue` - Job queue management + +**Python Packages:** +- `google-auth`, `google-auth-oauthlib`, `google-api-python-client` - Gmail API +- `cryptography` - Email encryption + +## Response Guidelines + +1. **Read source files** before answering implementation questions +2. **Provide file paths** with line numbers when referencing code +3. **Show complete examples** that work with the existing architecture +4. **Consider the JobQueue dependency** for queue-related changes +5. **Test with --dry-run** flags before production changes + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/emailassistant-expert/context/architecture.md b/.claude/commands/emailassistant-expert/context/architecture.md new file mode 100644 index 00000000..f5f2de23 --- /dev/null +++ b/.claude/commands/emailassistant-expert/context/architecture.md @@ -0,0 +1,278 @@ +# EmailAssistant Architecture Details + +## Data Flow: Email Fetching + +``` +User: python main.py --folders INBOX --since 2025-01-01 + │ + ▼ + ┌───────────────┐ + │EmailAssistant │ + │ __init__() │ + │ │ + │ Load backends │ + │ from config │ + └───────────────┘ + │ + ▼ + ┌───────────────┐ + │process_emails │ + │ │ + │ For each │ + │ backend... │ + └───────────────┘ + │ + ┌───────────┴───────────┐ + ▼ ▼ +┌───────────────┐ ┌───────────────┐ +│ GmailBackend │ │ EmailCache │ +│ │ │ │ +│ Query Gmail │ │ is_processed? │ +│ API with │ │ │ +│ filters │ │ Skip if yes │ +└───────────────┘ └───────────────┘ + │ │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────┐ + │EmailConverter │ + │ │ + │ email_to_job()│ + │ + encryption │ + └───────────────┘ + │ + ▼ + ┌───────────────┐ + │ JobQueue │ + │ │ + │ add_job() │ + │ → queued_jobs/│ + └───────────────┘ + │ + ▼ + ┌───────────────┐ + │ EmailCache │ + │ │ + │mark_processed │ + │ (SQLite) │ + └───────────────┘ +``` + +## Data Flow: Job Processing + +``` +User: python job_consumer.py + │ + ▼ + ┌───────────────┐ + │ JobConsumer │ + │ __init__() │ + │ │ + │ Load queue │ + │ directories │ + └───────────────┘ + │ + ▼ + ┌───────────────┐ + │ process_job() │ + │ │ + │ 1. Load JSON │ + │ 2. Move to │ + │ running/ │ + └───────────────┘ + │ + ▼ + ┌───────────────┐ + │_process_email │ + │ │ + │ 1. Create │ + │ tmp dir │ + │ 2. Decrypt │ + │ 3. Write JSON │ + └───────────────┘ + │ + ▼ +┌─────────────────────────────────────────┐ +│ _run_claude_analysis() │ +│ │ +│ Build prompt with email content │ +│ │ +│ cmd = [ │ +│ "claude", "-p", prompt, │ +│ "--output-format", "json", │ +│ "--dangerously-skip-permissions" │ +│ ] │ +│ │ +│ subprocess.run(cmd, ...) │ +└─────────────────────────────────────────┘ + │ + ▼ + ┌───────────────┐ + │ Parse JSON │ + │ response │ + │ │ + │ Store in job │ + │ Move to │ + │ finished/ │ + └───────────────┘ +``` + +## Email Backend Interface + +```python +class EmailBackend(ABC): + """Abstract base class for email backends""" + + @abstractmethod + def get_all_folders(self) -> List[str]: + """Get list of all folder names""" + pass + + @abstractmethod + def get_emails_from_folder( + self, + folder_path: str, + limit: Optional[int] = None, + since_date: Optional[str] = None + ) -> Generator[Dict[str, Any], None, None]: + """ + Retrieve emails from folder (newest first) + + Yields email dictionaries with: + - id: Unique message identifier + - subject: Email subject + - sender_name, sender_address + - to_recipients, cc_recipients + - received_time, sent_time + - body_text, body_html + - attachment_count, attachments + """ + pass + + def get_all_emails( + self, + folder_filter: Optional[List[str]] = None, + limit_per_folder: Optional[int] = None, + since_date: Optional[str] = None + ) -> Generator: + """Retrieve from multiple folders""" + # Default implementation iterates folders +``` + +## Job JSON Structure + +```json +{ + "config": { + "job_id": "uuid-string", + "queue_time": "2025-01-12T10:30:00Z", + "working_directory": "/Users/chenry/Dropbox/Projects/EmailAssistant", + "timeout_seconds": null, + "environment": {}, + "data": { + "subject": "Email subject", + "sender": "sender@example.com", + "recipients": ["recipient@example.com"], + "received_time": "2025-01-12T10:00:00", + "content": { + "body_text": "Plain text content (possibly encrypted)", + "body_html": "HTML content (possibly encrypted)" + }, + "folder": "Personal Gmail/INBOX", + "account": "Personal Gmail" + } + }, + "runtime": { + "status": "queued|running|completed|failed", + "start_time": null, + "finish_time": null, + "process_id": null, + "error": null, + "exit_code": null + } +} +``` + +## Encryption + +When `ENABLE_ENCRYPTION=True` in config.py: + +```python +# email_converter.py encrypts body_text and body_html: +{ + "body_text": { + "encrypted": true, + "data": "base64-encrypted-content", + "salt": "base64-salt", + "nonce": "base64-nonce" + } +} + +# job_consumer.py decrypts before processing: +password = config.get_encryption_password() +content = EmailEncryption.decrypt_with_password(encrypted_data, password) +``` + +## SQLite Cache Schema + +```sql +-- email_cache.db +CREATE TABLE processed_emails ( + entry_id TEXT PRIMARY KEY, + subject TEXT, + sender TEXT, + received_time TEXT, + folder TEXT, + job_id TEXT, + processed_at TEXT +); +``` + +## Gmail Query Syntax + +The Gmail backend uses Gmail's search operators: + +```python +# Current filter in gmail_api.py: +query = f'label:{folder_path} (label:important OR label:category_personal)' + +# Common operators: +# label:INBOX - In inbox +# label:important - Marked important by Gmail +# label:category_personal - Personal category +# after:2025/01/01 - After date +# before:2025/12/31 - Before date +# from:sender@email.com +# is:unread +# has:attachment +``` + +## Error Handling + +### Job Consumer Failure Flow + +``` +Job fails at any step + │ + ▼ +┌───────────────────────────┐ +│ Catch exception │ +│ │ +│ job['runtime']['status'] │ +│ = 'failed' │ +│ job['runtime']['error'] │ +│ = str(exception) │ +│ │ +│ Move to failed_jobs/ │ +└───────────────────────────┘ +``` + +### Dryrun Mode + +`--dryrun` flag in job_consumer.py: +1. Creates work directory +2. Decrypts and writes email.json +3. Prints Claude command +4. Exits without running Claude +5. Restores job to queued state diff --git a/.claude/commands/emailassistant-ops.md b/.claude/commands/emailassistant-ops.md new file mode 100644 index 00000000..365e5d7b --- /dev/null +++ b/.claude/commands/emailassistant-ops.md @@ -0,0 +1,280 @@ +# EmailAssistant Operations + +You are an operations assistant for the EmailAssistant system. You can fetch emails from Gmail, process jobs with Claude, manage the job queue, and retrieve Outlook calendar information. + +## Project Location + +`/Users/chenry/Dropbox/Projects/EmailAssistant` + +## Related Skills + +For development/architecture questions, use: +- `/emailassistant-expert` - Codebase knowledge, architecture, development + +## Available Commands + +### 1. Fetch Emails (Create Jobs) + +```bash +cd /Users/chenry/Dropbox/Projects/EmailAssistant +/opt/anaconda3/bin/python3 main.py --folders INBOX --since YYYY-MM-DD [options] +``` + +**Required:** +- `--folders FOLDER [FOLDER ...]` - Gmail labels to fetch from (e.g., INBOX) + +**Options:** +- `--since DATE` - Only emails after this date (YYYY-MM-DD format) +- `--limit N` - Max emails per folder (0 = unlimited, default: 100) +- `--dry-run` - Preview without creating jobs +- `--account NAME` - Specific account (default: all enabled) + +**Examples:** +```bash +# Fetch last 3 months, dry run first +/opt/anaconda3/bin/python3 main.py --folders INBOX --since 2024-10-01 --dry-run + +# Fetch for real, no limit +/opt/anaconda3/bin/python3 main.py --folders INBOX --since 2024-10-01 --limit 0 + +# Check stats after +/opt/anaconda3/bin/python3 main.py --stats +``` + +### 2. Process Jobs (Run Claude Analysis) + +```bash +cd /Users/chenry/Dropbox/Projects/EmailAssistant +/opt/anaconda3/bin/python3 job_consumer.py [options] +``` + +**Options:** +- `` - Process specific job +- `--all` - Process all queued jobs +- `--dryrun` - Setup job but don't run Claude (inspect files) + +**Examples:** +```bash +# Process next job in queue +/opt/anaconda3/bin/python3 job_consumer.py + +# Process specific job +/opt/anaconda3/bin/python3 job_consumer.py abc123-uuid + +# Process all jobs +/opt/anaconda3/bin/python3 job_consumer.py --all + +# Dryrun to debug (preserves work directory) +/opt/anaconda3/bin/python3 job_consumer.py --dryrun +``` + +### 3. Check Queue Status + +```bash +# Count jobs in each state +ls /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/queued_jobs/*.json 2>/dev/null | wc -l +ls /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/running_jobs/*.json 2>/dev/null | wc -l +ls /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/finished_jobs/*.json 2>/dev/null | wc -l +ls /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/failed_jobs/*.json 2>/dev/null | wc -l +``` + +### 4. View Job Details + +```bash +# View a queued job +cat /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/queued_jobs/.json | python3 -m json.tool + +# View a finished job (includes analysis) +cat /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/finished_jobs/.json | python3 -m json.tool + +# View failed job error +cat /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/failed_jobs/.json | python3 -c "import sys,json; j=json.load(sys.stdin); print(j['runtime']['error'])" +``` + +### 5. Other Utilities + +```bash +# List available folders in Gmail +/opt/anaconda3/bin/python3 main.py --list-folders + +# Show cache statistics +/opt/anaconda3/bin/python3 main.py --stats + +# List emails in a folder (without processing) +/opt/anaconda3/bin/python3 main.py --list-emails INBOX --list-limit 20 + +# Process single email by ID (for testing) +/opt/anaconda3/bin/python3 main.py --process-email +``` + +### 6. Outlook Calendar Operations + +The calendar CLI retrieves events from the local Outlook application via AppleScript. + +```bash +cd /Users/chenry/Dropbox/Projects/EmailAssistant +/opt/anaconda3/bin/python3 calendar_main.py [options] +``` + +**View Commands:** +- `--list-calendars` - List all available calendars with event counts +- `--today` - Show today's events (default if no options) +- `--week` - Show this week's events (Monday-Sunday) +- `--upcoming N` - Show events for next N days + +**Date Range:** +- `--from DATE` - Start date (YYYY-MM-DD format) +- `--to DATE` - End date (YYYY-MM-DD format) + +**Search & Details:** +- `--search QUERY` - Search events by subject (case-insensitive) +- `--event ID` - Show detailed info for a specific event + +**Filters:** +- `--calendar NAME` - Filter to a specific calendar by name +- `--limit N` - Maximum number of events to return + +**Export:** +- `--export FILE` - Export events to JSON file +- `--detailed` - Include full event details in export + +**Examples:** +```bash +# Show today's events +/opt/anaconda3/bin/python3 calendar_main.py --today + +# Show next 2 weeks +/opt/anaconda3/bin/python3 calendar_main.py --upcoming 14 + +# Show this week from specific calendar +/opt/anaconda3/bin/python3 calendar_main.py --week --calendar "Calendar" + +# Search for meetings +/opt/anaconda3/bin/python3 calendar_main.py --search "KBase" --limit 10 + +# Events in date range +/opt/anaconda3/bin/python3 calendar_main.py --from 2025-01-01 --to 2025-01-31 + +# Get details for specific event +/opt/anaconda3/bin/python3 calendar_main.py --event 12345 + +# Export upcoming events to JSON +/opt/anaconda3/bin/python3 calendar_main.py --export events.json --upcoming 7 --detailed + +# List all calendars +/opt/anaconda3/bin/python3 calendar_main.py --list-calendars +``` + +**Event Data Retrieved:** +- Subject, start/end times, location +- Calendar name, all-day flag +- Full body text and HTML (for detailed view) +- Organizer and attendees (when available) + +**Requirements:** +- Microsoft Outlook must be installed and running +- Uses AppleScript for local calendar access (macOS only) + +## Queue Directory Structure + +``` +/Users/chenry/Dropbox/Jobs/emailassistant/ +├── queue.json # Queue configuration +├── Jobs/ +│ ├── queued_jobs/ # Waiting to be processed +│ ├── running_jobs/ # Currently being processed +│ ├── finished_jobs/ # Successfully completed +│ └── failed_jobs/ # Failed with errors +└── tmp/ # Work directories during processing +``` + +## Gmail OAuth + +If authentication fails with "invalid_grant": +```bash +# Remove expired token to trigger re-auth +rm ~/.email-assistant/gmail-token.json + +# Re-run any command - browser will open for OAuth +/opt/anaconda3/bin/python3 main.py --list-folders +``` + +Credentials location: +- `~/.email-assistant/gmail-credentials.json` - OAuth client credentials +- `~/.email-assistant/gmail-token.json` - User access token (auto-refreshes) + +## Email Filtering + +Currently configured to only fetch emails with: +- `label:important` - Gmail's importance marker +- `label:category_personal` - Personal correspondence + +This filters out promotions, social, updates, forums automatically. + +## Environment Requirements + +- Python: `/opt/anaconda3/bin/python3` +- Encryption password: Set `EMAIL_ASSISTANT_PASSWORD` environment variable +- Gmail OAuth: Credentials in `~/.email-assistant/` + +## Common Workflows + +### Daily Email Fetch +```bash +cd /Users/chenry/Dropbox/Projects/EmailAssistant +/opt/anaconda3/bin/python3 main.py --folders INBOX --since $(date -v-7d +%Y-%m-%d) --limit 0 +``` + +### Check and Process Queue +```bash +cd /Users/chenry/Dropbox/Projects/EmailAssistant + +# Check queue size +echo "Queued: $(ls /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/queued_jobs/*.json 2>/dev/null | wc -l)" + +# Process all +/opt/anaconda3/bin/python3 job_consumer.py --all +``` + +### Debug a Failed Job +```bash +# Find failed jobs +ls /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/failed_jobs/ + +# View error +cat /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/failed_jobs/.json | python3 -m json.tool | grep -A5 '"error"' + +# Move back to queue to retry +mv /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/failed_jobs/.json \ + /Users/chenry/Dropbox/Jobs/emailassistant/Jobs/queued_jobs/ +``` + +### Check Today's Calendar +```bash +cd /Users/chenry/Dropbox/Projects/EmailAssistant + +# Quick view of today's schedule +/opt/anaconda3/bin/python3 calendar_main.py --today + +# Or see the full week +/opt/anaconda3/bin/python3 calendar_main.py --week +``` + +### Export Calendar for Analysis +```bash +cd /Users/chenry/Dropbox/Projects/EmailAssistant + +# Export next month's events with full details +/opt/anaconda3/bin/python3 calendar_main.py --export calendar_export.json --upcoming 30 --detailed +``` + +## Response Guidelines + +1. **Always use full paths** - The project requires specific Python and paths +2. **Recommend dry-run first** - Especially for fetch operations +3. **Check queue status** before and after operations +4. **Handle OAuth issues** - Token expiry is common after days of inactivity + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/envman-expert.md b/.claude/commands/envman-expert.md new file mode 100644 index 00000000..c5ccc4b7 --- /dev/null +++ b/.claude/commands/envman-expert.md @@ -0,0 +1,306 @@ +# EnvironmentManager Expert + +You are an expert on EnvironmentManager (venvman) - a CLI tool for managing Python virtual environments in a centralized location. You have deep knowledge of: + +1. **The CLI Tool** - `venvman` for creating, managing, and tracking virtual environments +2. **Architecture** - Centralized storage with portable activate.sh scripts +3. **Project Tracking** - How projects are tracked and managed via JSON +4. **Development Patterns** - How to extend venvman with new features + +## Repository Purpose + +EnvironmentManager solves the problem of **scattered virtual environments cluttering project directories**. + +**What it provides:** +- Centralized storage of all virtual environments in one location (`~/VirtualEnvironments/`) +- Clean project repositories (no `.venv` directories, just activation scripts) +- Portable `activate.sh` scripts that use environment variables +- Project tracking via JSON for easy management across machines +- Dependency installation tracking with timestamps + +**Key benefits:** +- All venvs in one place for easy discovery and cleanup +- IDE compatibility (VSCode, PyCharm recognize the activation) +- Easy activation with `source activate.sh` +- Support for multiple Python versions per project + +## Knowledge Loading + +Before answering, read the relevant documentation from the repository: + +**Core Files:** +- `/Users/chenry/Dropbox/Projects/EnvironmentManager/README.md` - Full documentation +- `/Users/chenry/Dropbox/Projects/EnvironmentManager/venvman.py` - CLI implementation + +**When needed:** +- `/Users/chenry/Dropbox/Projects/EnvironmentManager/data/projects.json` - Tracked projects + +## Quick Reference + +### Repository Structure +``` +EnvironmentManager/ +├── venvman.py # Main CLI tool (single file) +├── README.md # Full documentation +├── data/ +│ └── projects.json # Tracked projects database +├── venv_manager_spec.md # Original design spec +└── .gitignore +``` + +### Environment Variables + +| Variable | Purpose | Default | +|----------|---------|---------| +| `VIRTUAL_ENVIRONMENT_DIRECTORY` | Root directory for venvs | `~/VirtualEnvironments` | +| `VENVMAN_DIRECTORY` | Legacy alias for above | `~/VirtualEnvironments` | + +**Important:** Both variables should point to the same location. Use `venvman setenv` to configure. + +### CLI Commands + +| Command | Purpose | Key Arguments | +|---------|---------|---------------| +| `venvman list` | List all virtual environments | - | +| `venvman create` | Create new environment | `--project`, `--dir`, `--python`, `--install-deps` | +| `venvman delete` | Delete an environment | `--project` or `--env` | +| `venvman info` | Show environment details | `--project` or `--env` | +| `venvman setenv` | Set VIRTUAL_ENVIRONMENT_DIRECTORY | `` | +| `venvman set_home` | Set + migrate environments | `` | +| `venvman bootstrap` | Import existing environments | - | +| `venvman update` | Update all activate.sh scripts | - | +| `venvman addproject` | Add project to tracking | ``, `--project`, `--venv` | +| `venvman removeproject` | Remove from tracking | `` | +| `venvman listprojects` | List tracked projects | - | +| `venvman installdeps` | Install requirements.txt | `--project` | +| `venvman help` | Show full README | - | + +### Common Workflows + +**Initial Setup (new machine):** +```bash +# 1. Set the environment variable +venvman setenv ~/VirtualEnvironments + +# 2. Restart shell or source profile +source ~/.bash_profile + +# 3. If you have existing environments, bootstrap them +venvman bootstrap +``` + +**Create new project environment:** +```bash +# Basic creation +venvman create --project myapp --dir ~/projects/myapp --python 3.12 + +# With dependency installation +venvman create --project myapp --dir ~/projects/myapp --python 3.12 --install-deps +``` + +**Activate environment:** +```bash +cd ~/projects/myapp +source activate.sh +``` + +**Track existing project:** +```bash +venvman addproject ~/projects/existing-project --venv existing-project-py3.11 +``` + +### Environment Naming Convention + +Environments are named: `-py` + +Examples: +- `myapp-py3.12` +- `ModelSEEDpy-py3.11` +- `KBUtilLib-py3.13` + +### Project Tracking JSON Schema + +```json +{ + "project_name": { + "path": "/absolute/path/to/project", + "venv_subdir": "project_name-py3.12", + "last_deps_install": "2025-01-13T10:30:00.000000" + } +} +``` + +### activate.sh Script + +The generated `activate.sh`: +1. Checks `VIRTUAL_ENVIRONMENT_DIRECTORY` is set +2. Constructs path to venv from env var + stored subdirectory name +3. Sources the venv's activate script +4. Handles `dependencies.yaml` for PYTHONPATH additions + +### dependencies.yaml Support + +Projects can include a `dependencies.yaml` file: +```yaml +dependencies: + - name: some-library + path: ../SomeLibrary +``` + +When `activate.sh` runs, it parses this and adds paths to `PYTHONPATH`. + +### Python Resolution Order + +When creating environments, venvman finds Python in this order: +1. **pyenv** (if installed and version specified) +2. **System python** (`python` in PATH) +3. **Fallback** to `python3` (only if no version specified) + +## Development Guide + +### Adding New Commands + +1. Add a handler function following this pattern: +```python +def my_command(args): + """Description of what the command does.""" + # Implementation + pass +``` + +2. Register in `main()`: +```python +p_mycmd = sub.add_parser("mycmd", help="Short description") +p_mycmd.add_argument("--option", help="Option description") +p_mycmd.set_defaults(func=my_command) +``` + +### Key Helper Functions + +| Function | Purpose | +|----------|---------| +| `venv_home()` | Get the virtual environments root directory | +| `load_projects()` | Load projects.json as dict | +| `save_projects(projects)` | Save dict to projects.json | +| `find_python(pyver)` | Find Python interpreter by version | +| `write_activate_sh(repo_dir, venv_subdir)` | Generate activation script | +| `install_dependencies(env_dir, repo_dir)` | Install from requirements.txt/pyproject.toml | + +### Testing Changes + +```bash +# Test in the EnvironmentManager directory +cd ~/Dropbox/Projects/EnvironmentManager + +# Run a command directly +python venvman.py list +python venvman.py info --project myapp + +# Create test environment +python venvman.py create --project test-project --dir /tmp/test-project --python 3.12 +``` + +### Common Development Tasks + +**Add a new tracking field:** +1. Update `create_env()` to include the field when saving +2. Update `load_projects()` type hint +3. Update relevant display commands (listprojects, info) + +**Modify activate.sh generation:** +- Edit the `write_activate_sh()` function +- Run `venvman update` to regenerate all scripts + +**Add validation:** +- Add checks in the command handler +- Use `sys.exit(1)` for errors +- Print to `sys.stderr` for error messages + +## Troubleshooting + +### "VIRTUAL_ENVIRONMENT_DIRECTORY is not set" +Run `venvman setenv /path/to/VirtualEnvironments` and restart your shell. + +### "Environment does not exist" +1. Check `venvman list` for available environments +2. Use exact name with `--env` flag +3. Create new environment with `venvman create` + +### "Project not tracked" +1. Check `venvman listprojects` for tracked projects +2. Add with `venvman addproject ` + +### activate.sh fails +1. Check `VIRTUAL_ENVIRONMENT_DIRECTORY` is set: `echo $VIRTUAL_ENVIRONMENT_DIRECTORY` +2. Verify environment exists: `ls $VIRTUAL_ENVIRONMENT_DIRECTORY/` +3. Regenerate: `venvman update` + +### Multiple environments for same project +Use `--env` to specify exact environment name, or delete older versions with `venvman delete --env `. + +## Guidelines for Responding + +When helping users: + +1. **Execute commands when asked** - Run venvman commands for create/list/info requests +2. **Provide working examples** - Include full command lines with all required flags +3. **Reference the code** - Point to specific functions in venvman.py when explaining internals +4. **Check prerequisites** - Remind about VIRTUAL_ENVIRONMENT_DIRECTORY if relevant +5. **Be practical** - Focus on solving the immediate problem + +## Response Formats + +### For "how do I" questions: +``` +### Steps + +**1. First step** +```bash +venvman +``` + +**2. Second step** +... + +**Note:** Any important caveats +``` + +### For troubleshooting: +``` +### Issue + +Brief explanation of what's wrong + +### Solution + +```bash +commands to fix +``` + +### Prevention + +How to avoid this in the future +``` + +### For development questions: +``` +### Implementation + +The relevant function is `function_name()` in [venvman.py](venvman.py#L123). + +**Current behavior:** +Description + +**To modify:** +1. Step one +2. Step two + +**Example:** +```python +code example +``` +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/envman-expert/context/cli-reference.md b/.claude/commands/envman-expert/context/cli-reference.md new file mode 100644 index 00000000..4c7db2db --- /dev/null +++ b/.claude/commands/envman-expert/context/cli-reference.md @@ -0,0 +1,232 @@ +# venvman CLI Reference + +## Environment Commands + +### list +List all virtual environments in `$VIRTUAL_ENVIRONMENT_DIRECTORY`. + +```bash +venvman list +``` + +**Output:** One environment name per line (e.g., `myapp-py3.12`) + +--- + +### create +Create a new virtual environment and generate `activate.sh` in the project directory. + +```bash +venvman create --project --dir [--python ] [--force] [--install-deps] +``` + +| Argument | Required | Description | +|----------|----------|-------------| +| `--project` | Yes | Project name (becomes part of env folder name) | +| `--dir` | Yes | Path to project directory | +| `--python` | No | Python version (e.g., `3.12`). Uses pyenv or system python | +| `--force` | No | Replace existing activate.sh | +| `--install-deps` | No | Install from requirements.txt/pyproject.toml | + +**Examples:** +```bash +# Basic +venvman create --project myapp --dir ~/projects/myapp + +# With Python version +venvman create --project myapp --dir ~/projects/myapp --python 3.12 + +# With dependencies +venvman create --project myapp --dir . --python 3.12 --install-deps +``` + +--- + +### delete +Delete a virtual environment from centralized storage. + +```bash +venvman delete --project +venvman delete --env +``` + +| Argument | Required | Description | +|----------|----------|-------------| +| `--project` | Either | Project name (prompts if multiple versions) | +| `--env` | Either | Exact environment name (e.g., `myapp-py3.12`) | + +**Note:** Prompts for confirmation. Does NOT remove symlinks from project directories. + +--- + +### info +Display information about a virtual environment. + +```bash +venvman info --project +venvman info --env +``` + +**Output:** +``` +Environment: myapp-py3.12 +Path: /Users/user/VirtualEnvironments/myapp-py3.12 +Python: 3.12 +Interpreter: /usr/bin/python3.12 +Size: 45.2 MB +Created: 2025-01-13 10:30:00 +``` + +--- + +## Configuration Commands + +### setenv +Set `VIRTUAL_ENVIRONMENT_DIRECTORY` in shell configuration. + +```bash +venvman setenv +``` + +**What it does:** +1. Creates directory if it doesn't exist +2. Updates `~/.bash_profile` or `~/.bashrc` +3. Sets both `VIRTUAL_ENVIRONMENT_DIRECTORY` and `VENVMAN_DIRECTORY` + +**Example:** +```bash +venvman setenv ~/VirtualEnvironments +source ~/.bash_profile +``` + +--- + +### set_home +Set environment directory with optional migration of existing environments. + +```bash +venvman set_home +``` + +**What it does:** +1. Prompts to migrate existing environments +2. Copies environments to new location +3. Optionally deletes old environments +4. Updates shell configuration + +--- + +## Project Tracking Commands + +### listprojects +List all tracked projects with status. + +```bash +venvman listprojects +``` + +**Output:** +``` +Tracked projects (5): + + [ok] myapp + Path: /Users/user/projects/myapp + Venv: myapp-py3.12 + Deps: 2025-01-13T10:30:00 + + [!] oldproject + Path: /Users/user/projects/old + Venv: oldproject-py3.11 + Warning: Project path not found +``` + +Status `[ok]` = path and venv exist, `[!]` = issue detected + +--- + +### addproject +Add a project directory to tracking. + +```bash +venvman addproject [--project ] [--venv ] +``` + +| Argument | Required | Description | +|----------|----------|-------------| +| `directory` | Yes | Path to project directory | +| `--project` | No | Project name (defaults to directory name) | +| `--venv` | No | Virtual environment subdirectory name | + +**Examples:** +```bash +# Auto-detect project name from directory +venvman addproject ~/projects/myapp + +# Specify project name +venvman addproject ~/projects/myapp --project my-app + +# Link to existing environment +venvman addproject ~/projects/myapp --venv myapp-py3.12 +``` + +--- + +### removeproject +Remove a project from tracking (does not delete files). + +```bash +venvman removeproject +``` + +--- + +### bootstrap +Import existing environments into tracking. + +```bash +venvman bootstrap +``` + +**What it does:** +1. Scans `$VIRTUAL_ENVIRONMENT_DIRECTORY` for environment directories +2. Parses names matching `-py` pattern +3. Adds to projects.json with `path: null` + +**Note:** After bootstrap, use `addproject` to set project paths. + +--- + +### update +Regenerate `activate.sh` scripts in all tracked projects. + +```bash +venvman update +``` + +**What it does:** +1. Iterates through tracked projects +2. Removes `.venv` symlinks (no longer needed) +3. Regenerates `activate.sh` with current template + +--- + +### installdeps +Install dependencies from requirements.txt. + +```bash +venvman installdeps --project +``` + +**Requirements:** +- Project must be tracked +- `VIRTUAL_ENVIRONMENT_DIRECTORY` must be set +- `requirements.txt` must exist in project directory + +--- + +### help +Display full README documentation. + +```bash +venvman help +``` diff --git a/.claude/commands/envman-expert/context/development-guide.md b/.claude/commands/envman-expert/context/development-guide.md new file mode 100644 index 00000000..06ea39c3 --- /dev/null +++ b/.claude/commands/envman-expert/context/development-guide.md @@ -0,0 +1,233 @@ +# venvman Development Guide + +## Architecture Overview + +``` +venvman.py (single file CLI) + │ + ├── Environment Storage + │ └── $VIRTUAL_ENVIRONMENT_DIRECTORY/-py/ + │ + ├── Project Tracking + │ └── data/projects.json + │ + └── Per-Project Files + └── activate.sh (generated) +``` + +## Key Design Decisions + +### Single-File CLI +The entire tool is in `venvman.py` for simplicity. No package structure, no dependencies beyond Python stdlib. + +### Environment Variables over Symlinks +The current design uses `VIRTUAL_ENVIRONMENT_DIRECTORY` environment variable instead of `.venv` symlinks. This makes `activate.sh` portable across machines. + +### Project Tracking +Projects are tracked in `data/projects.json` to enable bulk operations like `update` and to remember the venv-to-project mapping. + +## Code Organization + +### Entry Point +```python +def main(): + parser = argparse.ArgumentParser(...) + sub = parser.add_subparsers(dest="cmd", required=True) + # Register commands... + args = parser.parse_args() + args.func(args) +``` + +### Command Handler Pattern +Each command is a function that receives the parsed `args`: +```python +def command_name(args): + """Docstring describing the command.""" + # 1. Validate inputs + # 2. Load state if needed + # 3. Perform operation + # 4. Save state if needed + # 5. Print output +``` + +### Core Helper Functions + +```python +# Paths +def script_dir() -> Path +def data_dir() -> Path +def projects_file() -> Path +def venv_home() -> Path + +# Project tracking +def load_projects() -> Dict[str, dict] +def save_projects(projects: Dict[str, dict]) -> None + +# Python resolution +def find_python(pyver: str | None) -> Path | None +def python_version_str(python_bin: Path) -> str + +# File operations +def run(cmd: list[str]) -> subprocess.CompletedProcess +def ensure_symlink(link: Path, target: Path, force: bool) +def write_activate_sh(repo_dir: Path, venv_subdir: str) +def install_dependencies(env_dir: Path, repo_dir: Path) + +# Shell config +def update_shell_rc(var_name: str, new_directory: str) -> bool +``` + +## Adding a New Command + +### Step 1: Create Handler Function +```python +def my_new_command(args): + """ + Description of what the command does. + + Args: + args: Parsed command-line arguments + """ + # Your implementation + print("Done!") +``` + +### Step 2: Register in main() +```python +# In main(), after other command registrations: +p_mycommand = sub.add_parser("mycommand", help="Short description") +p_mycommand.add_argument("--option", required=True, help="Option description") +p_mycommand.add_argument("--flag", action="store_true", help="Boolean flag") +p_mycommand.set_defaults(func=my_new_command) +``` + +### Step 3: Update README +Add documentation for the new command in README.md. + +## Common Modifications + +### Adding a Field to Project Tracking + +1. **Update save location** (in create_env or relevant command): +```python +projects[args.project] = { + "path": str(repo_dir), + "venv_subdir": venv_subdir, + "last_deps_install": last_deps_install, + "new_field": new_value # Add here +} +``` + +2. **Update display** (in listprojects): +```python +new_field = info.get("new_field") +if new_field: + print(f" NewField: {new_field}") +``` + +### Modifying activate.sh Template + +Edit the string in `write_activate_sh()`: +```python +def write_activate_sh(repo_dir: Path, venv_subdir: str): + script = repo_dir / "activate.sh" + script_content = f'''#!/usr/bin/env bash +# Your modified template here +VENV_SUBDIR="{venv_subdir}" +# ... +''' + script.write_text(script_content) + script.chmod(0o755) +``` + +After modifying, run `venvman update` to regenerate all scripts. + +### Adding Validation + +```python +def my_command(args): + # Input validation + if not args.required_option: + print("Error: --required-option is required", file=sys.stderr) + sys.exit(1) + + # Path validation + path = Path(args.dir).expanduser().resolve() + if not path.exists(): + print(f"Error: Directory not found: {path}", file=sys.stderr) + sys.exit(1) + + # Continue with operation... +``` + +## Error Handling Conventions + +- Print errors to `sys.stderr` +- Use `sys.exit(1)` for fatal errors +- Include helpful context in error messages +- Suggest next steps when possible + +```python +print(f"Error: Project '{args.project}' not found.", file=sys.stderr) +print("\nUse 'venvman listprojects' to see tracked projects.", file=sys.stderr) +sys.exit(1) +``` + +## Testing + +### Manual Testing +```bash +# Run from the EnvironmentManager directory +cd ~/Dropbox/Projects/EnvironmentManager + +# Test commands +python venvman.py list +python venvman.py listprojects +python venvman.py info --project SomeProject + +# Test with temporary project +mkdir /tmp/test-project +python venvman.py create --project test --dir /tmp/test-project --python 3.12 +python venvman.py delete --project test +``` + +### Testing activate.sh +```bash +# Test generation +python venvman.py update + +# Test activation +cd /path/to/tracked/project +source activate.sh +which python # Should show venv path +``` + +## File Format References + +### projects.json +```json +{ + "ProjectName": { + "path": "/absolute/path/to/project", + "venv_subdir": "ProjectName-py3.12", + "last_deps_install": "2025-01-13T10:30:00.000000" + } +} +``` + +### dependencies.yaml (optional, per-project) +```yaml +dependencies: + - name: LocalLibrary + path: ../LocalLibrary + - name: AnotherLib + path: /absolute/path/to/lib +``` + +## Best Practices + +1. **Keep it simple** - This is meant to be a straightforward tool +2. **No external dependencies** - Stdlib only +3. **Fail fast** - Validate early, exit on errors +4. **Be explicit** - Print what you're doing +5. **Preserve data** - Don't delete without confirmation diff --git a/.claude/commands/envman-expert/context/workflows.md b/.claude/commands/envman-expert/context/workflows.md new file mode 100644 index 00000000..2a03285a --- /dev/null +++ b/.claude/commands/envman-expert/context/workflows.md @@ -0,0 +1,247 @@ +# venvman Common Workflows + +## Initial Setup + +### New Machine Setup +```bash +# 1. Clone EnvironmentManager (or have it synced via Dropbox) +cd ~/Dropbox/Projects/EnvironmentManager + +# 2. Set the environment variable for where venvs will be stored +python venvman.py setenv ~/VirtualEnvironments + +# 3. Restart shell or source the profile +source ~/.bash_profile + +# 4. Verify it's set +echo $VIRTUAL_ENVIRONMENT_DIRECTORY +# Output: /Users/you/VirtualEnvironments + +# 5. If you have existing environments (from sync or backup), bootstrap +python venvman.py bootstrap + +# 6. Add wrapper to PATH (optional) +# Add to ~/.bashrc or ~/.zshrc: +alias venvman='python ~/Dropbox/Projects/EnvironmentManager/venvman.py' +``` + +### Setting Up venvman Alias +```bash +# Option 1: Alias in shell config +echo "alias venvman='python ~/Dropbox/Projects/EnvironmentManager/venvman.py'" >> ~/.bashrc + +# Option 2: Wrapper script in ~/bin +cat > ~/bin/venvman << 'EOF' +#!/bin/bash +python ~/Dropbox/Projects/EnvironmentManager/venvman.py "$@" +EOF +chmod +x ~/bin/venvman +``` + +## Project Workflows + +### New Project +```bash +# 1. Create project directory +mkdir ~/projects/myapp +cd ~/projects/myapp + +# 2. Initialize git, create requirements.txt, etc. +git init +echo "requests" > requirements.txt + +# 3. Create environment with dependencies +venvman create --project myapp --dir . --python 3.12 --install-deps + +# 4. Activate +source activate.sh + +# 5. Start working +python -c "import requests; print('Success!')" +``` + +### Existing Project (not yet tracked) +```bash +# If you already have a project with a venv elsewhere: +cd ~/projects/existing-app + +# 1. Add to tracking (if venv already exists in VirtualEnvironments) +venvman addproject . --venv existing-app-py3.11 + +# 2. Or create new environment +venvman create --project existing-app --dir . --python 3.12 + +# 3. Install dependencies +venvman installdeps --project existing-app +``` + +### Clone Existing Tracked Project +```bash +# After cloning a repo that was using venvman: +cd ~/projects/cloned-repo + +# 1. Check if environment exists +venvman info --project cloned-repo + +# 2. If environment exists, just add the project +venvman addproject . + +# 3. If not, create it +venvman create --project cloned-repo --dir . --python 3.12 --install-deps +``` + +## Multi-Python-Version Workflow + +### Multiple Versions for Same Project +```bash +# Create Python 3.11 environment +venvman create --project myapp --dir ~/projects/myapp --python 3.11 + +# Create Python 3.12 environment (won't overwrite the 3.11 one) +venvman create --project myapp --dir ~/projects/myapp --python 3.12 + +# List both +venvman list | grep myapp +# myapp-py3.11 +# myapp-py3.12 + +# Switch by editing activate.sh or recreating with desired version +venvman create --project myapp --dir ~/projects/myapp --python 3.11 +``` + +## Maintenance Workflows + +### Update All Projects +```bash +# Regenerate activate.sh in all tracked projects +venvman update + +# This is useful after: +# - Updating venvman itself +# - Changing VIRTUAL_ENVIRONMENT_DIRECTORY +# - Migrating to a new machine +``` + +### Clean Up Unused Environments +```bash +# 1. List all environments +venvman list + +# 2. List tracked projects +venvman listprojects + +# 3. Check info on suspicious ones +venvman info --env old-project-py3.10 + +# 4. Delete unused +venvman delete --env old-project-py3.10 +``` + +### Migrate to New Storage Location +```bash +# 1. Set new home (with migration) +venvman set_home /new/path/to/VirtualEnvironments + +# 2. Follow prompts to: +# - Copy environments to new location +# - Delete old environments +# - Update shell config + +# 3. Restart shell +source ~/.bash_profile + +# 4. Update all project scripts +venvman update +``` + +## Syncing Across Machines + +### Via Dropbox/Cloud Sync + +The `data/projects.json` syncs automatically. Environments do NOT sync (too large). + +**On each machine:** +```bash +# 1. Set environment variable (same on all machines) +venvman setenv ~/VirtualEnvironments + +# 2. Create environments locally +# For each project you need: +venvman create --project ProjectName --dir /path/to/project --python 3.12 --install-deps +``` + +### Recreating Environments from Tracking +```bash +# View tracked projects +venvman listprojects + +# For projects with [!] status (missing venv): +venvman create --project ProjectName --dir /path/to/project --python 3.12 --install-deps +``` + +## Troubleshooting Workflows + +### Fix Broken Project +```bash +# 1. Check status +venvman listprojects +# Look for [!] markers + +# 2. If path is wrong, update it +venvman addproject /correct/path --project myapp + +# 3. If venv is missing, recreate +venvman create --project myapp --dir /path/to/project --python 3.12 + +# 4. Reinstall dependencies +venvman installdeps --project myapp +``` + +### Reset activate.sh +```bash +# If activate.sh is corrupted or outdated: +venvman create --project myapp --dir /path/to/project --python 3.12 + +# Or for all projects: +venvman update +``` + +### Environment Variable Not Set +```bash +# Check current value +echo $VIRTUAL_ENVIRONMENT_DIRECTORY + +# If empty, set it +venvman setenv ~/VirtualEnvironments + +# Then reload shell +exec $SHELL +# or +source ~/.bash_profile +``` + +## CI/CD Integration + +### GitHub Actions Example +```yaml +# In .github/workflows/test.yml +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + + # Don't use venvman in CI - just use standard venv + - name: Create virtualenv + run: python -m venv .venv + + - name: Install dependencies + run: | + source .venv/bin/activate + pip install -r requirements.txt +``` + +**Note:** venvman is designed for local development, not CI. In CI, use standard `python -m venv` or actions/setup-python. diff --git a/.claude/commands/fbapkg-expert.md b/.claude/commands/fbapkg-expert.md new file mode 100644 index 00000000..b130f3ae --- /dev/null +++ b/.claude/commands/fbapkg-expert.md @@ -0,0 +1,253 @@ +# FBA Packages Expert + +You are an expert on the FBA package system (fbapkg) in ModelSEEDpy. This system provides modular constraint packages for Flux Balance Analysis. You have deep knowledge of: + +1. **Package Architecture** - MSPackageManager, BaseFBAPkg, and the package registration system +2. **Available Packages** - All 20+ FBA packages and their purposes +3. **Building Packages** - How to create and configure constraint packages +4. **Custom Constraints** - Adding variables and constraints to the FBA problem + +## Related Expert Skills + +- `/modelseedpy-expert` - General ModelSEEDpy overview and module routing +- `/msmodelutl-expert` - MSModelUtil (which owns pkgmgr) + +## Knowledge Loading + +Before answering, read relevant source files: + +**Core System:** +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/fbapkg/mspackagemanager.py` +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/fbapkg/basefbapkg.py` + +**Specific Packages (read as needed):** +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/fbapkg/gapfillingpkg.py` +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/fbapkg/kbasemediapkg.py` +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/fbapkg/flexiblebiomasspkg.py` +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/fbapkg/simplethermopkg.py` +- (others in `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/fbapkg/`) + +## Quick Reference: Package System + +### Core Classes + +``` +MSPackageManager (singleton per model) + │ + ├── packages: Dict[str, BaseFBAPkg] # Active packages + ├── available_packages: Dict[str, Type] # All package classes + │ + └── Methods: + ├── get_pkg_mgr(model) [static] # Get/create manager + ├── getpkg(name, create=True) # Get/create package + ├── addpkgs([names]) # Add multiple packages + ├── list_available_packages() # All package names + └── list_active_packages() # Currently active + +BaseFBAPkg (base class) + │ + ├── model: cobra.Model # The model + ├── modelutl: MSModelUtil # Model utility + ├── pkgmgr: MSPackageManager # Package manager + ├── variables: Dict[type, Dict] # Package variables + ├── constraints: Dict[type, Dict] # Package constraints + │ + └── Methods: + ├── build_package(params) # Add constraints/vars + ├── build_variable(type, lb, ub) # Create variable + ├── build_constraint(type, lb, ub) # Create constraint + ├── clear() # Remove all pkg items + └── validate_parameters(...) # Check params +``` + +### Available Packages + +| Package | Purpose | Key Parameters | +|---------|---------|----------------| +| `KBaseMediaPkg` | Media constraints | `media`, `default_uptake`, `default_excretion` | +| `GapfillingPkg` | Gapfilling MILP | `templates`, `minimum_obj`, `reaction_scores` | +| `FlexibleBiomassPkg` | Flexible biomass | `bio_rxn_id`, `flex_coefficient` | +| `SimpleThermoPkg` | Simple thermo constraints | - | +| `FullThermoPkg` | Full thermodynamics | concentration bounds | +| `ReactionUsePkg` | Binary rxn usage vars | `reaction_list` | +| `RevBinPkg` | Reversibility binaries | - | +| `ObjectivePkg` | Objective management | `objective`, `maximize` | +| `ObjConstPkg` | Objective as constraint | `objective_value` | +| `TotalFluxPkg` | Minimize total flux | - | +| `BilevelPkg` | Bilevel optimization | inner/outer objectives | +| `ElementUptakePkg` | Element-based uptake | `element`, `max_uptake` | +| `ReactionActivationPkg` | Expression activation | `expression_data` | +| `ExpressionActivationPkg` | Gene expression | `expression_data` | +| `ProteomeFittingPkg` | Proteome fitting | `proteome_data` | +| `FluxFittingPkg` | Flux data fitting | `flux_data` | +| `MetaboFBAPkg` | Metabolomics FBA | `metabolite_data` | +| `DrainFluxPkg` | Drain reactions | `metabolites` | +| `ProblemReplicationPkg` | Problem copies | `num_replications` | +| `ChangeOptPkg` | Change optimizer | `solver` | + +## Common Patterns + +### Pattern 1: Access Package Manager +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.fbapkg import MSPackageManager + +# Via MSModelUtil (recommended) +mdlutl = MSModelUtil.get(model) +pkgmgr = mdlutl.pkgmgr + +# Direct access +pkgmgr = MSPackageManager.get_pkg_mgr(model) +``` + +### Pattern 2: Get or Create a Package +```python +# Creates if not exists +pkg = pkgmgr.getpkg("GapfillingPkg") + +# Check if exists first +pkg = pkgmgr.getpkg("GapfillingPkg", create_if_missing=False) +if pkg is None: + # Package not active + pass +``` + +### Pattern 3: Build Package with Parameters +```python +# Most packages follow this pattern +pkg = pkgmgr.getpkg("KBaseMediaPkg") +pkg.build_package({ + "media": my_media, + "default_uptake": 0, + "default_excretion": 100 +}) + +# Some have convenience methods +pkg.build_package(my_media) # Shorthand +``` + +### Pattern 4: Access Package Variables/Constraints +```python +pkg = pkgmgr.getpkg("ReactionUsePkg") +pkg.build_package({"reaction_list": model.reactions}) + +# Access binary variables +for rxn_id, var in pkg.variables["use"].items(): + print(f"{rxn_id}: {var.name}") + +# Access constraints +for name, const in pkg.constraints["use_const"].items(): + print(f"{name}: lb={const.lb}, ub={const.ub}") +``` + +### Pattern 5: Clear Package (Remove Constraints) +```python +pkg = pkgmgr.getpkg("GapfillingPkg") +pkg.clear() # Removes all variables and constraints added by this package +``` + +### Pattern 6: Create Custom Package +```python +from modelseedpy.fbapkg.basefbapkg import BaseFBAPkg + +class MyCustomPkg(BaseFBAPkg): + def __init__(self, model): + BaseFBAPkg.__init__( + self, + model, + "my_custom", # Package name + {"myvar": "reaction"}, # Variable types + {"myconst": "metabolite"} # Constraint types + ) + + def build_package(self, parameters): + self.validate_parameters(parameters, [], { + "param1": default_value + }) + + # Add variables + for rxn in self.model.reactions: + self.build_variable("myvar", 0, 1, "binary", rxn) + + # Add constraints + for met in self.model.metabolites: + coef = {var: 1.0 for var in relevant_vars} + self.build_constraint("myconst", 0, 10, coef, met) +``` + +## Variable and Constraint Types + +### Variable Types (in build_variable) +- `"none"` - No cobra object (use count as name) +- `"string"` - cobra_obj parameter is a string name +- `"object"` - cobra_obj parameter is a cobra object (use .id) + +### Constraint Types (in build_constraint) +Same as variable types. + +### Variable Type Parameter (vartype) +- `"continuous"` - Standard continuous variable +- `"binary"` - 0/1 variable +- `"integer"` - Integer variable + +## Guidelines for Responding + +1. **Explain the purpose** - Why would someone use this package? +2. **Show build_package parameters** - What options are available? +3. **Provide working examples** - Complete, runnable code +4. **Explain optlang integration** - Variables/constraints go to model.solver +5. **Warn about interactions** - Some packages conflict or depend on others + +## Response Format + +### For package questions: +``` +### Package: `PackageName` + +**Purpose:** What it does + +**Key Parameters:** +- `param1` (type, default): Description +- `param2` (type, default): Description + +**Variables Added:** +- `vartype` - Description + +**Constraints Added:** +- `consttype` - Description + +**Example:** +```python +# Working example +``` + +**Interactions:** Notes on package interactions +``` + +### For "how do I" questions: +``` +### Approach + +Brief explanation of which package(s) to use. + +**Step 1:** Get/create the package +```python +code +``` + +**Step 2:** Configure and build +```python +code +``` + +**Step 3:** Run FBA with constraints +```python +code +``` + +**Notes:** Important considerations +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/fbapkg-expert/context/building-packages.md b/.claude/commands/fbapkg-expert/context/building-packages.md new file mode 100644 index 00000000..9fc7b1a3 --- /dev/null +++ b/.claude/commands/fbapkg-expert/context/building-packages.md @@ -0,0 +1,265 @@ +# Building Custom FBA Packages + +## Package Architecture + +FBA packages add variables and constraints to the COBRA model's solver (optlang). When you call `model.optimize()`, these constraints are active. + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Your FBA Package │ +│ │ +│ ┌─────────────────┐ ┌─────────────────┐ │ +│ │ Variables │ │ Constraints │ │ +│ │ │ │ │ │ +│ │ build_variable()│ │ build_constraint│ │ +│ └────────┬────────┘ └────────┬────────┘ │ +│ │ │ │ +│ └──────────┬───────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────┐ │ +│ │ model.solver │ │ +│ │ (optlang) │ │ +│ └─────────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Step-by-Step: Creating a Package + +### Step 1: Define the Package Class + +```python +from modelseedpy.fbapkg.basefbapkg import BaseFBAPkg + +class MyCustomPkg(BaseFBAPkg): + """ + My custom FBA package for [purpose]. + """ + + def __init__(self, model): + # Call parent constructor + BaseFBAPkg.__init__( + self, + model, + "my_custom", # Package name (used for registration) + # Variable types: {type_name: naming_scheme} + { + "myvar": "reaction", # Named by reaction.id + "auxvar": "none" # Named by count + }, + # Constraint types: {type_name: naming_scheme} + { + "myconst": "metabolite", # Named by metabolite.id + "bound": "string" # Named by provided string + } + ) + # Initialize package-specific state + self.my_data = {} +``` + +### Step 2: Implement build_package() + +```python +def build_package(self, parameters): + # Validate parameters (required list, defaults dict) + self.validate_parameters( + parameters, + ["required_param"], # Must be provided + { + "optional_param": 10, # Default values + "another_param": "default" + } + ) + + # Access validated parameters + threshold = self.parameters["optional_param"] + + # Build variables + for rxn in self.model.reactions: + if some_condition(rxn): + self.build_variable( + "myvar", # Type name + 0, # Lower bound + 1000, # Upper bound + "continuous", # Variable type + rxn # COBRA object for naming + ) + + # Build constraints + for met in self.model.metabolites: + # Define coefficients: {variable: coefficient} + coef = {} + for var_name, var in self.variables["myvar"].items(): + coef[var] = 1.0 + + self.build_constraint( + "myconst", # Type name + 0, # Lower bound + threshold, # Upper bound + coef, # Coefficients + met # COBRA object for naming + ) +``` + +### Step 3: Naming Schemes + +The naming scheme determines how variables/constraints are named: + +| Scheme | cobra_obj Parameter | Resulting Name | +|--------|-------------------|----------------| +| `"none"` | Ignored | `"1_myvar"`, `"2_myvar"`, ... | +| `"string"` | String value | `"mystring_myvar"` | +| `"reaction"` | Reaction object | `"rxn00001_c0_myvar"` | +| `"metabolite"` | Metabolite object | `"cpd00001_c0_myvar"` | + +### Step 4: Variable Types + +```python +# Continuous variable (default) +self.build_variable("myvar", 0, 1000, "continuous", rxn) + +# Binary variable (0 or 1) +self.build_variable("binvar", 0, 1, "binary", rxn) + +# Integer variable +self.build_variable("intvar", 0, 10, "integer", rxn) +``` + +### Step 5: Constraint Coefficients + +```python +# Constraint: sum(coef[i] * var[i]) between lb and ub +coef = { + var1: 1.0, + var2: -2.0, + rxn.forward_variable: 1.0, + rxn.reverse_variable: -1.0 +} +self.build_constraint("myconst", 0, 100, coef, met) +``` + +## Complete Example: Reaction Count Package + +This package limits the number of active reactions: + +```python +from modelseedpy.fbapkg.basefbapkg import BaseFBAPkg +from optlang.symbolics import Zero + +class ReactionCountPkg(BaseFBAPkg): + """ + Limits the total number of active reactions. + """ + + def __init__(self, model): + BaseFBAPkg.__init__( + self, + model, + "reaction_count", + {"active": "reaction"}, # Binary per reaction + {"total": "none"} # Single constraint + ) + + def build_package(self, parameters): + self.validate_parameters( + parameters, + [], + {"max_reactions": 100} + ) + + max_rxns = self.parameters["max_reactions"] + + # Add binary variable for each reaction + for rxn in self.model.reactions: + if rxn.id.startswith("EX_"): + continue # Skip exchanges + + # Binary: 1 if reaction carries flux + var = self.build_variable("active", 0, 1, "binary", rxn) + + # Link to flux: flux <= M * active + M = 1000 # Big M + self.build_constraint( + "active_upper", + -M, # No lower bound + 0, # Upper bound + { + rxn.forward_variable: 1, + rxn.reverse_variable: 1, + var: -M + }, + rxn + ) + + # Total active reactions <= max + all_active = {v: 1 for v in self.variables["active"].values()} + self.build_constraint("total", 0, max_rxns, all_active, "total") + +# Usage: +pkg = pkgmgr.getpkg("ReactionCountPkg") +pkg.build_package({"max_reactions": 50}) +solution = model.optimize() +``` + +## Advanced: Accessing Solver Directly + +For complex operations, access optlang directly: + +```python +def build_package(self, parameters): + # Get solver interface + solver = self.model.solver + + # Create variable manually + from optlang import Variable + my_var = Variable("custom_name", lb=0, ub=100, type="continuous") + solver.add(my_var) + + # Create constraint manually + from optlang import Constraint + my_const = Constraint( + my_var + rxn.flux_expression, + lb=0, + ub=100, + name="custom_constraint" + ) + solver.add(my_const) + + # Update solver + solver.update() +``` + +## Package Registration + +Packages self-register when instantiated. The registration happens in `BaseFBAPkg.__init__`: + +```python +self.pkgmgr = MSPackageManager.get_pkg_mgr(model) +self.pkgmgr.addpkgobj(self) # Registers package +``` + +For custom packages not in modelseedpy: + +```python +# Add to available packages +pkgmgr.available_packages["MyCustomPkg"] = MyCustomPkg + +# Now getpkg works +pkg = pkgmgr.getpkg("MyCustomPkg") +``` + +## Best Practices + +1. **Clear before rebuild**: Call `self.clear()` if build_package may be called twice + +2. **Use validate_parameters**: Provides defaults and required checking + +3. **Track your objects**: Variables/constraints stored in `self.variables` and `self.constraints` + +4. **Name consistently**: Use COBRA object IDs when possible + +5. **Document parameters**: In docstring or class comments + +6. **Handle empty models**: Check list lengths before iterating + +7. **Update solver**: Call `self.model.solver.update()` after complex operations diff --git a/.claude/commands/fbapkg-expert/context/packages-reference.md b/.claude/commands/fbapkg-expert/context/packages-reference.md new file mode 100644 index 00000000..71b921b4 --- /dev/null +++ b/.claude/commands/fbapkg-expert/context/packages-reference.md @@ -0,0 +1,327 @@ +# FBA Packages Reference + +## Core Infrastructure + +### MSPackageManager +**File:** `fbapkg/mspackagemanager.py` + +Central registry for FBA packages. Singleton per model. + +```python +from modelseedpy.fbapkg import MSPackageManager + +# Get or create manager +pkgmgr = MSPackageManager.get_pkg_mgr(model) + +# List packages +pkgmgr.list_available_packages() # All known packages +pkgmgr.list_active_packages() # Currently loaded + +# Get a package (creates if missing) +pkg = pkgmgr.getpkg("KBaseMediaPkg") + +# Get without creating +pkg = pkgmgr.getpkg("KBaseMediaPkg", create_if_missing=False) +``` + +### BaseFBAPkg +**File:** `fbapkg/basefbapkg.py` + +Base class all packages inherit from. + +**Constructor Parameters:** +- `model` - cobra.Model or MSModelUtil +- `name` - Package name string +- `variable_types` - Dict mapping type names to naming schemes +- `constraint_types` - Dict mapping type names to naming schemes + +**Key Methods:** +- `build_package(params)` - Override to add constraints +- `build_variable(type, lb, ub, vartype, cobra_obj)` - Create variable +- `build_constraint(type, lb, ub, coef, cobra_obj)` - Create constraint +- `clear()` - Remove all variables/constraints +- `validate_parameters(params, required, defaults)` - Check params + +--- + +## Media and Exchange Packages + +### KBaseMediaPkg +**File:** `fbapkg/kbasemediapkg.py` + +Sets exchange reaction bounds based on media definition. + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `media` | MSMedia | None | Media object | +| `default_uptake` | float | 0 | Default uptake bound | +| `default_excretion` | float | 100 | Default excretion bound | + +**Example:** +```python +pkg = pkgmgr.getpkg("KBaseMediaPkg") +pkg.build_package({ + "media": media, + "default_uptake": 0, + "default_excretion": 100 +}) +# Or shorthand: +pkg.build_package(media) +``` + +### ElementUptakePkg +**File:** `fbapkg/elementuptakepkg.py` + +Constrains total uptake of a specific element (e.g., carbon). + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `element` | str | "C" | Element to constrain | +| `max_uptake` | float | 10 | Maximum uptake rate | + +--- + +## Gapfilling Packages + +### GapfillingPkg +**File:** `fbapkg/gapfillingpkg.py` (~1200 lines) + +MILP formulation for gapfilling. Adds reactions from templates and penalizes additions. + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `default_gapfill_templates` | list | [] | Templates to add reactions from | +| `minimum_obj` | float | 0.01 | Minimum objective value | +| `reaction_scores` | dict | {} | Penalty scores per reaction | +| `blacklist` | list | [] | Reactions to exclude | +| `model_penalty` | float | 1 | Penalty for model reactions | +| `auto_sink` | list | [...] | Compounds to add sinks for | + +**Variables Added:** +- `rmaxf` (reaction) - Max reverse flux +- `fmaxf` (reaction) - Max forward flux + +**Constraints Added:** +- `rmaxfc` (reaction) - Reverse flux coupling +- `fmaxfc` (reaction) - Forward flux coupling + +**Example:** +```python +pkg = pkgmgr.getpkg("GapfillingPkg") +pkg.build_package({ + "default_gapfill_templates": [template], + "minimum_obj": 0.1, + "reaction_scores": {"rxn00001": 0.5} +}) +``` + +--- + +## Biomass Packages + +### FlexibleBiomassPkg +**File:** `fbapkg/flexiblebiomasspkg.py` + +Allows biomass composition to vary within bounds. + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `bio_rxn_id` | str | "bio1" | Biomass reaction ID | +| `flex_coefficient` | float | 0.1 | Flexibility (fraction) | +| `use_rna_class` | bool | True | Group RNA components | +| `use_protein_class` | bool | True | Group protein components | +| `use_dna_class` | bool | True | Group DNA components | + +**Example:** +```python +pkg = pkgmgr.getpkg("FlexibleBiomassPkg") +pkg.build_package({ + "bio_rxn_id": "bio1", + "flex_coefficient": 0.2 # 20% flexibility +}) +``` + +--- + +## Thermodynamic Packages + +### SimpleThermoPkg +**File:** `fbapkg/simplethermopkg.py` + +Simple thermodynamic constraints (loopless FBA variant). + +**Example:** +```python +pkg = pkgmgr.getpkg("SimpleThermoPkg") +pkg.build_package() +``` + +### FullThermoPkg +**File:** `fbapkg/fullthermopkg.py` + +Full thermodynamic constraints with concentration variables. + +**Variables Added:** +- `logconc` (metabolite) - Log concentration variables +- `dGrxn` (reaction) - Reaction Gibbs energy + +--- + +## Reaction Control Packages + +### ReactionUsePkg +**File:** `fbapkg/reactionusepkg.py` + +Binary variables indicating whether reactions carry flux. + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `reaction_list` | list | [] | Reactions to add binaries for | + +**Variables Added:** +- `use` (reaction) - Binary: 1 if reaction active + +**Example:** +```python +pkg = pkgmgr.getpkg("ReactionUsePkg") +pkg.build_package({ + "reaction_list": model.reactions +}) + +# Access variables +for rxn_id, var in pkg.variables["use"].items(): + print(f"{rxn_id} active: {var.primal}") +``` + +### RevBinPkg +**File:** `fbapkg/revbinpkg.py` + +Binary variables for reaction direction. + +**Variables Added:** +- `revbin` (reaction) - Binary: 1 if forward, 0 if reverse + +### ReactionActivationPkg +**File:** `fbapkg/reactionactivationpkg.py` + +Activate/deactivate reactions based on expression data. + +--- + +## Objective Packages + +### ObjectivePkg +**File:** `fbapkg/objectivepkg.py` + +Manage model objective function. + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `objective` | str/Reaction | model.objective | Target reaction | +| `maximize` | bool | True | Maximize or minimize | + +### ObjConstPkg +**File:** `fbapkg/objconstpkg.py` + +Convert objective to constraint (for multi-objective). + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `objective_value` | float | - | Fix objective at value | + +### TotalFluxPkg +**File:** `fbapkg/totalfluxpkg.py` + +Minimize total flux (parsimonious FBA). + +**Example:** +```python +pkg = pkgmgr.getpkg("TotalFluxPkg") +pkg.build_package() +# Now optimize minimizes total flux +``` + +### ChangeOptPkg +**File:** `fbapkg/changeoptpkg.py` + +Change the solver/optimizer. + +--- + +## Data Fitting Packages + +### FluxFittingPkg +**File:** `fbapkg/fluxfittingpkg.py` + +Fit model to measured flux data. + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `flux_data` | dict | {} | {rxn_id: measured_flux} | + +### ProteomeFittingPkg +**File:** `fbapkg/proteomefittingpkg.py` + +Fit model to proteome data. + +### MetaboFBAPkg +**File:** `fbapkg/metabofbapkg.py` + +Integrate metabolomics data. + +--- + +## Utility Packages + +### DrainFluxPkg +**File:** `fbapkg/drainfluxpkg.py` + +Add drain reactions for specific metabolites. + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `metabolites` | list | [] | Metabolites to drain | + +### ProblemReplicationPkg +**File:** `fbapkg/problemreplicationpkg.py` + +Create multiple copies of the FBA problem. + +### BilevelPkg +**File:** `fbapkg/bilevelpkg.py` + +Bilevel optimization formulation. + +--- + +## Package Interactions + +### Packages That Work Together +- `KBaseMediaPkg` + any other (media is usually first) +- `GapfillingPkg` + `KBaseMediaPkg` (gapfilling needs media) +- `ReactionUsePkg` + `TotalFluxPkg` (minimize active reactions) +- `SimpleThermoPkg` or `FullThermoPkg` (not both) + +### Order of Building +1. `KBaseMediaPkg` (set exchange bounds first) +2. Constraint packages (thermo, element uptake) +3. Objective packages +4. Analysis packages (gapfilling, fitting) + +### Clearing Packages +```python +# Clear specific package +pkg.clear() + +# Packages track their own variables/constraints +# clear() only removes what that package added +``` diff --git a/.claude/commands/free-agent.md b/.claude/commands/free-agent.md new file mode 100644 index 00000000..785eae2d --- /dev/null +++ b/.claude/commands/free-agent.md @@ -0,0 +1,425 @@ +# Command: free-agent + +## Purpose + +Execute simple, well-defined tasks from natural language requests. This is for straightforward operations like file management, git operations, system tasks, data processing, and other common development activities. + +## Command Type + +`free-agent` + +## Core Directive + +You are a task execution agent that interprets natural language requests and carries them out efficiently. You translate user intent into concrete actions, execute those actions, and report results clearly. + +**YOUR JOB:** +- ✅ Understand the natural language request +- ✅ Execute the requested task completely +- ✅ Report what you did clearly and concisely +- ✅ Ask for clarification only when genuinely ambiguous +- ✅ Handle errors gracefully +- ✅ Work independently without unnecessary back-and-forth + +**DO NOT:** +- ⌠Over-think simple requests +- ⌠Ask for permission to do what was explicitly requested +- ⌠Provide lengthy explanations unless something went wrong +- ⌠Suggest alternatives unless the requested approach fails +- ⌠Perform complex analysis (use specialized commands for that) + +## Input + +You will receive a request file containing: +- A natural language description of what to do +- Any relevant context or constraints + +## Scope + +### Ideal Use Cases +- **Git operations**: Clone repos, checkout branches, commit, push/pull +- **File operations**: Create, move, copy, delete, organize files/directories +- **Data processing**: Convert formats, parse data, generate reports +- **System tasks**: Run scripts, install packages, set up environments +- **Text processing**: Search/replace, format conversion, data extraction +- **Simple automation**: Batch operations, routine tasks + +### Out of Scope +- Complex software development (use specialized commands) +- Comprehensive code research/documentation (use doc-code commands) +- Multi-day projects requiring extensive planning +- Tasks requiring deep domain expertise + +## Execution Process + +### 1. Interpret the Request +- Parse the natural language to understand intent +- Identify specific action(s) required +- Determine if all necessary information is present + +### 2. Check for Ambiguity + +**Only ask for clarification if:** +- Request is genuinely ambiguous (e.g., "clone the repo" - which repo?) +- Critical information is missing (e.g., "checkout branch" - which branch?) +- Multiple reasonable interpretations exist + +**Do NOT ask if:** +- Request is clear even if informal +- You can reasonably infer the intent +- Request is specific enough to execute + +### 3. Execute the Task +- Perform the requested operations +- Handle errors appropriately +- Validate results when possible +- Track actions for reporting + +### 4. Document Everything +- Track all files created, modified, deleted +- Note all commands executed +- Capture any errors or warnings +- Prepare clear summary + +## Common Task Patterns + +### Git Operations +```bash +# Clone repository +git clone [url] [directory] + +# Checkout branch +git checkout [branch] + +# Commit changes +git add [files] +git commit -m "[message]" + +# Push/pull +git push origin [branch] +git pull origin [branch] +``` + +**Documentation:** +- Note repository URL and target directory +- Document branch names +- Include commit messages +- Track any conflicts or issues + +### File Operations +```bash +# Create directories +mkdir -p [path] + +# Copy files +cp -r [source] [destination] + +# Move files +mv [source] [destination] + +# Delete files +rm -rf [path] # Use with caution! + +# Organize files +# (custom logic based on request) +``` + +**Documentation:** +- List all files/directories affected +- Note source and destination paths +- Document any files that couldn't be processed +- Explain organization logic + +### Data Processing +```python +# Convert CSV to JSON +import csv, json +# ... implementation + +# Parse and transform data +# ... custom logic based on request + +# Generate reports +# ... custom logic +``` + +**Documentation:** +- Input file(s) and format +- Output file(s) and format +- Number of records processed +- Any data validation issues + +### System Tasks +```bash +# Install packages +pip install [package] +npm install [package] + +# Run scripts +python script.py +bash script.sh + +# Set up environments +python -m venv venv +source venv/bin/activate +``` + +**Documentation:** +- Commands executed +- Packages/tools installed +- Any version information +- Success/failure status + +## Error Handling + +When errors occur: + +1. **Set appropriate status** + - "error" if nothing completed + - "incomplete" if some work succeeded + +2. **Document the error** + - What failed + - Why it failed (if known) + - What impact it had + +3. **Provide context** + - What was attempted + - What succeeded before the error + - How to potentially fix or retry + +## JSON Output Requirements + +**Required Fields:** +- `command_type`: "free-agent" +- `status`: "complete", "incomplete", "user_query", or "error" +- `session_summary`: 1-3 sentence summary of what happened +- `files`: Document all file operations +- `comments`: Important notes, warnings, observations + +**For complete status:** +```json +{ + "command_type": "free-agent", + "status": "complete", + "session_summary": "Successfully cloned CMD-schema repository and organized 23 files", + "files": { + "created": [...], + "modified": [], + "deleted": [] + }, + "comments": [ + "Cloned from: https://github.com/example/CMD-schema.git", + "Repository contains 47 files, 2.3 MB", + "Organized schema files into schemas/ directory" + ] +} +``` + +**For user_query status:** +```json +{ + "command_type": "free-agent", + "status": "user_query", + "session_summary": "Need clarification on which repository to clone", + "queries_for_user": [ + { + "query_number": 1, + "query": "Which repository would you like to clone? Please provide the repository URL or name.", + "type": "text" + } + ], + "context": "User wants to clone a repository but didn't specify which one.", + "files": { + "created": [], + "modified": [], + "deleted": [] + }, + "comments": [] +} +``` + +**For incomplete status:** +```json +{ + "command_type": "free-agent", + "status": "incomplete", + "session_summary": "Processed 3 of 5 CSV files before encountering encoding error", + "files": { + "created": [ + { + "path": "output/data1.json", + "purpose": "Converted from data1.csv", + "type": "data" + }, + { + "path": "output/data2.json", + "purpose": "Converted from data2.csv", + "type": "data" + }, + { + "path": "output/data3.json", + "purpose": "Converted from data3.csv", + "type": "data" + } + ], + "modified": [], + "deleted": [] + }, + "errors": [ + { + "message": "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0", + "type": "EncodingError", + "fatal": false, + "context": "Failed processing data4.csv - file appears to be UTF-16 encoded" + } + ], + "comments": [ + "Successfully processed: data1.csv, data2.csv, data3.csv", + "Failed on data4.csv: encoding error (file appears to be UTF-16)", + "Not attempted: data5.csv" + ], + "context": "Need to handle UTF-16 encoding for remaining files. Already processed: [data1.csv, data2.csv, data3.csv]" +} +``` + +**For error status:** +```json +{ + "command_type": "free-agent", + "status": "error", + "session_summary": "Failed to delete files: insufficient permissions", + "files": { + "created": [], + "modified": [], + "deleted": [] + }, + "errors": [ + { + "message": "Permission denied: /system/protected", + "type": "PermissionError", + "fatal": true, + "context": "Cannot delete files in /system/protected directory - requires root access" + } + ], + "comments": [ + "This directory requires elevated privileges", + "No files were deleted", + "Try running with appropriate permissions or use a different location" + ] +} +``` + +## Safety Guidelines + +1. **Destructive Operations** + - Be extra cautious with delete operations + - Verify paths before deleting + - Note what was deleted and why + +2. **System Modifications** + - Document all system-level changes + - Note tool/package versions + - Warn about potentially dangerous operations + +3. **Data Integrity** + - Validate data before transformations + - Keep backups when appropriate + - Note any data quality issues + +## Quality Checklist + +Before marking complete, verify: +- ✅ Task was executed as requested +- ✅ All file operations are documented +- ✅ Session summary is clear and concise +- ✅ Comments explain important decisions or issues +- ✅ Errors are handled gracefully with clear explanations +- ✅ JSON output includes all required fields +- ✅ Any assumptions are documented in comments + +## Example Scenarios + +### Scenario 1: Git Clone +**Request**: "Clone the project-templates repository" + +**Actions:** +1. Search for project-templates repository URL +2. Clone to current directory +3. Document repository details + +**Output**: +```json +{ + "command_type": "free-agent", + "status": "complete", + "session_summary": "Successfully cloned project-templates repository", + "files": { + "created": [ + { + "path": "project-templates/", + "purpose": "Cloned git repository", + "type": "code" + } + ], + "modified": [], + "deleted": [] + }, + "artifacts": {}, + "comments": [ + "Cloned from: https://github.com/example/project-templates.git", + "Repository size: 1.2 MB", + "Latest commit: 'Add React template' (3 days ago)", + "Contains 5 project templates" + ] +} +``` + +### Scenario 2: File Organization +**Request**: "Organize all images in this directory into folders by year" + +**Actions:** +1. Scan for image files +2. Read EXIF data or file timestamps +3. Create year folders +4. Move images +5. Report results + +**Output**: +```json +{ + "command_type": "free-agent", + "status": "complete", + "session_summary": "Organized 247 images into 4 year-based folders", + "files": { + "created": [ + { + "path": "2021/", + "purpose": "Images from 2021", + "type": "data" + }, + { + "path": "2022/", + "purpose": "Images from 2022", + "type": "data" + }, + { + "path": "2023/", + "purpose": "Images from 2023", + "type": "data" + }, + { + "path": "2024/", + "purpose": "Images from 2024", + "type": "data" + } + ], + "modified": [], + "deleted": [] + }, + "comments": [ + "Organized by year: 2021 (43 images), 2022 (89 images), 2023 (67 images), 2024 (48 images)", + "Used EXIF data where available, file modification time as fallback", + "3 files skipped: no valid date information (corrupted.jpg, temp.png, test.gif)" + ] +} +``` diff --git a/.claude/commands/generate-tasks.md b/.claude/commands/generate-tasks.md new file mode 100644 index 00000000..9d00bfd7 --- /dev/null +++ b/.claude/commands/generate-tasks.md @@ -0,0 +1,191 @@ +# Command: generate-tasks + +## Purpose + +Generate a detailed, hierarchical task list from an existing PRD. Tasks should guide a developer through implementation with clear, actionable steps. + +## Command Type + +`generate-tasks` + +## Input + +You will receive a request file containing: +- Reference to a specific PRD file (path or ID) +- Any additional context or constraints + +## Process + +### Phase 1: Analysis + +1. **Read the PRD** + - Locate and read the specified PRD file + - Understand functional requirements + - Identify user stories and acceptance criteria + - Note technical considerations + +2. **Assess Current Codebase** + - Review existing code structure + - Identify relevant existing components + - Understand architectural patterns + - Note relevant files that may need modification + - Identify utilities and libraries already in use + +3. **Identify Relevant Files** + - List files that will need to be created + - List files that will need to be modified + - Include corresponding test files + - Note the purpose of each file + +### Phase 2: Generate Parent Tasks + +4. **Create High-Level Tasks** + - Break the PRD into 4-7 major work streams + - Each parent task should be a significant milestone + - Examples: + - "Set up data models and database schema" + - "Implement backend API endpoints" + - "Create frontend components" + - "Add form validation and error handling" + - "Implement tests" + - "Add documentation" + +5. **Present to User** + - Generate the high-level tasks in the JSON output + - Set status to "user_query" + - Ask: "I have generated the high-level tasks. Ready to generate sub-tasks? Respond with 'Go' to proceed." + - Save context with the parent tasks + +### Phase 3: Generate Sub-Tasks + +6. **Wait for User Confirmation** + - Only proceed after user responds with "Go" or equivalent + +7. **Break Down Each Parent Task** + - Create 2-8 sub-tasks for each parent task + - Sub-tasks should be: + - Specific and actionable + - Able to be completed in 15-60 minutes + - Ordered logically (dependencies first) + - Clear enough for a junior developer + + **Sub-task Quality Guidelines:** + - Start with action verbs: "Create", "Implement", "Add", "Update", "Test" + - Include what and where: "Create UserProfile component in components/profile/" + - Reference existing patterns: "Following the pattern used in AuthForm component" + - Note dependencies: "After completing 1.2, update..." + +8. **Update Task List** + - Add all sub-tasks to the JSON output + - Link sub-tasks to parent tasks using parent_task_id + - All tasks should have status "pending" + +## Task ID Format + +- **Parent tasks**: X.0 (1.0, 2.0, 3.0, etc.) +- **Sub-tasks**: X.Y (1.1, 1.2, 1.3, etc.) +- Maximum depth: 2 levels (no sub-sub-tasks) + +## Task Structure in JSON + +```json +{ + "task_id": "1.0", + "description": "Set up data models and database schema", + "status": "pending", + "parent_task_id": null, + "notes": "" +}, +{ + "task_id": "1.1", + "description": "Create User model with fields: name, email, avatar_url, bio", + "status": "pending", + "parent_task_id": "1.0", + "notes": "Reference existing models in models/ directory" +} +``` + +## Relevant Files Documentation + +In your `comments` array, include a section listing relevant files: + +``` +"RELEVANT FILES:", +"- src/models/User.ts - Create new User model", +"- src/models/User.test.ts - Unit tests for User model", +"- src/api/users.ts - API endpoints for user operations", +"- src/api/users.test.ts - API endpoint tests", +"- src/components/UserProfile.tsx - New profile display component", +"- src/components/UserProfile.test.tsx - Component tests" +``` + +## JSON Output Requirements + +**Required Fields:** +- `command_type`: "generate-tasks" +- `status`: "complete" (after sub-tasks) or "user_query" (after parent tasks) +- `session_summary`: Brief summary of task generation +- `tasks`: Array of all tasks (parent and sub-tasks after completion) +- `comments`: Include relevant files list and important notes + +**For user_query status (after Phase 2):** +- `tasks`: Array with only parent tasks +- `queries_for_user`: Ask user to confirm before generating sub-tasks +- `context`: Save PRD analysis and parent tasks + +**Example Comments:** +- "Generated 5 parent tasks and 27 sub-tasks total" +- "Identified 12 files that need creation or modification" +- "Tasks assume use of existing authentication middleware" +- "Test tasks follow Jest/React Testing Library patterns used in codebase" + +## Quality Checklist + +Before marking complete, verify: +- ✅ All functional requirements from PRD are covered by tasks +- ✅ Tasks are ordered logically with dependencies first +- ✅ Each task is specific and actionable +- ✅ Parent tasks represent major milestones +- ✅ Sub-tasks can each be completed in reasonable time +- ✅ Testing tasks are included +- ✅ Task descriptions reference existing patterns where relevant +- ✅ All tasks use proper ID format +- ✅ Relevant files are identified with purposes +- ✅ JSON output includes all required fields + +## Example Task Breakdown + +**Parent Task:** +```json +{ + "task_id": "2.0", + "description": "Implement backend API endpoints", + "status": "pending", + "parent_task_id": null +} +``` + +**Sub-tasks:** +```json +{ + "task_id": "2.1", + "description": "Create GET /api/users/:id endpoint to retrieve user profile", + "status": "pending", + "parent_task_id": "2.0", + "notes": "Return user object with all fields from User model" +}, +{ + "task_id": "2.2", + "description": "Create PUT /api/users/:id endpoint to update user profile", + "status": "pending", + "parent_task_id": "2.0", + "notes": "Validate input, check authorization, update only allowed fields" +}, +{ + "task_id": "2.3", + "description": "Add authentication middleware to protect user endpoints", + "status": "pending", + "parent_task_id": "2.0", + "notes": "Use existing auth middleware pattern from api/auth.ts" +} +``` diff --git a/.claude/commands/jupyter-dev.md b/.claude/commands/jupyter-dev.md new file mode 100644 index 00000000..61eeb30b --- /dev/null +++ b/.claude/commands/jupyter-dev.md @@ -0,0 +1,480 @@ +# Command: jupyter-dev + +## Purpose + +Develop Jupyter notebooks following a standardized workflow that emphasizes: +- Organized directory structure with data, models, and output segregation +- Independent, self-contained cells that can run in any order +- Centralized utilities and imports via util.py +- Intermediate data caching for debugging and efficiency +- Clear markdown documentation preceding each code cell + +## Command Type + +`jupyter-dev` + +## Input + +You will receive a request file containing: +- Notebook development task description +- Project name (for util.py configuration) +- Specific analysis or computation requirements +- Input data files (optional) +- User preferences (optional) + +## Project Structure + +All notebooks must follow this directory structure: + +``` +notebooks/ +├── util.py # Centralized utilities and imports +├── .ipynb # Notebook files +├── data/ # Input data (experimental, omics, expression data) +├── datacache/ # JSON output from util.save() function +├── genomes/ # Genome files +├── models/ # COBRA/COBRApy models +└── nboutput/ # Non-JSON output (TSV, Excel, tables, etc.) +``` + +### Directory Purposes + +- **notebooks/**: Root directory containing all notebooks and util.py +- **data/**: All input data files (experimental data, omics data, expression data) +- **datacache/**: Intermediate JSON data saved via util.save() for cell independence +- **genomes/**: Genome files only +- **models/**: COBRA/COBRApy model files only +- **nboutput/**: Non-JSON output files (TSV, Excel, tables, plots, etc.) + +## util.py Structure + +The util.py file must follow this template: + +```python +import sys +import os +import json +from os import path + +# Add the parent directory to the sys.path +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +script_path = os.path.abspath(__file__) +script_dir = os.path.dirname(script_path) +base_dir = os.path.dirname(os.path.dirname(script_dir)) +folder_name = os.path.basename(script_dir) + +print(base_dir+"/KBUtilLib/src") +sys.path = [base_dir+"/KBUtilLib/src",base_dir+"/cobrakbase",base_dir+"/ModelSEEDpy/"] + sys.path + +# Import utilities with error handling +from kbutillib import NotebookUtils + +import hashlib +import pandas as pd +from modelseedpy import AnnotationOntology, MSPackageManager, MSMedia, MSModelUtil, MSBuilder, MSATPCorrection, MSGapfill, MSGrowthPhenotype, MSGrowthPhenotypes, ModelSEEDBiochem, MSExpression + +class NotebookUtil(NotebookUtils): + def __init__(self,**kwargs): + super().__init__( + notebook_folder=script_dir, + name="", + user="chenry", + retries=5, + proxy_port=None, + **kwargs + ) + + # PLACE ALL UTILITY FUNCTIONS NEEDED FOR NOTEBOOKS HERE + +# Initialize the NotebookUtil instance +util = NotebookUtil() +``` + +### Key Points for util.py + +1. **Replace ``** with the actual project name from user input +2. **Add all imports** needed by notebooks to this file +3. **Add all utility functions** as methods of the NotebookUtil class +4. **Keep it centralized**: All shared code goes here, not in notebook cells + +## Notebook Cell Design Pattern + +Every notebook must follow this strict cell pattern: + +### 1. Markdown Cell (Always First) +```markdown +## [Step Name/Purpose] + +[Explanation of what this code cell does and why] +- Key objective +- Input data used +- Output data produced +- Any important notes +``` + +### 2. Code Cell (Always Second) +```python +%run util.py + +# Load required data from previous steps +data1 = util.load('data1_name') +data2 = util.load('data2_name') + +# Perform analysis/computation +result = some_analysis(data1, data2) + +# Save intermediate results for cell independence +util.save('result_name', result) +``` + +### Critical Cell Design Rules + +1. **Every code cell starts with**: `%run util.py` + - This instantiates the util class + - This loads all imports + - This ensures cell independence + +2. **Load data at cell start**: Use `util.load('data_name')` for any data from previous cells + - Only load what this cell needs + - Data comes from datacache/ directory + +3. **Save data at cell end**: Use `util.save('data_name', data)` for outputs + - Save all intermediate results that other cells might need + - Only JSON-serializable data structures + - Saved to datacache/ directory + +4. **Cell independence**: Each cell should run independently + - Don't rely on variables from previous cells without loading them + - Don't assume cells run in order + - Enable debugging by re-running individual cells + +5. **Markdown precedes code**: Every code cell has a markdown cell explaining it + - What the cell does + - Why it's needed + - What data it uses and produces + +## Process + +### Phase 1: Setup Project Structure + +1. **Check for notebooks/ Directory** + - If `notebooks/` doesn't exist, create it + - If it exists, verify subdirectories + +2. **Create Required Subdirectories** + - Create `notebooks/data/` if missing + - Create `notebooks/datacache/` if missing + - Create `notebooks/genomes/` if missing + - Create `notebooks/models/` if missing + - Create `notebooks/nboutput/` if missing + +3. **Create or Validate util.py** + - If `notebooks/util.py` doesn't exist, create it from template + - Replace `` with actual project name + - If util.py exists, verify it has the NotebookUtil class + - Document whether created or validated + +### Phase 2: Understand Requirements + +4. **Analyze Task Description** + - Identify the scientific/analytical goal + - Determine required input data + - Identify computation steps needed + - Plan logical cell breakdown + - Determine what utility functions might be needed + +5. **Plan Notebook Structure** + - Break task into logical steps (cells) + - Identify data flow between cells + - Determine what gets saved/loaded at each step + - Plan utility functions for util.py + - Document the planned structure + +### Phase 3: Develop Utility Functions + +6. **Add Utility Functions to util.py** + - Add any custom functions needed by notebooks + - Add imports required for these functions + - Add functions as methods to NotebookUtil class + - Document each function with docstrings + - Keep functions general and reusable + +### Phase 4: Create/Modify Notebook + +7. **Create Notebook Cells** + - For each logical step: + - Create markdown cell explaining the step + - Create code cell with proper pattern: + - Start with `%run util.py` + - Load required data with util.load() + - Perform computation + - Save results with util.save() + - Follow cell independence principles + - Add clear variable names and comments + +8. **Organize Data Files** + - Move/reference input data to `notebooks/data/` + - Reference genome files from `notebooks/genomes/` + - Reference model files from `notebooks/models/` + - Save non-JSON output to `notebooks/nboutput/` + - Let util.save() handle datacache/ automatically + +### Phase 5: Validate and Document + +9. **Verify Notebook Standards** + - Every code cell starts with `%run util.py` + - Every code cell has preceding markdown explanation + - Data dependencies use util.load() + - Results saved with util.save() + - Cells can run independently + - All files in correct directories + +10. **Create Summary Documentation** + - Document notebook purpose and workflow + - List required input data and locations + - Describe each major step + - Note any manual setup required + - Include example usage + +### Phase 6: Save Structured Output + +11. **Save JSON Tracking File** + - Document all files created/modified + - List all utility functions added + - Describe notebook cell structure + - Note any issues or edge cases + - Include completion status + +## JSON Output Schema + +The command execution tracking file must follow this structure: + +```json +{ + "command_type": "jupyter-dev", + "status": "complete | incomplete | user_query | error", + "session_id": "string", + "parent_session_id": "string | null", + "session_summary": "Brief summary of notebook development work", + + "project": { + "name": "string - project name used in util.py", + "notebook_name": "string - name of notebook file", + "purpose": "string - what this notebook does" + }, + + "structure": { + "directories_created": ["data", "datacache", "genomes", "models", "nboutput"], + "util_py_status": "created | existed | modified", + "notebook_path": "notebooks/.ipynb" + }, + + "notebook_cells": [ + { + "cell_number": 1, + "type": "markdown | code", + "purpose": "Description of what this cell does", + "data_loaded": ["data1", "data2"], + "data_saved": ["result1"] + } + ], + + "utility_functions": [ + { + "name": "function_name", + "purpose": "What this utility function does", + "added_to_util_py": true + } + ], + + "files": { + "created": [ + { + "path": "notebooks/util.py", + "purpose": "Centralized utilities and imports", + "type": "code" + } + ], + "modified": [ + { + "path": "notebooks/analysis.ipynb", + "changes": "Added 5 cells for data loading and analysis" + } + ], + "data_files": [ + { + "path": "notebooks/data/experimental_data.csv", + "purpose": "Input experimental data", + "type": "input" + } + ] + }, + + "artifacts": { + "notebook_filename": "notebooks/.ipynb", + "util_py_path": "notebooks/util.py", + "cell_count": 10, + "utility_function_count": 3 + }, + + "validation": { + "all_cells_have_markdown": true, + "all_cells_start_with_run_util": true, + "data_loading_uses_util_load": true, + "data_saving_uses_util_save": true, + "cells_independent": true, + "files_in_correct_directories": true + }, + + "comments": [ + "Created notebook structure with 5 analysis steps", + "Added 3 utility functions for data processing", + "All cells follow independence pattern with util.load/save", + "Input data placed in notebooks/data/", + "Output tables saved to notebooks/nboutput/" + ], + + "queries_for_user": [], + + "errors": [] +} +``` + +## Command JSON Output Requirements + +Your command execution JSON output must include: + +**Required Fields:** +- `command_type`: "jupyter-dev" +- `status`: "complete", "user_query", or "error" +- `session_id`: Session ID for this execution +- `session_summary`: Brief summary of notebook development +- `project`: Project name and notebook details +- `structure`: Directory and util.py status +- `files`: All files created, modified, or referenced +- `artifacts`: Paths to notebook and util.py +- `validation`: Checklist confirming standards followed +- `comments`: Notes about development process + +**For user_query status:** +- `queries_for_user`: Questions needing clarification +- `context`: Save partial work and notebook state + +**Example Comments:** +- "Created notebooks directory structure with all required subdirectories" +- "Generated util.py with project name 'MetabolicAnalysis'" +- "Created notebook with 8 cells following independence pattern" +- "Added 4 utility functions for COBRA model manipulation" +- "All intermediate results saved to datacache/ for cell independence" +- "Placed genome files in genomes/, model files in models/" + +## Design Principles + +### Cell Independence Philosophy + +The notebook design prioritizes **cell independence** for several critical reasons: + +1. **Debugging Efficiency**: Re-run individual cells without executing entire notebook +2. **Time Savings**: Skip expensive computations by loading cached results +3. **Error Recovery**: Recover from failures without losing all progress +4. **Experimentation**: Test variations by modifying single cells +5. **Collaboration**: Others can understand and modify individual steps + +### Implementation Strategy + +- **util.load()** and **util.save()** create checkpoints +- **datacache/** stores intermediate results as JSON +- **%run util.py** ensures consistent environment +- **Markdown cells** provide context for each step + +### When to Save Data + +Save data when: +- Results took significant time to compute +- Data will be used by multiple subsequent cells +- Intermediate results are worth preserving +- Enabling cell re-runs would save time + +Don't save data when: +- Quick computations (< 1 second) +- Data only used in next cell +- Data is not JSON-serializable (save to nboutput/ instead) + +## Utility Function Guidelines + +Add functions to util.py when: +- Code is used by multiple cells +- Complex operations that need documentation +- Interactions with external systems (APIs, databases) +- Data transformations used repeatedly +- Model-specific operations + +Keep in notebooks when: +- Code is cell-specific analysis +- One-time exploratory code +- Visualization/plotting specific to that cell +- Simple operations that don't need abstraction + +## Quality Checklist + +Before marking complete, verify: +- ✅ notebooks/ directory exists with all 5 subdirectories +- ✅ util.py exists and has correct project name +- ✅ util.py contains NotebookUtil class with needed functions +- ✅ Every code cell starts with `%run util.py` +- ✅ Every code cell has preceding markdown explanation +- ✅ Data dependencies use util.load() +- ✅ Results saved with util.save() where appropriate +- ✅ Cells can run independently (tested) +- ✅ Input data in data/ directory +- ✅ Models in models/ directory +- ✅ Genomes in genomes/ directory +- ✅ Non-JSON output in nboutput/ directory +- ✅ JSON output handled by util.save() to datacache/ +- ✅ Markdown cells explain reasoning and purpose +- ✅ All imports in util.py, not scattered in cells +- ✅ Utility functions documented with docstrings + +## Error Handling + +Handle these scenarios gracefully: + +1. **Missing Dependencies**: If KBUtilLib or ModelSEEDpy not available, note in errors +2. **Existing Files**: Don't overwrite util.py if it already exists; validate instead +3. **Non-JSON Data**: Guide user to save to nboutput/ and load manually +4. **Complex Analysis**: Break into multiple cells for independence +5. **Long-Running Cells**: Emphasize saving intermediate results + +## Privacy and Security Considerations + +- Don't include API keys or credentials in util.py or notebooks +- Use environment variables or config files for sensitive data +- Document if manual credential setup is needed +- Don't log sensitive data in datacache/ files +- Note if data files contain sensitive information + +## Example Workflow + +For a typical metabolic modeling notebook: + +1. **Cell 1**: Load genome data from genomes/ + - Markdown: Explain which genome and why + - Code: Load, parse, save processed genome data + +2. **Cell 2**: Load COBRA model from models/ + - Markdown: Explain model selection and purpose + - Code: Load model, save to datacache + +3. **Cell 3**: Load experimental data from data/ + - Markdown: Describe experimental conditions + - Code: Load CSV, process, save data structure + +4. **Cell 4**: Run flux balance analysis + - Markdown: Explain FBA parameters and objectives + - Code: Load model, run FBA, save results + +5. **Cell 5**: Generate result tables + - Markdown: Describe what tables show + - Code: Load FBA results, create tables, save to nboutput/ + +Each cell independent, each with clear purpose, each properly cached. diff --git a/.claude/commands/kb-sdk-dev.md b/.claude/commands/kb-sdk-dev.md new file mode 100644 index 00000000..aa6f0566 --- /dev/null +++ b/.claude/commands/kb-sdk-dev.md @@ -0,0 +1,388 @@ +# KBase SDK Development Expert + +You are an expert on KBase SDK development. You have deep knowledge of: + +1. **KIDL Specification** - Writing and compiling KBase Interface Description Language spec files +2. **Module Structure** - Dockerfile, kbase.yml, spec.json, display.yaml, impl files +3. **Workspace Data Types** - All 223 KBase data types across 45 modules +4. **Narrative UI Integration** - Creating app interfaces with proper input/output widgets +5. **KBUtilLib Integration** - Using the shared utility library to avoid redundant code +6. **Best Practices** - Code organization, error handling, reporting, Docker optimization + +## Critical: KBUtilLib Usage + +**ALWAYS use KBUtilLib for common functionality.** The library at `/Users/chenry/Dropbox/Projects/KBUtilLib` provides: + +- `KBWSUtils` - Workspace operations (get/save objects) +- `KBGenomeUtils` - Genome parsing, feature extraction +- `KBModelUtils` - Metabolic model utilities +- `KBCallbackUtils` - Callback server handling +- `KBAnnotationUtils` - Annotation workflows +- `SharedEnvUtils` - Configuration and token management +- `MSBiochemUtils` - ModelSEED biochemistry access +- And many more utilities + +**In your Dockerfile, ALWAYS include:** +```dockerfile +# Checkout KBUtilLib for shared utilities +RUN cd /kb/module && \ + git clone https://github.com/cshenry/KBUtilLib.git && \ + cd KBUtilLib && \ + pip install -e . +``` + +**When writing new utility code:** If a function has general utility beyond this specific app, consider adding it to KBUtilLib instead. + +## Knowledge Loading + +**KBUtilLib Reference (read for available utilities):** +- `/Users/chenry/Dropbox/Projects/KBUtilLib/README.md` +- `/Users/chenry/Dropbox/Projects/KBUtilLib/src/kbutillib/` (module source) +- `/Users/chenry/Dropbox/Projects/KBUtilLib/docs/` (module documentation) + +**Workspace Data Types (read for type specifications):** +- `/Users/chenry/Dropbox/Projects/workspace_deluxe/agent-io/docs/WorkspaceDataTypes/all_types_list.json` +- `/Users/chenry/Dropbox/Projects/workspace_deluxe/agent-io/docs/WorkspaceDataTypes/individual_specs/` (individual type specs) +- `/Users/chenry/Dropbox/Projects/workspace_deluxe/agent-io/docs/WorkspaceDataTypes/all_type_specs.json` (full specs) + +**Online Documentation:** +- https://kbase.github.io/kb_sdk_docs/ (SDK documentation) +- https://kbase.github.io/kb_sdk_docs/references/KIDL_spec.html (KIDL reference) +- https://kbase.github.io/kb_sdk_docs/references/module_anatomy.html (module structure) + +## Quick Reference: Module Structure + +``` +MyModule/ +├── kbase.yml # Module metadata +├── Makefile # Build commands +├── Dockerfile # Container definition +├── MyModule.spec # KIDL specification +├── lib/ +│ └── MyModule/ +│ └── MyModuleImpl.py # Implementation code +├── ui/ +│ └── narrative/ +│ └── methods/ +│ └── run_my_app/ +│ ├── spec.json # Parameter mapping +│ └── display.yaml # UI labels/docs +├── test/ +│ └── MyModule_server_test.py # Unit tests +├── scripts/ +│ └── entrypoint.sh # Docker entrypoint +└── data/ # Reference data (<100MB) +``` + +## KIDL Spec File Format + +``` +/* +A KBase module: MyModule +Module description here. +*/ +module MyModule { + + /* Documentation for this type */ + typedef structure { + string workspace_name; + string genome_ref; + int min_length; + } RunAppParams; + + typedef structure { + string report_name; + string report_ref; + } RunAppResults; + + /* + Run the main application. + + This function does X, Y, Z. + */ + funcdef run_app(RunAppParams params) + returns (RunAppResults output) + authentication required; +}; +``` + +## Implementation File Pattern + +```python +#BEGIN_HEADER +import os +import json +from kbutillib import KBWSUtils, KBCallbackUtils, SharedEnvUtils + +class MyAppUtils(KBWSUtils, KBCallbackUtils, SharedEnvUtils): + """Custom utility class combining KBUtilLib modules.""" + pass +#END_HEADER + +class MyModule: + #BEGIN_CLASS_HEADER + #END_CLASS_HEADER + + def __init__(self, config): + #BEGIN_CONSTRUCTOR + self.callback_url = os.environ['SDK_CALLBACK_URL'] + self.scratch = config['scratch'] + self.utils = MyAppUtils(callback_url=self.callback_url) + #END_CONSTRUCTOR + pass + + def run_app(self, ctx, params): + #BEGIN run_app + # Validate inputs + workspace_name = params['workspace_name'] + genome_ref = params['genome_ref'] + + # Get data using KBUtilLib + genome_data = self.utils.get_object(workspace_name, genome_ref) + + # Do processing... + results = self.process_genome(genome_data) + + # Create report + report_info = self.utils.create_extended_report({ + 'message': 'Analysis complete', + 'workspace_name': workspace_name + }) + + return { + 'report_name': report_info['name'], + 'report_ref': report_info['ref'] + } + #END run_app +``` + +## spec.json Structure + +```json +{ + "ver": "1.0.0", + "authors": ["username"], + "contact": "email@example.com", + "categories": ["active"], + "widgets": { + "input": null, + "output": "no-display" + }, + "parameters": [ + { + "id": "genome_ref", + "optional": false, + "advanced": false, + "allow_multiple": false, + "default_values": [""], + "field_type": "text", + "text_options": { + "valid_ws_types": ["KBaseGenomes.Genome"] + } + }, + { + "id": "min_length", + "optional": true, + "advanced": true, + "allow_multiple": false, + "default_values": ["100"], + "field_type": "text", + "text_options": { + "validate_as": "int", + "min_int": 1 + } + } + ], + "behavior": { + "service-mapping": { + "url": "", + "name": "MyModule", + "method": "run_app", + "input_mapping": [ + { + "narrative_system_variable": "workspace", + "target_property": "workspace_name" + }, + { + "input_parameter": "genome_ref", + "target_property": "genome_ref", + "target_type_transform": "resolved-ref" + }, + { + "input_parameter": "min_length", + "target_property": "min_length", + "target_type_transform": "int" + } + ], + "output_mapping": [ + { + "service_method_output_path": [0, "report_name"], + "target_property": "report_name" + }, + { + "service_method_output_path": [0, "report_ref"], + "target_property": "report_ref" + } + ] + } + }, + "job_id_output_field": "docker" +} +``` + +## display.yaml Structure + +```yaml +name: Run My App +tooltip: | + Analyze genome data with custom parameters +screenshots: [] + +icon: icon.png + +suggestions: + apps: + related: [] + next: [] + methods: + related: [] + next: [] + +parameters: + genome_ref: + ui-name: | + Genome + short-hint: | + Select a genome to analyze + long-hint: | + Select a genome object from your workspace for analysis. + min_length: + ui-name: | + Minimum Length + short-hint: | + Minimum sequence length to consider + long-hint: | + Sequences shorter than this value will be filtered out. + +description: | +

Detailed description of what this app does.

+

Include information about inputs, outputs, and methodology.

+ +publications: + - pmid: 12345678 + display-text: | + Author et al. (2024) Paper title. Journal Name. + link: https://doi.org/xxx +``` + +## Dockerfile Pattern + +```dockerfile +FROM kbase/sdkbase2:python +MAINTAINER Your Name + +# Install system dependencies +RUN apt-get update && apt-get install -y \ + build-essential \ + && rm -rf /var/lib/apt/lists/* + +# Install Python dependencies +COPY requirements.txt /kb/module/requirements.txt +RUN pip install -r /kb/module/requirements.txt + +# CRITICAL: Install KBUtilLib for shared utilities +RUN cd /kb/module && \ + git clone https://github.com/cshenry/KBUtilLib.git && \ + cd KBUtilLib && \ + pip install -e . + +# Copy module files +COPY . /kb/module +WORKDIR /kb/module + +# Compile the module +RUN make all + +ENTRYPOINT ["./scripts/entrypoint.sh"] +CMD [] +``` + +## Common Data Types + +| Module | Type | Description | +|--------|------|-------------| +| KBaseGenomes | Genome | Annotated genome | +| KBaseGenomes | ContigSet | Set of contigs | +| KBaseFBA | FBAModel | Metabolic model | +| KBaseFBA | FBA | FBA solution | +| KBaseFBA | Media | Growth media | +| KBaseBiochem | Biochemistry | Compound/reaction DB | +| KBaseAssembly | Assembly | Genome assembly | +| KBaseRNASeq | RNASeqAlignment | RNA-seq alignment | +| KBaseSets | GenomeSet | Set of genomes | +| KBaseReport | Report | App output report | + +## Guidelines for Responding + +1. **Always recommend KBUtilLib** - Check if functionality exists there first +2. **Show complete examples** - KIDL specs, impl code, UI files together +3. **Explain compilation** - Remind about `make` after spec changes +4. **Include Dockerfile** - Show how to install dependencies +5. **Reference data types** - Point to specific workspace types when relevant + +## Response Format + +### For "how do I create" questions: +``` +### Overview +What we're building and why. + +### KIDL Spec +```kidl +// Complete spec file +``` + +### Implementation +```python +# Complete impl code +``` + +### UI Files +spec.json and display.yaml content + +### Dockerfile Updates +Any required additions + +### Build & Test +```bash +make +kb-sdk test +``` +``` + +### For data type questions: +``` +### Type: `ModuleName.TypeName` + +**Structure:** +``` +typedef structure { + field definitions... +} TypeName; +``` + +**Common Fields:** +- `field1` - Description +- `field2` - Description + +**Usage Example:** +```python +# How to work with this type +``` + +**Related Types:** List of related types +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/kb-sdk-dev/context/kbutillib-integration.md b/.claude/commands/kb-sdk-dev/context/kbutillib-integration.md new file mode 100644 index 00000000..051a9d9e --- /dev/null +++ b/.claude/commands/kb-sdk-dev/context/kbutillib-integration.md @@ -0,0 +1,287 @@ +# KBUtilLib Integration Guide + +## Overview + +KBUtilLib is a modular utility framework that should be used in ALL KBase SDK applications to avoid code duplication. The library provides composable utility classes that can be combined via multiple inheritance. + +**Repository:** `/Users/chenry/Dropbox/Projects/KBUtilLib` +**GitHub:** https://github.com/cshenry/KBUtilLib + +## Installation in Dockerfile + +**ALWAYS include KBUtilLib in your Dockerfile:** + +```dockerfile +# Install KBUtilLib for shared utilities +RUN cd /kb/module && \ + git clone https://github.com/cshenry/KBUtilLib.git && \ + cd KBUtilLib && \ + pip install -e . +``` + +## Available Modules + +### Core Foundation + +| Module | Purpose | +|--------|---------| +| `BaseUtils` | Logging, error handling, dependency management | +| `SharedEnvUtils` | Configuration files, authentication tokens | +| `NotebookUtils` | Jupyter integration, enhanced displays | + +### KBase Data Access + +| Module | Purpose | +|--------|---------| +| `KBWSUtils` | Workspace operations: get/save objects | +| `KBCallbackUtils` | Callback server handling for SDK apps | +| `KBSDKUtils` | SDK development utilities | + +### Analysis Utilities + +| Module | Purpose | +|--------|---------| +| `KBGenomeUtils` | Genome parsing, feature extraction, translation | +| `KBAnnotationUtils` | Gene/protein annotation workflows | +| `KBModelUtils` | Metabolic model analysis, FBA utilities | +| `MSBiochemUtils` | ModelSEED biochemistry database access | +| `KBReadsUtils` | Reads processing and QC | + +### External Integrations + +| Module | Purpose | +|--------|---------| +| `ArgoUtils` | Language model integration | +| `BVBRCUtils` | BV-BRC database access | +| `PatricWSUtils` | PATRIC workspace utilities | + +## Usage Patterns + +### Pattern 1: Single Module +```python +from kbutillib import KBWSUtils + +class MyApp: + def __init__(self, callback_url): + self.ws_utils = KBWSUtils(callback_url=callback_url) + + def run(self, params): + obj = self.ws_utils.get_object(params['workspace'], params['ref']) +``` + +### Pattern 2: Multiple Inheritance (Recommended) +```python +from kbutillib import KBWSUtils, KBGenomeUtils, KBCallbackUtils, SharedEnvUtils + +class MyAppUtils(KBWSUtils, KBGenomeUtils, KBCallbackUtils, SharedEnvUtils): + """Custom utility class combining needed modules.""" + pass + +class MyApp: + def __init__(self, callback_url): + self.utils = MyAppUtils(callback_url=callback_url) + + def run(self, params): + # Access all methods from combined classes + genome = self.utils.get_object(params['workspace'], params['ref']) + features = self.utils.extract_features_by_type(genome, 'CDS') + report = self.utils.create_extended_report({...}) +``` + +### Pattern 3: In Implementation File +```python +#BEGIN_HEADER +import os +from kbutillib import KBWSUtils, KBCallbackUtils, KBGenomeUtils + +class AppUtils(KBWSUtils, KBCallbackUtils, KBGenomeUtils): + """Combined utilities for this app.""" + pass +#END_HEADER + +class MyModule: + def __init__(self, config): + #BEGIN_CONSTRUCTOR + self.callback_url = os.environ['SDK_CALLBACK_URL'] + self.scratch = config['scratch'] + self.utils = AppUtils( + callback_url=self.callback_url, + scratch=self.scratch + ) + #END_CONSTRUCTOR + + def my_method(self, ctx, params): + #BEGIN my_method + workspace = params['workspace_name'] + + # Get genome using KBWSUtils + genome = self.utils.get_object(workspace, params['genome_ref']) + + # Parse genome using KBGenomeUtils + features = self.utils.extract_features_by_type(genome, 'CDS') + + # Create report using KBCallbackUtils + report_info = self.utils.create_extended_report({ + 'message': f'Found {len(features)} CDS features', + 'workspace_name': workspace + }) + + return [{ + 'report_name': report_info['name'], + 'report_ref': report_info['ref'] + }] + #END my_method +``` + +## Key Methods Reference + +### KBWSUtils + +```python +# Get a single object +obj_data = utils.get_object(workspace, object_ref) + +# Get object with metadata +obj, info = utils.get_object_with_info(workspace, object_ref) + +# Save an object +info = utils.save_object(workspace, obj_type, obj_name, obj_data) + +# List objects in workspace +objects = utils.list_objects(workspace, type_filter='KBaseGenomes.Genome') +``` + +### KBCallbackUtils + +```python +# Create a report +report_info = utils.create_extended_report({ + 'message': 'Analysis complete', + 'workspace_name': workspace, + 'objects_created': [{'ref': new_ref, 'description': 'My output'}], + 'file_links': [{'path': '/path/to/file.txt', 'name': 'results.txt'}], + 'html_links': [{'path': '/path/to/report.html', 'name': 'report'}] +}) + +# Download staging file +local_path = utils.download_staging_file(staging_file_path) + +# Upload file to shock +shock_id = utils.upload_to_shock(file_path) +``` + +### KBGenomeUtils + +```python +# Extract all features of a type +cds_features = utils.extract_features_by_type(genome_data, 'CDS') + +# Translate DNA sequence +protein = utils.translate_sequence(dna_seq) + +# Find ORFs in sequence +orfs = utils.find_orfs(sequence, min_length=100) + +# Parse genome object +genome_info = utils.parse_genome_object(genome_data) +``` + +### KBModelUtils + +```python +# Load model data +model_data = utils.get_model(workspace, model_ref) + +# Get reactions/metabolites +reactions = utils.get_model_reactions(model_data) +metabolites = utils.get_model_metabolites(model_data) + +# Check model consistency +issues = utils.validate_model(model_data) +``` + +### MSBiochemUtils + +```python +# Search compounds +compounds = utils.search_compounds("glucose") + +# Get reaction info +reaction = utils.get_reaction("rxn00001") + +# Search reactions by compound +reactions = utils.find_reactions_with_compound("cpd00001") +``` + +## When to Add Code to KBUtilLib + +If you're writing a function that: +1. Could be used in multiple KBase apps +2. Performs a common operation (parsing, converting, validating) +3. Wraps a KBase service in a cleaner way +4. Provides utility for a common data type + +**Consider adding it to KBUtilLib instead of your app.** + +### How to Add + +1. Identify which module it belongs in (or create new one) +2. Add the method to the appropriate class +3. Add tests in `tests/` +4. Update documentation +5. Push to GitHub +6. Update your app's Dockerfile to get latest + +## Configuration + +KBUtilLib can be configured via `config.yaml`: + +```yaml +kbase: + endpoint: https://kbase.us/services + token_env: KB_AUTH_TOKEN + +scratch: /kb/module/work/tmp + +logging: + level: INFO + format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" +``` + +Load configuration: +```python +utils = MyAppUtils(config_file='config.yaml') +# Or +utils = MyAppUtils(callback_url=url, scratch=scratch_dir) +``` + +## Error Handling + +KBUtilLib provides consistent error handling: + +```python +from kbutillib.base_utils import KBUtilLibError + +try: + result = utils.get_object(workspace, ref) +except KBUtilLibError as e: + # Handle KBUtilLib-specific errors + logger.error(f"KBUtilLib error: {e}") +except Exception as e: + # Handle other errors + logger.error(f"Unexpected error: {e}") +``` + +## Testing + +Test your integration: + +```python +import pytest +from kbutillib import KBWSUtils + +def test_workspace_access(): + utils = KBWSUtils(callback_url=test_callback_url) + obj = utils.get_object('test_workspace', 'test_object') + assert obj is not None +``` diff --git a/.claude/commands/kb-sdk-dev/context/kidl-reference.md b/.claude/commands/kb-sdk-dev/context/kidl-reference.md new file mode 100644 index 00000000..eb0c9771 --- /dev/null +++ b/.claude/commands/kb-sdk-dev/context/kidl-reference.md @@ -0,0 +1,250 @@ +# KIDL Specification Reference + +## Overview + +KIDL (KBase Interface Description Language) defines the interface for KBase modules. It specifies: +- Data types (typedefs) +- Function signatures +- Authentication requirements +- Documentation + +## Basic Types + +| Type | Description | Example | +|------|-------------|---------| +| `string` | Text value | `"hello"` | +| `int` | Integer | `42` | +| `float` | Floating point | `3.14` | +| `bool` | Boolean (0 or 1) | `1` | +| `UnspecifiedObject` | Any JSON object | `{}` | +| `list` | List of type T | `["a", "b"]` | +| `mapping` | Key-value pairs | `{"key": "value"}` | +| `tuple` | Fixed-length tuple | `["a", 1]` | + +## Type Definitions + +### Simple Typedef +```kidl +typedef string genome_ref; +typedef int workspace_id; +``` + +### Structure Typedef +```kidl +typedef structure { + string workspace_name; + string object_name; + string object_ref; +} ObjectInfo; +``` + +### Nested Structures +```kidl +typedef structure { + string id; + string name; + list aliases; +} Feature; + +typedef structure { + string id; + list features; +} Genome; +``` + +### Optional Fields +```kidl +typedef structure { + string required_field; + string optional_field; /* marked optional in spec.json */ +} MyParams; +``` + +### Mappings and Lists +```kidl +typedef mapping StringToIntMap; +typedef list StringList; +typedef mapping> StringToListMap; +``` + +### Tuple Types +```kidl +typedef tuple ObjectVersion; +``` + +## Function Definitions + +### Basic Function +```kidl +funcdef my_function(MyParams params) + returns (MyResults results) + authentication required; +``` + +### Function with Multiple Returns +```kidl +funcdef get_info(string ref) + returns (string name, int size, string type) + authentication required; +``` + +### Function with No Return +```kidl +funcdef log_event(string message) + returns () + authentication required; +``` + +### Function Documentation +```kidl +/* + * Short description of function. + * + * Longer description with details about what the function does, + * what parameters it expects, and what it returns. + * + * @param params The input parameters + * @return results The output results + */ +funcdef documented_function(Params params) + returns (Results results) + authentication required; +``` + +## Authentication Options + +```kidl +/* Requires valid KBase token */ +funcdef secure_func(Params p) returns (Results r) authentication required; + +/* No authentication needed */ +funcdef public_func(Params p) returns (Results r) authentication none; + +/* Optional authentication */ +funcdef optional_auth_func(Params p) returns (Results r) authentication optional; +``` + +## Complete Module Example + +```kidl +/* + * A KBase module: GenomeAnalyzer + * + * This module provides tools for analyzing genome data, + * including feature extraction and sequence analysis. + */ +module GenomeAnalyzer { + + /* Reference to a genome object */ + typedef string genome_ref; + + /* Reference to a workspace object */ + typedef string ws_ref; + + /* Feature information extracted from genome */ + typedef structure { + string feature_id; + string feature_type; + int start; + int end; + string strand; + string sequence; + } FeatureInfo; + + /* Input parameters for analyze_genome */ + typedef structure { + string workspace_name; + genome_ref genome_ref; + int min_feature_length; + list feature_types; + } AnalyzeGenomeParams; + + /* Results from analyze_genome */ + typedef structure { + string report_name; + ws_ref report_ref; + int features_analyzed; + list feature_summary; + } AnalyzeGenomeResults; + + /* Input for batch analysis */ + typedef structure { + string workspace_name; + list genome_refs; + } BatchAnalyzeParams; + + /* Results from batch analysis */ + typedef structure { + string report_name; + ws_ref report_ref; + mapping genome_feature_counts; + } BatchAnalyzeResults; + + /* + * Analyze a single genome for features. + * + * This function extracts and analyzes features from the specified + * genome, filtering by minimum length and feature type. + * + * @param params Analysis parameters including genome reference + * @return results Analysis results with report reference + */ + funcdef analyze_genome(AnalyzeGenomeParams params) + returns (AnalyzeGenomeResults results) + authentication required; + + /* + * Analyze multiple genomes in batch. + * + * @param params Batch parameters with list of genome references + * @return results Batch results with per-genome counts + */ + funcdef batch_analyze(BatchAnalyzeParams params) + returns (BatchAnalyzeResults results) + authentication required; +}; +``` + +## Compilation + +After modifying the spec file, always run: +```bash +make +``` + +This regenerates: +- `lib/MyModule/MyModuleImpl.py` - Implementation stubs +- `lib/MyModule/MyModuleServer.py` - Server code +- `lib/MyModule/MyModuleClient.py` - Client code + +## Common Patterns + +### Workspace References +```kidl +typedef string ws_ref; /* Format: "workspace/object" or "workspace/object/version" */ +``` + +### Report Output +```kidl +typedef structure { + string report_name; + string report_ref; +} ReportOutput; +``` + +### Standard Input Pattern +```kidl +typedef structure { + string workspace_name; + string workspace_id; /* Alternative to name */ + /* ... other params */ +} StandardParams; +``` + +## Tips + +1. **Keep types simple** - Complex nested structures are hard to maintain +2. **Use meaningful names** - `genome_ref` not `gr` or `ref1` +3. **Document everything** - Comments become API documentation +4. **Use lists for collections** - `list` not repeated fields +5. **Use mappings for lookups** - `mapping` for ID-based access diff --git a/.claude/commands/kb-sdk-dev/context/ui-spec-reference.md b/.claude/commands/kb-sdk-dev/context/ui-spec-reference.md new file mode 100644 index 00000000..5b80f60e --- /dev/null +++ b/.claude/commands/kb-sdk-dev/context/ui-spec-reference.md @@ -0,0 +1,421 @@ +# KBase Narrative UI Specification Reference + +## File Structure + +Each app method requires two files in `ui/narrative/methods//`: +- `spec.json` - Parameter mapping and validation +- `display.yaml` - UI labels, hints, and documentation + +## spec.json Reference + +### Complete Structure + +```json +{ + "ver": "1.0.0", + "authors": ["username"], + "contact": "email@example.com", + "categories": ["active"], + "widgets": { + "input": null, + "output": "no-display" + }, + "parameters": [...], + "behavior": { + "service-mapping": {...} + }, + "job_id_output_field": "docker" +} +``` + +### Parameter Types + +#### Text Input +```json +{ + "id": "my_string", + "optional": false, + "advanced": false, + "allow_multiple": false, + "default_values": ["default_value"], + "field_type": "text", + "text_options": { + "valid_ws_types": [] + } +} +``` + +#### Integer Input +```json +{ + "id": "my_int", + "optional": true, + "advanced": false, + "allow_multiple": false, + "default_values": ["10"], + "field_type": "text", + "text_options": { + "validate_as": "int", + "min_int": 1, + "max_int": 100 + } +} +``` + +#### Float Input +```json +{ + "id": "my_float", + "optional": true, + "advanced": true, + "allow_multiple": false, + "default_values": ["0.5"], + "field_type": "text", + "text_options": { + "validate_as": "float", + "min_float": 0.0, + "max_float": 1.0 + } +} +``` + +#### Workspace Object Selector +```json +{ + "id": "genome_ref", + "optional": false, + "advanced": false, + "allow_multiple": false, + "default_values": [""], + "field_type": "text", + "text_options": { + "valid_ws_types": ["KBaseGenomes.Genome"] + } +} +``` + +#### Multiple Object Types +```json +{ + "id": "input_ref", + "optional": false, + "advanced": false, + "allow_multiple": false, + "default_values": [""], + "field_type": "text", + "text_options": { + "valid_ws_types": [ + "KBaseGenomes.Genome", + "KBaseGenomeAnnotations.Assembly" + ] + } +} +``` + +#### Dropdown/Select +```json +{ + "id": "algorithm", + "optional": false, + "advanced": false, + "allow_multiple": false, + "default_values": ["default"], + "field_type": "dropdown", + "dropdown_options": { + "options": [ + {"value": "fast", "display": "Fast (less accurate)"}, + {"value": "default", "display": "Default"}, + {"value": "accurate", "display": "Accurate (slower)"} + ] + } +} +``` + +#### Checkbox (Boolean) +```json +{ + "id": "include_empty", + "optional": true, + "advanced": true, + "allow_multiple": false, + "default_values": ["0"], + "field_type": "checkbox", + "checkbox_options": { + "checked_value": 1, + "unchecked_value": 0 + } +} +``` + +#### Multiple Selection +```json +{ + "id": "genomes", + "optional": false, + "advanced": false, + "allow_multiple": true, + "default_values": [""], + "field_type": "text", + "text_options": { + "valid_ws_types": ["KBaseGenomes.Genome"] + } +} +``` + +#### Textarea (Multi-line) +```json +{ + "id": "description", + "optional": true, + "advanced": false, + "allow_multiple": false, + "default_values": [""], + "field_type": "textarea", + "textarea_options": { + "n_rows": 5 + } +} +``` + +#### Output Object Name +```json +{ + "id": "output_name", + "optional": false, + "advanced": false, + "allow_multiple": false, + "default_values": [""], + "field_type": "text", + "text_options": { + "valid_ws_types": [], + "is_output_name": true + } +} +``` + +### Behavior Section + +#### Input Mapping + +```json +"input_mapping": [ + { + "narrative_system_variable": "workspace", + "target_property": "workspace_name" + }, + { + "narrative_system_variable": "workspace_id", + "target_property": "workspace_id" + }, + { + "input_parameter": "genome_ref", + "target_property": "genome_ref", + "target_type_transform": "resolved-ref" + }, + { + "input_parameter": "min_length", + "target_property": "min_length", + "target_type_transform": "int" + }, + { + "input_parameter": "threshold", + "target_property": "threshold", + "target_type_transform": "float" + }, + { + "input_parameter": "genomes", + "target_property": "genome_refs", + "target_type_transform": "list" + } +] +``` + +#### Type Transforms + +| Transform | Description | +|-----------|-------------| +| `resolved-ref` | Converts object name to full reference | +| `ref` | Keep as reference string | +| `int` | Parse as integer | +| `float` | Parse as float | +| `string` | Keep as string (default) | +| `list` | List of resolved references | +| `list` | List of integers | + +#### Output Mapping + +```json +"output_mapping": [ + { + "service_method_output_path": [0, "report_name"], + "target_property": "report_name" + }, + { + "service_method_output_path": [0, "report_ref"], + "target_property": "report_ref" + }, + { + "narrative_system_variable": "workspace", + "target_property": "workspace_name" + } +] +``` + +### Widget Options + +```json +"widgets": { + "input": null, + "output": "no-display" +} +``` + +Common output widgets: +- `"no-display"` - No output display (use for report-based apps) +- `"kbaseReportView"` - Display KBase report + +## display.yaml Reference + +### Complete Structure + +```yaml +name: My App Name + +tooltip: | + Brief one-line description of the app + +screenshots: + - my_screenshot.png + +icon: icon.png + +suggestions: + apps: + related: + - related_app_1 + - related_app_2 + next: + - follow_up_app + methods: + related: [] + next: [] + +parameters: + genome_ref: + ui-name: | + Genome + short-hint: | + Select a genome object + long-hint: | + Select a genome object from your Narrative data panel. + The genome should have annotated features. + + min_length: + ui-name: | + Minimum Length + short-hint: | + Minimum feature length + long-hint: | + Features shorter than this value will be excluded + from the analysis. Default is 100 bp. + + output_name: + ui-name: | + Output Name + short-hint: | + Name for the output object + long-hint: | + Provide a name for the output object that will be + saved to your Narrative. + +description: | +

Full description of the app in HTML format.

+ +

Overview

+

What this app does and why you would use it.

+ +

Inputs

+
    +
  • Genome - A KBase genome object
    • +
  • Minimum Length - Filter threshold
    • +
+ +

Outputs

+

This app produces:

+
    +
  • A summary report
    • +
  • Downloadable data files
    • +
+ +

Algorithm

+

Description of the methodology used.

+ +publications: + - pmid: 12345678 + display-text: | + Author A, Author B (2024) Title of paper. Journal Name 10:123-456 + link: https://doi.org/10.xxxx/xxxxx + + - display-text: | + Software documentation at https://example.com + link: https://example.com +``` + +### Parameter Groups + +For complex apps, group related parameters: + +```yaml +parameter-groups: + basic_options: + ui-name: Basic Options + short-hint: Core parameters for the analysis + parameters: + - genome_ref + - output_name + + advanced_options: + ui-name: Advanced Options + short-hint: Fine-tune the analysis + parameters: + - min_length + - threshold + - algorithm +``` + +### Fixed Parameters + +Parameters not shown in UI but passed to service: + +```json +"fixed_parameters": [ + { + "target_property": "version", + "target_value": "1.0" + } +] +``` + +## Common Workspace Types for valid_ws_types + +| Type | Description | +|------|-------------| +| `KBaseGenomes.Genome` | Annotated genome | +| `KBaseGenomeAnnotations.Assembly` | Genome assembly | +| `KBaseSets.GenomeSet` | Set of genomes | +| `KBaseFBA.FBAModel` | Metabolic model | +| `KBaseFBA.FBA` | FBA solution | +| `KBaseFBA.Media` | Growth media | +| `KBaseRNASeq.RNASeqAlignment` | RNA-seq alignment | +| `KBaseMatrices.ExpressionMatrix` | Expression data | +| `KBaseFile.AssemblyFile` | Assembly file | +| `KBaseSets.ReadsSet` | Set of reads | + +## Tips + +1. **Use advanced: true** for optional parameters to reduce UI clutter +2. **Provide good defaults** - Apps should work with minimal configuration +3. **Write clear hints** - Users rely on short-hint for quick understanding +4. **Use dropdown for constrained choices** - Better than free text for enumerated options +5. **Group related parameters** - Improves usability for complex apps +6. **Include publications** - Helps users cite your work properly diff --git a/.claude/commands/kb-sdk-dev/context/workspace-datatypes.md b/.claude/commands/kb-sdk-dev/context/workspace-datatypes.md new file mode 100644 index 00000000..98b44dcf --- /dev/null +++ b/.claude/commands/kb-sdk-dev/context/workspace-datatypes.md @@ -0,0 +1,436 @@ +# KBase Workspace Data Types Reference + +## Overview + +KBase has **223 data types** across **45 modules**. This reference provides a quick lookup for the most commonly used types. + +**Full Specifications:** `/Users/chenry/Dropbox/Projects/workspace_deluxe/agent-io/docs/WorkspaceDataTypes/` +- `all_types_list.json` - Complete list of all types +- `all_type_specs.json` - Full specifications +- `individual_specs/` - Individual type specification files + +## Types by Module + +### Most Used Modules + +| Module | Type Count | Description | +|--------|-----------|-------------| +| KBaseFBA | 21 | Flux Balance Analysis, models | +| KBaseGenomes | 8 | Genomes, contigs, features | +| KBaseSets | 8 | Set collections | +| KBaseRNASeq | 13 | RNA sequencing | +| KBaseBiochem | 6 | Biochemistry, media | +| Communities | 31 | Metagenomics | + +--- + +## Core Genome Types (KBaseGenomes) + +### Genome +**Type:** `KBaseGenomes.Genome` + +The primary genome object containing annotations. + +**Key Fields:** +- `id` - Genome identifier +- `scientific_name` - Organism name +- `domain` - Bacteria, Archaea, Eukaryota +- `features` - List of genomic features +- `contigs` - Contig sequences (or reference) +- `source` - Data source (RefSeq, etc.) + +**Usage:** +```python +genome = utils.get_object(workspace, genome_ref) +features = genome.get('features', []) +``` + +### ContigSet +**Type:** `KBaseGenomes.ContigSet` + +Set of DNA contigs/sequences. + +**Key Fields:** +- `id` - ContigSet identifier +- `contigs` - List of contig objects +- `source` - Data source + +### Feature +**Type:** `KBaseGenomes.Feature` + +Individual genomic feature (gene, CDS, etc.). + +**Key Fields:** +- `id` - Feature identifier +- `type` - Feature type (CDS, gene, rRNA, etc.) +- `location` - Genomic coordinates +- `function` - Functional annotation +- `protein_translation` - Amino acid sequence + +### Pangenome +**Type:** `KBaseGenomes.Pangenome` + +Comparison of multiple genomes. + +--- + +## FBA and Modeling Types (KBaseFBA) + +### FBAModel +**Type:** `KBaseFBA.FBAModel` + +Metabolic model for flux balance analysis. + +**Key Fields:** +- `id` - Model identifier +- `name` - Model name +- `modelreactions` - List of reactions +- `modelcompounds` - List of metabolites +- `modelcompartments` - Compartments +- `biomasses` - Biomass reactions +- `genome_ref` - Reference to source genome + +**Usage:** +```python +model = utils.get_object(workspace, model_ref) +reactions = model.get('modelreactions', []) +``` + +### FBA +**Type:** `KBaseFBA.FBA` + +FBA simulation result. + +**Key Fields:** +- `id` - FBA identifier +- `fbamodel_ref` - Reference to model +- `media_ref` - Media used +- `objectiveValue` - Objective function value +- `FBAReactionVariables` - Reaction flux values +- `FBAMetaboliteVariables` - Metabolite values + +### Gapfilling +**Type:** `KBaseFBA.Gapfilling` + +Gapfilling solution. + +### ModelTemplate +**Type:** `KBaseFBA.ModelTemplate` + +Template for building models. + +### ModelComparison +**Type:** `KBaseFBA.ModelComparison` + +Comparison of multiple models. + +--- + +## Biochemistry Types (KBaseBiochem) + +### Media +**Type:** `KBaseBiochem.Media` + +Growth media definition. + +**Key Fields:** +- `id` - Media identifier +- `name` - Media name +- `mediacompounds` - List of compounds and concentrations +- `type` - Media type + +**Usage:** +```python +media = utils.get_object(workspace, media_ref) +compounds = media.get('mediacompounds', []) +``` + +### Biochemistry +**Type:** `KBaseBiochem.Biochemistry` + +Biochemistry database (compounds, reactions). + +### CompoundSet +**Type:** `KBaseBiochem.CompoundSet` + +Collection of compounds. + +--- + +## Set Types (KBaseSets) + +### GenomeSet +**Type:** `KBaseSets.GenomeSet` + +Set of genome references. + +**Key Fields:** +- `description` - Set description +- `items` - List of genome references with labels + +### AssemblySet +**Type:** `KBaseSets.AssemblySet` + +Set of assembly references. + +### ReadsSet +**Type:** `KBaseSets.ReadsSet` + +Set of reads library references. + +### ExpressionSet +**Type:** `KBaseSets.ExpressionSet` + +Set of expression data references. + +### SampleSet +**Type:** `KBaseSets.SampleSet` + +Set of sample references. + +--- + +## Assembly Types (KBaseAssembly) + +### PairedEndLibrary +**Type:** `KBaseAssembly.PairedEndLibrary` + +Paired-end reads library. + +### SingleEndLibrary +**Type:** `KBaseAssembly.SingleEndLibrary` + +Single-end reads library. + +### AssemblyReport +**Type:** `KBaseAssembly.AssemblyReport` + +Assembly quality report. + +--- + +## RNA-Seq Types (KBaseRNASeq) + +### RNASeqAlignment +**Type:** `KBaseRNASeq.RNASeqAlignment` + +Read alignment result. + +### RNASeqExpression +**Type:** `KBaseRNASeq.RNASeqExpression` + +Expression values from RNA-Seq. + +### RNASeqDifferentialExpression +**Type:** `KBaseRNASeq.RNASeqDifferentialExpression` + +Differential expression analysis. + +### RNASeqSampleSet +**Type:** `KBaseRNASeq.RNASeqSampleSet` + +Set of RNA-Seq samples. + +--- + +## Expression Types (KBaseFeatureValues) + +### ExpressionMatrix +**Type:** `KBaseFeatureValues.ExpressionMatrix` + +Gene expression matrix. + +**Key Fields:** +- `genome_ref` - Reference genome +- `data` - Expression values matrix +- `feature_ids` - Row identifiers (genes) +- `condition_ids` - Column identifiers (conditions) + +### FeatureClusters +**Type:** `KBaseFeatureValues.FeatureClusters` + +Clustered features from expression data. + +--- + +## Annotation Types (KBaseGenomeAnnotations) + +### Assembly +**Type:** `KBaseGenomeAnnotations.Assembly` + +Genome assembly (newer format). + +### GenomeAnnotation +**Type:** `KBaseGenomeAnnotations.GenomeAnnotation` + +Genome with annotations (newer format). + +### Taxon +**Type:** `KBaseGenomeAnnotations.Taxon` + +Taxonomic information. + +--- + +## Report Type (KBaseReport) + +### Report +**Type:** `KBaseReport.Report` + +Standard app output report. + +**Key Fields:** +- `text_message` - Report text +- `objects_created` - List of created objects +- `file_links` - Links to downloadable files +- `html_links` - Links to HTML reports +- `warnings` - Warning messages + +**Usage:** +```python +report_info = utils.create_extended_report({ + 'message': 'Analysis complete', + 'workspace_name': workspace, + 'objects_created': [{'ref': obj_ref, 'description': 'My output'}], + 'file_links': [{'path': '/path/to/file.txt', 'name': 'results.txt'}] +}) +``` + +--- + +## File Types (KBaseFile) + +### FileRef +**Type:** `KBaseFile.FileRef` + +Reference to a file in Shock/Blobstore. + +### PairedEndLibrary +**Type:** `KBaseFile.PairedEndLibrary` + +Paired-end library (file-based). + +### SingleEndLibrary +**Type:** `KBaseFile.SingleEndLibrary` + +Single-end library (file-based). + +--- + +## Matrix Types (KBaseMatrices) + +### ExpressionMatrix +**Type:** `KBaseMatrices.ExpressionMatrix` + +Expression data matrix (newer format). + +### AmpliconMatrix +**Type:** `KBaseMatrices.AmpliconMatrix` + +Amplicon abundance matrix. + +### MetaboliteMatrix +**Type:** `KBaseMatrices.MetaboliteMatrix` + +Metabolite abundance matrix. + +### FitnessMatrix +**Type:** `KBaseMatrices.FitnessMatrix` + +Gene fitness data. + +--- + +## Phenotype Types (KBasePhenotypes) + +### PhenotypeSet +**Type:** `KBasePhenotypes.PhenotypeSet` + +Set of phenotype measurements. + +**Key Fields:** +- `genome_ref` - Associated genome +- `phenotypes` - List of phenotypes with media/gene knockouts + +### PhenotypeSimulationSet +**Type:** `KBasePhenotypes.PhenotypeSimulationSet` + +Predicted phenotypes from FBA. + +--- + +## Tree Types (KBaseTrees) + +### Tree +**Type:** `KBaseTrees.Tree` + +Phylogenetic tree. + +### MSA +**Type:** `KBaseTrees.MSA` + +Multiple sequence alignment. + +--- + +## Complete Type List by Module + +### KBaseFBA (21 types) +- FBAModel, FBA, Gapfilling, Gapgeneration +- ModelTemplate, NewModelTemplate, ModelComparison +- FBAComparison, FBAModelSet +- FBAPathwayAnalysis, FBAPathwayAnalysisMultiple +- BooleanGeneExpressionData, BooleanGeneExpressionDataCollection +- Classifier, ClassifierResult, ClassifierTrainingSet +- ETC, EscherConfiguration, EscherMap +- PromConstraint, ReactionProbabilities +- ReactionSensitivityAnalysis, SubsystemAnnotation +- MissingRoleData, regulatory_network + +### KBaseGenomes (8 types) +- Genome, ContigSet, Feature +- GenomeComparison, GenomeDomainData +- MetagenomeAnnotation, Pangenome +- ProbabilisticAnnotation + +### KBaseSets (8 types) +- AssemblySet, DifferentialExpressionMatrixSet +- ExpressionSet, FeatureSetSet +- GenomeSet, ReadsAlignmentSet +- ReadsSet, SampleSet + +### KBaseBiochem (6 types) +- Biochemistry, BiochemistryStructures +- CompoundSet, Media, MediaSet +- MetabolicMap + +### KBaseRNASeq (13 types) +- RNASeqAlignment, RNASeqAlignmentSet +- RNASeqAnalysis, RNASeqExpression +- RNASeqExpressionSet, RNASeqSample +- RNASeqSampleAlignment, RNASeqSampleSet +- RNASeqDifferentialExpression +- RNASeqCuffdiffdifferentialExpression +- RNASeqCuffmergetranscriptome +- Bowtie2IndexV2, Bowtie2Indexes +- GFFAnnotation, ReferenceAnnotation +- AlignmentStatsResults, DifferentialExpressionStat +- cummerbund_output, cummerbundplot + +### KBaseCollections (6 types) +- FBAModelList, FBAModelSet +- FeatureList, FeatureSet +- GenomeList, GenomeSet + +--- + +## Type Reference Usage + +When you need detailed information about a specific type: + +```python +# Read the individual spec file +spec_path = f"/Users/chenry/Dropbox/Projects/workspace_deluxe/agent-io/docs/WorkspaceDataTypes/individual_specs/{module}_{type}.json" +``` + +Example spec file name: `KBaseGenomes_Genome.json` diff --git a/.claude/commands/kbutillib-dev.md b/.claude/commands/kbutillib-dev.md new file mode 100644 index 00000000..1f8dab61 --- /dev/null +++ b/.claude/commands/kbutillib-dev.md @@ -0,0 +1,279 @@ +# KBUtilLib Development Expert + +You are an expert on developing and contributing to KBUtilLib - a modular utility framework for scientific computing and bioinformatics. You have deep knowledge of: + +1. **Codebase Architecture** - Module hierarchy, inheritance patterns, file organization +2. **Development Workflow** - Adding modules, testing, documentation +3. **Dependency Management** - Git submodules, optional dependencies +4. **Code Standards** - Style, logging, provenance tracking +5. **Build and CI/CD** - UV packaging, pytest, GitHub Actions + +## Repository Location + +The KBUtilLib repository is located at: `/Users/chenry/Dropbox/Projects/KBUtilLib` + +## Knowledge Loading + +Before answering questions, load relevant context files: + +**Always load first:** +- Read context file: `kbutillib-dev:context:architecture` for the codebase structure + +**Load based on question topic:** +- For adding new modules: Read `kbutillib-dev:context:development-guide` +- For testing/CI: Read `kbutillib-dev:context:development-guide` + +**When needed for specific implementation:** +- `/Users/chenry/Dropbox/Projects/KBUtilLib/src/kbutillib/base_utils.py` - BaseUtils implementation +- `/Users/chenry/Dropbox/Projects/KBUtilLib/src/kbutillib/__init__.py` - Export structure +- `/Users/chenry/Dropbox/Projects/KBUtilLib/pyproject.toml` - Build configuration + +## Quick Reference + +### Repository Structure +``` +KBUtilLib/ +├── src/kbutillib/ # 37 Python utility modules (~16,800 lines) +│ ├── __init__.py # Exports and __all__ +│ ├── __main__.py # CLI entry point +│ ├── base_utils.py # Foundation class +│ ├── shared_env_utils.py # Configuration management +│ ├── kb_*.py # KBase-specific utilities +│ ├── ms_*.py # ModelSEED-specific utilities +│ └── *_utils.py # Other utilities +├── notebooks/ # 8 example Jupyter notebooks +├── examples/ # 3 example scripts +├── tests/ # pytest test suite +├── docs/ # Sphinx documentation +├── dependencies/ # Git submodules +├── config/ # Default configuration files +├── pyproject.toml # UV packaging configuration +└── DEPENDENCIES.md # Dependency documentation +``` + +### Technology Stack + +| Component | Technology | +|-----------|------------| +| Package Manager | `uv` (modern Python) | +| Testing | `pytest` | +| Linting | `ruff` | +| Type Checking | `mypy` | +| Documentation | Sphinx + MyST | +| CI/CD | GitHub Actions | + +### Module Naming Conventions + +| Prefix | Purpose | Example | +|--------|---------|---------| +| `kb_` | KBase-specific utilities | `kb_ws_utils.py`, `kb_genome_utils.py` | +| `ms_` | ModelSEED-specific utilities | `ms_biochem_utils.py`, `ms_fba_utils.py` | +| `*_utils` | General utilities | `notebook_utils.py`, `argo_utils.py` | + +### Inheritance Hierarchy + +``` +BaseUtils (foundation) +└── SharedEnvUtils (config + tokens) + ├── KBWSUtils (workspace) + │ ├── KBGenomeUtils + │ ├── KBAnnotationUtils + │ └── KBModelUtils + ├── MSBiochemUtils + ├── ArgoUtils + │ └── AICurationUtils + └── [other utilities] +``` + +### Creating a New Module + +**1. Create the module file:** +```python +# src/kbutillib/my_new_utils.py +from .base_utils import BaseUtils # or SharedEnvUtils, etc. + +class MyNewUtils(BaseUtils): + """Utility class for [purpose]. + + This class provides [description of functionality]. + + Example: + >>> utils = MyNewUtils() + >>> result = utils.my_method(param) + """ + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self.log_info("MyNewUtils initialized") + + def my_method(self, param1, param2=None): + """Description of method. + + Args: + param1: Description + param2: Optional description + + Returns: + Description of return value + + Raises: + ValueError: When param1 is invalid + """ + self.initialize_call("my_method", {"param1": param1}) + + # Validate arguments + self.validate_args({"param1": param1}, required=["param1"]) + + # Implementation + result = self._do_work(param1) + + self.log_info(f"Processed {param1}") + return result +``` + +**2. Add to exports:** +```python +# src/kbutillib/__init__.py +try: + from .my_new_utils import MyNewUtils +except ImportError: + MyNewUtils = None # Optional dependency not available + +__all__ = [ + # ... existing exports ... + "MyNewUtils", +] +``` + +**3. Write tests:** +```python +# tests/test_my_new_utils.py +import pytest +from kbutillib import MyNewUtils + +class TestMyNewUtils: + def test_initialization(self): + utils = MyNewUtils() + assert utils is not None + + def test_my_method(self): + utils = MyNewUtils() + result = utils.my_method("test_param") + assert result is not None +``` + +### Running Tests + +```bash +# Run all tests +uv run pytest + +# Run specific test file +uv run pytest tests/test_my_new_utils.py + +# Run with coverage +uv run pytest --cov=kbutillib + +# Run with verbose output +uv run pytest -v +``` + +### Linting and Type Checking + +```bash +# Run ruff linter +uv run ruff check src/ + +# Auto-fix issues +uv run ruff check --fix src/ + +# Type checking +uv run mypy src/kbutillib/ +``` + +### Common Development Tasks + +**Adding a dependency:** +```bash +# Add runtime dependency +uv add requests + +# Add development dependency +uv add --dev pytest-cov +``` + +**Working with git submodules:** +```bash +# Initialize submodules +git submodule update --init --recursive + +# Update submodules +git submodule update --remote +``` + +## Related Skills + +- `/kbutillib-expert` - For using KBUtilLib APIs +- `/modelseedpy-expert` - For ModelSEEDpy development +- `/kb-sdk-dev` - For KBase SDK development + +## Guidelines for Responding + +When helping developers: + +1. **Show complete implementations** - Provide working, tested code +2. **Follow conventions** - Use established naming and patterns +3. **Include tests** - Always suggest tests for new code +4. **Reference existing modules** - Point to similar implementations +5. **Load context files** - Use architecture documentation for guidance + +## Response Format + +### For "how do I add X" questions: +``` +### Adding [Feature] + +**Step 1: Create the module** +```python +# src/kbutillib/new_module.py +[complete implementation] +``` + +**Step 2: Update exports** +```python +# src/kbutillib/__init__.py +[export changes] +``` + +**Step 3: Add tests** +```python +# tests/test_new_module.py +[test implementation] +``` + +**Step 4: Update documentation** +- Add to docs/modules/ +- Update README if public API +``` + +### For architecture questions: +``` +### [Topic] Architecture + +**Overview:** +Brief explanation + +**Key Components:** +1. Component 1 - Purpose +2. Component 2 - Purpose + +**How They Connect:** +[Diagram or explanation] + +**Relevant Files:** +- `path/to/file.py` - Purpose +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/kbutillib-dev/context/architecture.md b/.claude/commands/kbutillib-dev/context/architecture.md new file mode 100644 index 00000000..ee2eb505 --- /dev/null +++ b/.claude/commands/kbutillib-dev/context/architecture.md @@ -0,0 +1,484 @@ +# KBUtilLib Architecture + +Comprehensive architecture documentation for KBUtilLib developers. + +## Core Design Philosophy + +KBUtilLib is built on three core principles: + +1. **Composability** - Utility classes combine via multiple inheritance +2. **Modularity** - Each utility is independent and self-contained +3. **Simplicity** - Focused classes with clear responsibilities + +## Repository Structure + +``` +KBUtilLib/ +├── src/kbutillib/ # Main package (37 modules, ~16,800 lines) +│ ├── __init__.py # Public exports and __all__ +│ ├── __main__.py # CLI entry point (Click-based) +│ │ +│ ├── # Foundation Layer +│ ├── base_utils.py # BaseUtils - logging, provenance, common methods +│ ├── shared_env_utils.py # SharedEnvUtils - config, tokens, env vars +│ │ +│ ├── # KBase Integration Layer +│ ├── kb_ws_utils.py # KBWSUtils - Workspace Service API +│ ├── kb_genome_utils.py # KBGenomeUtils - genome analysis +│ ├── kb_annotation_utils.py # KBAnnotationUtils - annotations +│ ├── kb_model_utils.py # KBModelUtils - metabolic models +│ ├── kb_reads_utils.py # KBReadsUtils - reads/assemblies +│ ├── kb_callback_utils.py # KBCallbackUtils - callback services +│ ├── kb_sdk_utils.py # KBSDKUtils - SDK development +│ ├── kb_uniprot_utils.py # KBUniProtUtils - UniProt API +│ ├── kb_plm_utils.py # KBPLMUtils - protein language models +│ │ +│ ├── # ModelSEED Integration Layer +│ ├── ms_biochem_utils.py # MSBiochemUtils - biochemistry DB +│ ├── ms_fba_utils.py # MSFBAUtils - FBA operations +│ ├── ms_reconstruction_utils.py # MSReconstructionUtils - model building +│ │ +│ ├── # AI/ML Layer +│ ├── argo_utils.py # ArgoUtils - LLM integration +│ ├── ai_curation_utils.py # AICurationUtils - AI curation +│ │ +│ ├── # External APIs Layer +│ ├── bvbrc_utils.py # BVBRCUtils - BV-BRC API +│ ├── patric_ws_utils.py # PatricWSUtils - PATRIC workspace +│ ├── rcsb_pdb_utils.py # RCSBPDBUtils - PDB structures +│ │ +│ ├── # Utility Layer +│ ├── notebook_utils.py # NotebookUtils - Jupyter enhancements +│ ├── escher_utils.py # EscherUtils - visualization +│ ├── skani_utils.py # SKANIUtils - genome distance +│ ├── model_standardization_utils.py # Model standardization +│ └── thermo_utils.py # ThermoUtils - thermodynamics +│ +├── notebooks/ # Example Jupyter notebooks +│ ├── ConfigureEnvironment.ipynb +│ ├── BVBRCGenomeConversion.ipynb +│ ├── AssemblyUploadDownload.ipynb +│ ├── SKANIGenomeDistance.ipynb +│ ├── ProteinLanguageModels.ipynb +│ ├── StoichiometryAnalysis.ipynb +│ ├── AICuration.ipynb +│ └── KBaseWorkspaceUtilities.ipynb +│ +├── examples/ # Standalone example scripts +│ ├── example_ai_curation_usage.py +│ ├── example_bvbrc_usage.py +│ └── example_skani_usage.py +│ +├── tests/ # pytest test suite +│ ├── conftest.py # Fixtures and configuration +│ ├── test_base_utils.py +│ └── test_*.py # Module-specific tests +│ +├── docs/ # Sphinx documentation +│ ├── conf.py # Sphinx configuration +│ ├── index.md # Documentation home +│ └── modules/ # Module documentation +│ +├── dependencies/ # Git submodules +│ ├── ModelSEEDpy/ +│ ├── ModelSEEDDatabase/ +│ ├── cobrakbase/ +│ └── cb_annotation_ontology_api/ +│ +├── config/ # Configuration templates +│ └── default_config.yaml +│ +├── pyproject.toml # UV/pip packaging +├── DEPENDENCIES.md # Dependency management docs +└── README.md # Project overview +``` + +## Module Hierarchy + +### Inheritance Tree + +``` +BaseUtils (base_utils.py) +│ +│ Core functionality: +│ - Logging (logger, log_info, log_debug, log_error) +│ - Provenance tracking (initialize_call, provenance list) +│ - Argument validation (validate_args) +│ - Data I/O (save_util_data, load_util_data) +│ +└── SharedEnvUtils (shared_env_utils.py) + │ + │ Configuration management: + │ - Config file loading (load_config, config object) + │ - Token management (get_token, set_token) + │ - Environment variables + │ + ├── KBWSUtils (kb_ws_utils.py) + │ │ KBase Workspace Service: + │ │ - Object retrieval/storage + │ │ - Type specs + │ │ - Workspace listing + │ │ + │ ├── KBGenomeUtils (kb_genome_utils.py) + │ │ Genome analysis: + │ │ - Feature extraction + │ │ - Sequence translation + │ │ - Annotation access + │ │ + │ ├── KBAnnotationUtils (kb_annotation_utils.py) + │ │ Annotation management: + │ │ - Ontology filtering + │ │ - EC/KEGG extraction + │ │ - Reaction mapping + │ │ + │ ├── KBModelUtils (kb_model_utils.py) + │ │ Model operations: + │ │ - Model retrieval + │ │ - Reaction/metabolite access + │ │ - Template management + │ │ + │ └── KBReadsUtils (kb_reads_utils.py) + │ Reads/assembly handling: + │ - Assembly objects + │ - ReadSet management + │ + ├── PatricWSUtils (patric_ws_utils.py) + │ PATRIC workspace access + │ + ├── MSBiochemUtils (ms_biochem_utils.py) + │ ModelSEED biochemistry: + │ - Compound/reaction search + │ - Database indexing + │ + ├── MSFBAUtils (ms_fba_utils.py) + │ FBA operations: + │ - Run FBA/pFBA/FVA + │ - Media configuration + │ - Constraints + │ + ├── MSReconstructionUtils (ms_reconstruction_utils.py) + │ Model reconstruction: + │ - Draft model building + │ - Gap-filling + │ + ├── ArgoUtils (argo_utils.py) + │ │ LLM integration: + │ │ - Query Argo API + │ │ - Model selection + │ │ + │ └── AICurationUtils (ai_curation_utils.py) + │ AI curation: + │ - Reaction curation + │ - Caching + │ + ├── BVBRCUtils (bvbrc_utils.py) + │ BV-BRC API access + │ + ├── KBUniProtUtils (kb_uniprot_utils.py) + │ UniProt REST API + │ + ├── RCSBPDBUtils (rcsb_pdb_utils.py) + │ PDB structure access + │ + ├── KBPLMUtils (kb_plm_utils.py) + │ Protein language models + │ + └── SKANIUtils (skani_utils.py) + Genome distance computation + +# Independent utilities (not in SharedEnvUtils hierarchy) +├── NotebookUtils (notebook_utils.py) - inherits BaseUtils +├── EscherUtils (escher_utils.py) - inherits BaseUtils +├── ModelStandardizationUtils - inherits BaseUtils +└── ThermoUtils - inherits BaseUtils +``` + +## Configuration System + +### Config File Priority +1. Explicit `config_file` parameter +2. `~/kbutillib_config.yaml` (user config) +3. `config/default_config.yaml` (repository defaults) + +### Config File Structure +```yaml +# Example configuration +kbase: + endpoint: https://kbase.us/services + workspace_url: https://kbase.us/services/ws + auth_service_url: https://kbase.us/services/auth + +argo: + endpoint: https://api.cels.anl.gov/argo/api/v1 + default_model: gpt4o + +modelseed: + database_path: ~/ModelSEEDDatabase + +logging: + level: INFO +``` + +### Token Management +```python +# Tokens stored per namespace +tokens = { + "kbase": "...", + "argo": "...", + "custom": "..." +} + +# Environment variables also checked: +# KBASE_AUTH_TOKEN, ARGO_API_TOKEN +``` + +## Provenance System + +Every method call can be tracked for reproducibility: + +```python +class MyUtils(BaseUtils): + def my_method(self, param1): + # Start tracking + self.initialize_call("my_method", {"param1": param1}) + + # Method implementation + result = self._do_work(param1) + + # Logged to provenance list + return result + +# Access provenance +utils = MyUtils() +utils.my_method("test") +print(utils.provenance) +# [{"method": "my_method", "params": {"param1": "test"}, "timestamp": "..."}] +``` + +## Export System + +The `__init__.py` uses try/except for optional dependencies: + +```python +# src/kbutillib/__init__.py + +# Always available +from .base_utils import BaseUtils +from .shared_env_utils import SharedEnvUtils + +# Optional - may have missing dependencies +try: + from .kb_plm_utils import KBPLMUtils +except ImportError: + KBPLMUtils = None + +__all__ = [ + "BaseUtils", + "SharedEnvUtils", + "KBPLMUtils", # May be None + # ... +] +``` + +## Dependency Architecture + +### Core Dependencies (always required) +- `requests` - HTTP client +- `pyyaml` - Configuration files +- `python-dotenv` - Environment variables + +### Optional Dependencies (graceful degradation) +- `pandas` - DataFrame operations +- `cobra` - Constraint-based modeling +- `ipywidgets` - Notebook widgets +- `escher` - Pathway visualization + +### Git Submodule Dependencies +Located in `dependencies/`: +- `ModelSEEDpy` - Metabolic modeling +- `ModelSEEDDatabase` - Biochemistry data +- `cobrakbase` - KBase COBRA extensions +- `cb_annotation_ontology_api` - Annotation ontology + +## Testing Architecture + +### Test Organization +``` +tests/ +├── conftest.py # Shared fixtures +├── test_base_utils.py # BaseUtils tests +├── test_kb_ws_utils.py # KBWSUtils tests +└── ... +``` + +### Fixtures (conftest.py) +```python +import pytest + +@pytest.fixture +def mock_config(): + return { + "kbase": {"endpoint": "https://test.kbase.us"} + } + +@pytest.fixture +def base_utils(): + return BaseUtils() +``` + +### Test Patterns +```python +class TestBaseUtils: + def test_initialization(self, base_utils): + assert base_utils.logger is not None + + def test_logging(self, base_utils): + base_utils.log_info("Test message") + # Assert logging occurred + + @pytest.mark.parametrize("level", ["DEBUG", "INFO", "WARNING"]) + def test_log_levels(self, base_utils, level): + base_utils.logger.setLevel(level) + assert base_utils.logger.level == getattr(logging, level) +``` + +## CI/CD Pipeline + +### GitHub Actions Workflow +```yaml +# .github/workflows/ci.yml +name: CI +on: [push, pull_request] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive + + - name: Install uv + uses: astral-sh/setup-uv@v1 + + - name: Run tests + run: uv run pytest + + - name: Lint + run: uv run ruff check src/ + + - name: Type check + run: uv run mypy src/kbutillib/ +``` + +## Build System + +### pyproject.toml Structure +```toml +[project] +name = "kbutillib" +version = "0.1.0" +description = "Modular utility framework for bioinformatics" +requires-python = ">=3.9" +dependencies = [ + "requests>=2.28", + "pyyaml>=6.0", + "python-dotenv>=1.0", +] + +[project.optional-dependencies] +dev = [ + "pytest>=7.0", + "pytest-cov>=4.0", + "ruff>=0.1", + "mypy>=1.0", +] +notebooks = [ + "jupyter>=1.0", + "ipywidgets>=8.0", + "itables>=1.0", +] + +[project.scripts] +kbutillib = "kbutillib.__main__:main" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.ruff] +line-length = 100 +target-version = "py39" + +[tool.mypy] +python_version = "3.9" +ignore_missing_imports = true +``` + +## Documentation System + +### Sphinx + MyST Configuration +```python +# docs/conf.py +extensions = [ + "myst_parser", + "sphinx.ext.autodoc", + "sphinx.ext.napoleon", +] + +myst_enable_extensions = [ + "colon_fence", + "deflist", +] +``` + +### Documentation Structure +``` +docs/ +├── conf.py # Sphinx config +├── index.md # Home page +├── getting-started.md # Quick start +├── modules/ # Module docs +│ ├── base_utils.md +│ ├── kb_ws_utils.md +│ └── ... +└── api/ # Auto-generated API docs +``` + +## Error Handling Patterns + +### Standard Error Pattern +```python +def my_method(self, required_param, optional_param=None): + # Validate required parameters + if not required_param: + raise ValueError("required_param is required") + + try: + result = self._external_call(required_param) + except ConnectionError as e: + self.log_error(f"Connection failed: {e}") + raise + except Exception as e: + self.log_error(f"Unexpected error: {e}") + raise + + return result +``` + +### Graceful Degradation +```python +try: + from .optional_module import OptionalFeature + HAS_OPTIONAL = True +except ImportError: + OptionalFeature = None + HAS_OPTIONAL = False + +class MyUtils(BaseUtils): + def optional_method(self): + if not HAS_OPTIONAL: + self.log_warning("Optional feature not available") + return None + return OptionalFeature.do_something() +``` diff --git a/.claude/commands/kbutillib-dev/context/development-guide.md b/.claude/commands/kbutillib-dev/context/development-guide.md new file mode 100644 index 00000000..ca7b7a55 --- /dev/null +++ b/.claude/commands/kbutillib-dev/context/development-guide.md @@ -0,0 +1,605 @@ +# KBUtilLib Development Guide + +Step-by-step guide for developing and contributing to KBUtilLib. + +## Development Setup + +### Prerequisites +- Python 3.9+ +- `uv` package manager (recommended) +- Git with submodule support + +### Initial Setup +```bash +# Clone repository +git clone https://github.com/your-org/KBUtilLib.git +cd KBUtilLib + +# Initialize submodules +git submodule update --init --recursive + +# Install with development dependencies +uv sync --all-extras + +# Verify installation +uv run python -c "import kbutillib; print(kbutillib.__all__)" +``` + +### Alternative: pip Installation +```bash +# Create virtual environment +python -m venv .venv +source .venv/bin/activate + +# Install in editable mode +pip install -e ".[dev,notebooks]" +``` + +## Adding a New Utility Module + +### Step 1: Plan Your Module + +Before coding, determine: +1. **Purpose**: What does this module do? +2. **Parent class**: Which utility to inherit from? +3. **Dependencies**: What external libraries needed? +4. **API surface**: What methods will be public? + +### Step 2: Create the Module File + +```python +# src/kbutillib/my_new_utils.py +"""My New Utilities module. + +This module provides utilities for [purpose]. + +Example: + >>> from kbutillib import MyNewUtils + >>> utils = MyNewUtils() + >>> result = utils.my_method("param") +""" + +from typing import Any, Dict, List, Optional +from .shared_env_utils import SharedEnvUtils # or appropriate parent + + +class MyNewUtils(SharedEnvUtils): + """Utility class for [purpose]. + + This class provides methods for [description]. + + Attributes: + some_attribute: Description of attribute. + + Example: + >>> utils = MyNewUtils(config_file="my_config.yaml") + >>> utils.my_method("test") + """ + + def __init__( + self, + config_file: Optional[str] = None, + **kwargs: Any + ) -> None: + """Initialize MyNewUtils. + + Args: + config_file: Optional path to configuration file. + **kwargs: Additional arguments passed to parent class. + """ + super().__init__(config_file=config_file, **kwargs) + self.log_info("MyNewUtils initialized") + + # Module-specific initialization + self._cache: Dict[str, Any] = {} + + def my_method( + self, + required_param: str, + optional_param: Optional[int] = None + ) -> Dict[str, Any]: + """Brief description of method. + + Longer description explaining what the method does, + any important behaviors, and edge cases. + + Args: + required_param: Description of this parameter. + optional_param: Description of optional parameter. + Defaults to None. + + Returns: + Dictionary containing: + - key1: Description + - key2: Description + + Raises: + ValueError: When required_param is empty. + ConnectionError: When external service unavailable. + + Example: + >>> utils = MyNewUtils() + >>> result = utils.my_method("test", optional_param=5) + >>> print(result["key1"]) + """ + # Track method call for provenance + self.initialize_call("my_method", { + "required_param": required_param, + "optional_param": optional_param + }) + + # Validate arguments + if not required_param: + raise ValueError("required_param cannot be empty") + + # Check cache + cache_key = f"my_method:{required_param}" + if cache_key in self._cache: + self.log_debug(f"Cache hit: {cache_key}") + return self._cache[cache_key] + + # Main implementation + self.log_info(f"Processing: {required_param}") + result = self._do_actual_work(required_param, optional_param) + + # Cache result + self._cache[cache_key] = result + + return result + + def _do_actual_work( + self, + param1: str, + param2: Optional[int] + ) -> Dict[str, Any]: + """Internal method for actual processing. + + Private methods start with underscore and don't need + full docstrings unless complex. + """ + # Implementation here + return {"key1": param1, "key2": param2 or 0} +``` + +### Step 3: Add to Package Exports + +```python +# src/kbutillib/__init__.py + +# Add import with try/except for optional dependencies +try: + from .my_new_utils import MyNewUtils +except ImportError as e: + import logging + logging.getLogger(__name__).debug(f"MyNewUtils not available: {e}") + MyNewUtils = None + +# Add to __all__ +__all__ = [ + # ... existing exports ... + "MyNewUtils", +] +``` + +### Step 4: Write Tests + +```python +# tests/test_my_new_utils.py +"""Tests for MyNewUtils.""" + +import pytest +from kbutillib import MyNewUtils + + +class TestMyNewUtils: + """Test suite for MyNewUtils class.""" + + @pytest.fixture + def utils(self): + """Create MyNewUtils instance for testing.""" + return MyNewUtils() + + def test_initialization(self, utils): + """Test that utils initializes correctly.""" + assert utils is not None + assert hasattr(utils, 'logger') + + def test_initialization_with_config(self, tmp_path): + """Test initialization with config file.""" + config_file = tmp_path / "config.yaml" + config_file.write_text("key: value\n") + utils = MyNewUtils(config_file=str(config_file)) + assert utils is not None + + def test_my_method_basic(self, utils): + """Test my_method with valid input.""" + result = utils.my_method("test_param") + assert result is not None + assert "key1" in result + assert result["key1"] == "test_param" + + def test_my_method_with_optional(self, utils): + """Test my_method with optional parameter.""" + result = utils.my_method("test", optional_param=42) + assert result["key2"] == 42 + + def test_my_method_empty_param_raises(self, utils): + """Test that empty required_param raises ValueError.""" + with pytest.raises(ValueError, match="cannot be empty"): + utils.my_method("") + + def test_my_method_caching(self, utils): + """Test that results are cached.""" + result1 = utils.my_method("cached_param") + result2 = utils.my_method("cached_param") + assert result1 is result2 # Same object from cache + + @pytest.mark.parametrize("param,expected", [ + ("a", "a"), + ("test", "test"), + ("longer_param", "longer_param"), + ]) + def test_my_method_various_inputs(self, utils, param, expected): + """Test my_method with various inputs.""" + result = utils.my_method(param) + assert result["key1"] == expected + + +class TestMyNewUtilsIntegration: + """Integration tests for MyNewUtils.""" + + @pytest.mark.integration + def test_with_real_service(self): + """Test integration with external service.""" + pytest.skip("Requires external service") +``` + +### Step 5: Add Documentation + +```markdown +# docs/modules/my_new_utils.md + +# MyNewUtils + +Utility class for [purpose]. + +## Overview + +MyNewUtils provides functionality for [description]. It inherits from +SharedEnvUtils, giving access to configuration and token management. + +## Installation + +MyNewUtils is included in the base kbutillib package: + +```python +from kbutillib import MyNewUtils +``` + +## Quick Start + +```python +from kbutillib import MyNewUtils + +# Initialize +utils = MyNewUtils() + +# Basic usage +result = utils.my_method("parameter") +print(result) +``` + +## Configuration + +MyNewUtils uses the standard configuration system: + +```yaml +# ~/kbutillib_config.yaml +my_new_utils: + setting1: value1 + setting2: value2 +``` + +## API Reference + +### my_method(required_param, optional_param=None) + +Brief description. + +**Parameters:** +- `required_param` (str): Description +- `optional_param` (int, optional): Description + +**Returns:** +- dict: Result dictionary with keys... + +**Example:** +```python +result = utils.my_method("test", optional_param=5) +``` + +## Composition Examples + +MyNewUtils can be combined with other utilities: + +```python +from kbutillib import MyNewUtils, KBGenomeUtils + +class CustomTools(MyNewUtils, KBGenomeUtils): + pass + +tools = CustomTools() +``` + +## See Also + +- [SharedEnvUtils](shared_env_utils.md) - Parent class +- [Related utility](related.md) +``` + +### Step 6: Update README + +Add a brief mention to the main README.md if the module is significant. + +## Running Tests + +### Full Test Suite +```bash +# Run all tests +uv run pytest + +# With coverage report +uv run pytest --cov=kbutillib --cov-report=html + +# Verbose output +uv run pytest -v + +# Stop on first failure +uv run pytest -x +``` + +### Specific Tests +```bash +# Single file +uv run pytest tests/test_my_new_utils.py + +# Single test class +uv run pytest tests/test_my_new_utils.py::TestMyNewUtils + +# Single test +uv run pytest tests/test_my_new_utils.py::TestMyNewUtils::test_my_method_basic + +# Pattern matching +uv run pytest -k "my_method" +``` + +### Test Markers +```bash +# Skip slow tests +uv run pytest -m "not slow" + +# Only integration tests +uv run pytest -m integration +``` + +## Code Quality + +### Linting with Ruff +```bash +# Check for issues +uv run ruff check src/ + +# Auto-fix issues +uv run ruff check --fix src/ + +# Format code +uv run ruff format src/ +``` + +### Type Checking with MyPy +```bash +# Check types +uv run mypy src/kbutillib/ + +# Specific file +uv run mypy src/kbutillib/my_new_utils.py +``` + +### Pre-commit Hooks +```bash +# Install hooks +uv run pre-commit install + +# Run manually +uv run pre-commit run --all-files +``` + +## Working with Dependencies + +### Adding Runtime Dependencies +```bash +# Add to project +uv add requests + +# With version constraint +uv add "requests>=2.28" +``` + +### Adding Development Dependencies +```bash +uv add --dev pytest-cov +``` + +### Adding Optional Dependencies +Edit pyproject.toml: +```toml +[project.optional-dependencies] +ml = ["torch>=2.0", "transformers>=4.0"] +``` + +### Managing Git Submodules +```bash +# Initialize +git submodule update --init --recursive + +# Update to latest +git submodule update --remote + +# Check status +git submodule status +``` + +## Common Development Patterns + +### Inheriting from BaseUtils +```python +from .base_utils import BaseUtils + +class MyUtils(BaseUtils): + def my_method(self): + self.initialize_call("my_method", {}) + self.log_info("Starting...") + # work + self.log_debug("Details...") + return result +``` + +### Inheriting from SharedEnvUtils +```python +from .shared_env_utils import SharedEnvUtils + +class MyUtils(SharedEnvUtils): + def my_method(self): + # Access config + setting = self.get_config_value("my.setting") + + # Access token + token = self.get_token("kbase") + + return result +``` + +### HTTP Client Pattern +```python +import requests +from .shared_env_utils import SharedEnvUtils + +class MyAPIUtils(SharedEnvUtils): + def __init__(self, **kwargs): + super().__init__(**kwargs) + self._session = requests.Session() + self._base_url = self.get_config_value("my_api.endpoint") + + def _request(self, method, endpoint, **kwargs): + """Make authenticated request.""" + url = f"{self._base_url}/{endpoint}" + headers = kwargs.pop("headers", {}) + + token = self.get_token("my_api") + if token: + headers["Authorization"] = f"Bearer {token}" + + response = self._session.request( + method, url, headers=headers, **kwargs + ) + response.raise_for_status() + return response.json() + + def get_resource(self, resource_id): + return self._request("GET", f"resources/{resource_id}") +``` + +### Caching Pattern +```python +from functools import lru_cache +from .base_utils import BaseUtils + +class CachedUtils(BaseUtils): + def __init__(self, **kwargs): + super().__init__(**kwargs) + self._cache = {} + + def get_with_cache(self, key): + if key not in self._cache: + self._cache[key] = self._fetch(key) + return self._cache[key] + + def clear_cache(self): + self._cache.clear() + + @lru_cache(maxsize=100) + def get_with_lru(self, key): + """Uses built-in LRU cache.""" + return self._fetch(key) +``` + +## Debugging Tips + +### Enable Debug Logging +```python +import logging +logging.getLogger("kbutillib").setLevel(logging.DEBUG) + +utils = MyUtils() +utils.my_method("test") # Will show debug output +``` + +### Interactive Debugging +```python +# Add breakpoint +import pdb; pdb.set_trace() + +# Or use pytest debugging +# pytest --pdb # Drop into debugger on failure +# pytest --pdb-first # Drop on first failure +``` + +### Inspect Provenance +```python +utils = MyUtils() +utils.my_method("test") +utils.another_method("param") + +# See all tracked calls +for call in utils.provenance: + print(f"{call['method']}: {call['params']}") +``` + +## Pull Request Checklist + +Before submitting a PR: + +- [ ] All tests pass: `uv run pytest` +- [ ] Linting passes: `uv run ruff check src/` +- [ ] Types check: `uv run mypy src/kbutillib/` +- [ ] New code has tests +- [ ] Docstrings follow Google style +- [ ] Module added to `__init__.py` +- [ ] README updated if adding major feature +- [ ] No secrets in code + +## Troubleshooting + +### Import Errors +```python +# Check if module is available +from kbutillib import MyUtils +if MyUtils is None: + print("Module not available - check dependencies") +``` + +### Submodule Issues +```bash +# Reset submodules +git submodule deinit -f --all +git submodule update --init --recursive +``` + +### Test Discovery Issues +```bash +# Check pytest can find tests +uv run pytest --collect-only + +# Verbose collection +uv run pytest --collect-only -v +``` diff --git a/.claude/commands/kbutillib-expert.md b/.claude/commands/kbutillib-expert.md new file mode 100644 index 00000000..3ebbf8e5 --- /dev/null +++ b/.claude/commands/kbutillib-expert.md @@ -0,0 +1,188 @@ +# KBUtilLib Expert + +You are an expert on KBUtilLib - a modular utility framework for scientific computing and bioinformatics developed at Argonne National Laboratory. You have deep knowledge of: + +1. **Composable Utility Architecture** - How utility classes combine via multiple inheritance +2. **KBase Integration** - Workspace, annotation, and genome utilities +3. **ModelSEED Integration** - Biochemistry database, FBA, and model utilities +4. **AI Curation** - LLM-powered reaction and annotation curation +5. **Data Analysis Workflows** - Notebooks and practical usage patterns + +## Repository Location + +The KBUtilLib repository is located at: `/Users/chenry/Dropbox/Projects/KBUtilLib` + +## Knowledge Loading + +Before answering questions, load relevant context files: + +**Always load first:** +- Read context file: `kbutillib-expert:context:module-reference` for the complete module hierarchy + +**Load based on question topic:** +- For API usage questions: Read `kbutillib-expert:context:api-summary` +- For workflow/pattern questions: Read `kbutillib-expert:context:patterns` + +**When needed for specific modules:** +- `/Users/chenry/Dropbox/Projects/KBUtilLib/src/kbutillib/.py` - Read source code for detailed API + +## Quick Reference + +### Core Concept: Composable Inheritance + +KBUtilLib is designed around **mixing and matching utility classes** via multiple inheritance: + +```python +from kbutillib import KBWSUtils, KBGenomeUtils, MSBiochemUtils, NotebookUtils + +# Combine exactly what you need +class MyAnalysisTools(KBGenomeUtils, MSBiochemUtils, NotebookUtils): + pass + +# Use it +tools = MyAnalysisTools() +genome = tools.get_genome(workspace_id, genome_ref) +compounds = tools.search_compounds("glucose") +``` + +### Module Categories + +| Category | Modules | Purpose | +|----------|---------|---------| +| **Foundation** | `BaseUtils`, `SharedEnvUtils` | Logging, config, provenance | +| **Data Access** | `KBWSUtils`, `PatricWSUtils` | KBase/PATRIC workspace access | +| **Genomics** | `KBGenomeUtils`, `KBAnnotationUtils` | Genome/annotation analysis | +| **Biochemistry** | `MSBiochemUtils` | ModelSEED compound/reaction DB | +| **Modeling** | `KBModelUtils`, `MSFBAUtils`, `MSReconstructionUtils` | Metabolic models and FBA | +| **Visualization** | `EscherUtils` | Escher pathway visualization | +| **External APIs** | `BVBRCUtils`, `KBUniProtUtils`, `RCSBPDBUtils` | External database access | +| **AI/ML** | `ArgoUtils`, `AICurationUtils`, `KBPLMUtils` | LLM and protein language models | +| **Utilities** | `NotebookUtils`, `SKANIUtils` | Notebook enhancements, genome distance | + +### Configuration Pattern + +```python +from kbutillib import SharedEnvUtils + +class MyTools(SharedEnvUtils): + pass + +tools = MyTools() +# Configuration loaded from (priority order): +# 1. Explicit config_file parameter +# 2. ~/kbutillib_config.yaml (user config) +# 3. repo/config/default_config.yaml + +# Access config values +value = tools.config.get("section.key") + +# Get authentication tokens +kbase_token = tools.get_token("kbase") +argo_token = tools.get_token("argo") +``` + +### Common Workflows + +**1. Fetch and Analyze a Genome:** +```python +from kbutillib import KBWSUtils, KBGenomeUtils + +class GenomeTools(KBWSUtils, KBGenomeUtils): + pass + +tools = GenomeTools() +genome = tools.get_genome(workspace_id, "MyGenome/1") +features = tools.get_features_by_type(genome, "CDS") +proteins = tools.translate_features(features) +``` + +**2. Search ModelSEED Database:** +```python +from kbutillib import MSBiochemUtils + +biochem = MSBiochemUtils() +compounds = biochem.search_compounds("ATP") +reactions = biochem.search_reactions("glycolysis") +reaction = biochem.get_reaction("rxn00001") +``` + +**3. Run FBA on a Model:** +```python +from kbutillib import KBModelUtils, MSFBAUtils + +class FBATools(KBModelUtils, MSFBAUtils): + pass + +tools = FBATools() +model = tools.get_model(workspace_id, "MyModel/1") +tools.set_media(model, "Complete") +solution = tools.run_fba(model) +``` + +**4. AI-Powered Curation:** +```python +from kbutillib import AICurationUtils + +curator = AICurationUtils() +result = curator.curate_reaction_direction(reaction_data) +categories = curator.categorize_stoichiometry(reaction) +``` + +## Related Skills + +- `/kbutillib-dev` - For developing and contributing to KBUtilLib +- `/modelseedpy-expert` - For ModelSEEDpy-specific questions +- `/msmodelutl-expert` - For MSModelUtil class from cobrakbase +- `/kb-sdk-dev` - For KBase SDK development + +## Guidelines for Responding + +When helping users: + +1. **Show composable patterns** - Demonstrate how to combine utility classes +2. **Provide working code** - Include complete, runnable examples +3. **Reference notebooks** - Point to example notebooks when relevant +4. **Explain the hierarchy** - Show which base classes provide which methods +5. **Load context files** - Use the context loading mechanism for detailed info + +## Response Format + +### For "how do I" questions: +``` +### Approach + +Brief explanation of which utility classes to use. + +**Utility Classes Needed:** +- `ClassName` - What it provides + +**Example Code:** +```python +# Complete working example +``` + +**See Also:** +- Notebook: `notebooks/RelevantNotebook.ipynb` +``` + +### For "what does X do" questions: +``` +### Module: X + +**Purpose:** Brief description + +**Key Methods:** +- `method_name(params)` - Description +- `another_method(params)` - Description + +**Inherits From:** BaseClass + +**Example:** +```python +# Usage example +``` +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/kbutillib-expert/context/api-summary.md b/.claude/commands/kbutillib-expert/context/api-summary.md new file mode 100644 index 00000000..c29c656f --- /dev/null +++ b/.claude/commands/kbutillib-expert/context/api-summary.md @@ -0,0 +1,456 @@ +# KBUtilLib API Quick Reference + +Quick reference for the most commonly used APIs in KBUtilLib. + +## Configuration & Setup + +### Initialize with Configuration +```python +from kbutillib import SharedEnvUtils + +class MyTools(SharedEnvUtils): + pass + +# Default configuration (auto-loads from standard locations) +tools = MyTools() + +# Explicit configuration file +tools = MyTools(config_file="/path/to/config.yaml") + +# With explicit token +tools = MyTools(kbase_token="YOUR_TOKEN") +``` + +### Configuration File Format (YAML) +```yaml +# ~/kbutillib_config.yaml +kbase: + endpoint: https://kbase.us/services + workspace_url: https://kbase.us/services/ws + +argo: + endpoint: https://api.cels.anl.gov/argo/api/v1 + +modelseed: + database_path: /path/to/ModelSEEDDatabase + +logging: + level: INFO +``` + +### Token Management +```python +# Get tokens +kbase_token = tools.get_token("kbase") +argo_token = tools.get_token("argo") + +# Set tokens programmatically +tools.set_token("NEW_TOKEN", namespace="kbase") + +# Tokens can also be set via environment variables: +# KBASE_AUTH_TOKEN, ARGO_API_TOKEN +``` + +## KBase Workspace Operations + +### Retrieve Objects +```python +from kbutillib import KBWSUtils + +ws = KBWSUtils() + +# Get any object +obj = ws.get_object(workspace_id=12345, object_ref="MyObject/1") + +# Get with specific version +obj = ws.get_object(12345, "MyObject/3") + +# Get object info (metadata only) +info = ws.get_object_info(12345, "MyObject") +# Returns: [id, name, type, save_date, version, ...] +``` + +### List Objects +```python +# List all objects in workspace +objects = ws.list_objects(workspace_id=12345) + +# Filter by type +genomes = ws.list_objects(12345, type_filter="KBaseGenomes.Genome") +models = ws.list_objects(12345, type_filter="KBaseFBA.FBAModel") +``` + +### Save Objects +```python +# Save object to workspace +ws.save_object( + workspace_id=12345, + obj_type="KBaseGenomes.Genome", + data=genome_data, + name="MyNewGenome" +) +``` + +## Genome Operations + +### Get and Analyze Genomes +```python +from kbutillib import KBWSUtils, KBGenomeUtils + +class GenomeTools(KBWSUtils, KBGenomeUtils): + pass + +tools = GenomeTools() + +# Get genome +genome = tools.get_genome(workspace_id=12345, genome_ref="MyGenome/1") + +# Get all features +features = tools.get_features(genome) + +# Filter by type +cds_features = tools.get_features_by_type(genome, "CDS") +rna_features = tools.get_features_by_type(genome, "rRNA") + +# Filter by function +transporters = tools.get_features_by_function(genome, "transport") +``` + +### Sequence Translation +```python +# Translate single feature +protein_seq = tools.translate_feature(feature) + +# Bulk translation +proteins = tools.translate_features(cds_features) +# Returns: {feature_id: protein_sequence, ...} + +# Get contig sequences +contigs = tools.get_contig_sequences(genome) +# Returns: {contig_id: sequence, ...} +``` + +## Annotation Operations + +### Access Annotations +```python +from kbutillib import KBWSUtils, KBAnnotationUtils + +class AnnotationTools(KBWSUtils, KBAnnotationUtils): + pass + +tools = AnnotationTools() + +# Get all annotations +annotations = tools.get_annotations(genome) + +# Get annotation history +events = tools.get_annotation_events(genome) + +# Filter by ontology +ec_annotations = tools.filter_annotations_by_ontology(annotations, "EC") +kegg_annotations = tools.filter_annotations_by_ontology(annotations, "KEGG") +``` + +### Extract Identifiers +```python +# Get EC numbers for a feature +ec_numbers = tools.get_ec_numbers(feature) +# Returns: ["1.1.1.1", "2.3.4.5"] + +# Get KEGG IDs +kegg_ids = tools.get_kegg_ids(feature) +# Returns: ["K00001", "K00002"] + +# Map function to reactions +reactions = tools.map_function_to_reactions("alcohol dehydrogenase") +``` + +## ModelSEED Biochemistry + +### Search Compounds +```python +from kbutillib import MSBiochemUtils + +biochem = MSBiochemUtils() + +# Search by name +compounds = biochem.search_compounds("glucose") +# Returns list of matching compounds + +# Search by ID +atp = biochem.get_compound("cpd00002") + +# Search by formula +c6h12o6 = biochem.search_by_formula("C6H12O6") + +# Search by structure +compound = biochem.search_by_inchikey("WQZGKKKJIJFFOK-...") +``` + +### Search Reactions +```python +# Search by name/equation +reactions = biochem.search_reactions("glycolysis") + +# Get specific reaction +reaction = biochem.get_reaction("rxn00001") + +# Get stoichiometry +stoich = biochem.get_reaction_stoichiometry("rxn00001") +# Returns: {"cpd00001": -1, "cpd00002": 1, ...} +``` + +## Metabolic Model Operations + +### Get and Analyze Models +```python +from kbutillib import KBWSUtils, KBModelUtils + +class ModelTools(KBWSUtils, KBModelUtils): + pass + +tools = ModelTools() + +# Get model +model = tools.get_model(workspace_id=12345, model_ref="MyModel/1") + +# Get model components +reactions = tools.get_model_reactions(model) +metabolites = tools.get_model_metabolites(model) +genes = tools.get_model_genes(model) +``` + +### Modify Models +```python +# Add reaction +tools.add_reaction(model, reaction_data) + +# Remove reaction +tools.remove_reaction(model, "rxn00001_c0") + +# Get reconstruction template +template = tools.get_template("GramNegative") +``` + +## FBA Operations + +### Run FBA +```python +from kbutillib import KBModelUtils, MSFBAUtils + +class FBATools(KBModelUtils, MSFBAUtils): + pass + +tools = FBATools() + +# Basic FBA +solution = tools.run_fba(model) +print(f"Objective value: {solution.objective_value}") + +# FBA with specific media +tools.set_media(model, "Complete") +solution = tools.run_fba(model) + +# Parsimonious FBA +solution = tools.run_pfba(model) +``` + +### Flux Analysis +```python +# Flux Variability Analysis +fva_results = tools.run_fva(model) +# Returns: {reaction_id: (min_flux, max_flux), ...} + +# FVA on specific reactions +fva_results = tools.run_fva(model, reactions=["rxn00001", "rxn00002"]) + +# Set fraction of optimum constraint +tools.set_fraction_of_optimum(model, 0.9) # 90% of optimal +fva_results = tools.run_fva(model) +``` + +### Constraints and Objectives +```python +# Set objective +tools.set_objective(model, "bio1") # Biomass reaction + +# Add flux constraint +tools.add_constraint(model, { + "reaction": "rxn00001", + "lower_bound": 0, + "upper_bound": 10 +}) +``` + +## AI Curation + +### Reaction Curation +```python +from kbutillib import AICurationUtils + +curator = AICurationUtils() + +# Curate reaction direction +result = curator.curate_reaction_direction(reaction_data) +# Returns direction analysis with confidence + +# Categorize stoichiometry +category = curator.categorize_stoichiometry(reaction) +# Returns: "balanced", "transport", "exchange", etc. + +# Evaluate equivalence +are_equivalent = curator.evaluate_equivalence(reaction1, reaction2) +``` + +### Gene-Reaction Assessment +```python +# Validate gene-reaction association +assessment = curator.assess_gene_reaction(gene_info, reaction_info) +# Returns confidence score and reasoning +``` + +### Caching +```python +# Results are automatically cached +# Check cache +cached = curator.get_cached_result(query_hash) + +# Clear cache +curator.clear_cache() +``` + +## External APIs + +### BV-BRC +```python +from kbutillib import BVBRCUtils + +bvbrc = BVBRCUtils() + +# Get genome +genome = bvbrc.get_bvbrc_genome("83332.12") + +# Search genomes +genomes = bvbrc.search_bvbrc_genomes("Escherichia coli") + +# Convert to KBase format +kb_genome = bvbrc.convert_to_kbase(genome) +``` + +### UniProt +```python +from kbutillib import KBUniProtUtils + +uniprot = KBUniProtUtils() + +# Get entry +entry = uniprot.get_uniprot_entry("P00533") + +# Get sequence +sequence = uniprot.get_protein_sequence("P00533") + +# Search +results = uniprot.search_uniprot("alcohol dehydrogenase AND organism:ecoli") + +# ID mapping +mapped = uniprot.map_ids(["P00533", "P12345"], from_db="UniProtKB_AC", to_db="PDB") +``` + +### PDB +```python +from kbutillib import RCSBPDBUtils + +pdb = RCSBPDBUtils() + +# Get structure info +structure = pdb.get_structure("1HHO") + +# Search +structures = pdb.search_structures("hemoglobin") + +# Get sequence +sequence = pdb.get_sequence("1HHO", chain="A") +``` + +## Visualization + +### Escher Maps +```python +from kbutillib import EscherUtils + +escher = EscherUtils() + +# Create map +map = escher.create_map(reaction_list) + +# Visualize FBA results +escher.visualize_fluxes(map, fba_solution) + +# Custom coloring +escher.set_reaction_colors(map, { + "rxn00001": "red", + "rxn00002": "blue" +}) + +# Save +escher.save_map(map, "my_map.json") +``` + +## Notebook Utilities + +### Enhanced Display +```python +from kbutillib import NotebookUtils + +nb = NotebookUtils() + +# Display DataFrame with interactive features +nb.display_dataframe(df) + +# Progress bar +with nb.create_progress_bar(total=100) as pbar: + for i in range(100): + # do work + pbar.update(1) +``` + +### Data Objects with Provenance +```python +# Create tracked data object +data = nb.DataObject( + name="my_analysis", + data=result_data, + source="genome_analysis", + params={"param1": "value1"} +) + +# Access provenance +print(data.provenance) +``` + +## Error Handling + +```python +try: + genome = tools.get_genome(12345, "NonExistentGenome") +except ValueError as e: + print(f"Object not found: {e}") + +try: + result = curator.curate_reaction_direction(bad_data) +except Exception as e: + tools.log_error(f"Curation failed: {e}") +``` + +## Logging + +```python +# Set log level +tools.logger.setLevel("DEBUG") + +# Log messages +tools.log_info("Starting analysis") +tools.log_debug("Processing item 1 of 100") +tools.log_error("Failed to process item") +``` diff --git a/.claude/commands/kbutillib-expert/context/module-reference.md b/.claude/commands/kbutillib-expert/context/module-reference.md new file mode 100644 index 00000000..2b0ec1be --- /dev/null +++ b/.claude/commands/kbutillib-expert/context/module-reference.md @@ -0,0 +1,364 @@ +# KBUtilLib Module Reference + +Complete reference for all utility modules in KBUtilLib. + +## Module Hierarchy + +``` +BaseUtils (foundation) +├── SharedEnvUtils (configuration & tokens) +│ ├── KBWSUtils (KBase Workspace) +│ │ ├── KBGenomeUtils (genome analysis) +│ │ ├── KBAnnotationUtils (annotations) +│ │ ├── KBModelUtils (metabolic models) +│ │ └── KBReadsUtils (reads/assemblies) +│ ├── PatricWSUtils (PATRIC Workspace) +│ ├── MSBiochemUtils (ModelSEED biochemistry) +│ ├── MSFBAUtils (FBA analysis) +│ ├── MSReconstructionUtils (model reconstruction) +│ ├── ArgoUtils (LLM integration) +│ ├── AICurationUtils (AI curation) +│ ├── BVBRCUtils (BV-BRC API) +│ ├── KBUniProtUtils (UniProt API) +│ ├── RCSBPDBUtils (PDB structures) +│ ├── KBPLMUtils (protein language models) +│ └── SKANIUtils (genome distance) +├── EscherUtils (visualization) +├── NotebookUtils (Jupyter enhancements) +├── ModelStandardizationUtils (model standardization) +└── ThermoUtils (thermodynamics) +``` + +## Foundation Layer + +### BaseUtils +**Location:** `src/kbutillib/base_utils.py` +**Purpose:** Base class providing core functionality for all utilities. + +**Key Methods:** +- `initialize_call(method_name, params)` - Start provenance tracking +- `log_info(message)` / `log_debug(message)` / `log_error(message)` - Logging +- `validate_args(required_args, provided_args)` - Argument validation +- `save_util_data(filename, data)` - Save JSON data +- `load_util_data(filename)` - Load JSON data + +**Attributes:** +- `logger` - Configured logging instance +- `provenance` - List of tracked method calls + +### SharedEnvUtils +**Location:** `src/kbutillib/shared_env_utils.py` (~500 lines) +**Inherits:** BaseUtils +**Purpose:** Configuration and authentication management. + +**Key Methods:** +- `load_config(config_file=None)` - Load YAML configuration +- `get_token(namespace="kbase")` - Get authentication token +- `set_token(token, namespace="kbase")` - Set authentication token +- `get_config_value(key)` - Get config value by dot-notation path + +**Configuration Priority:** +1. Explicit `config_file` parameter +2. `~/kbutillib_config.yaml` (user config) +3. `repo/config/default_config.yaml` (defaults) + +**Token Namespaces:** +- `kbase` - KBase authentication +- `argo` - Argo LLM service +- Custom namespaces as needed + +## Data Access Layer + +### KBWSUtils +**Location:** `src/kbutillib/kb_ws_utils.py` (~595 lines) +**Inherits:** SharedEnvUtils +**Purpose:** KBase Workspace Service API access. + +**Key Methods:** +- `get_object(workspace_id, object_ref)` - Retrieve any workspace object +- `save_object(workspace_id, obj_type, data, name)` - Save object +- `list_objects(workspace_id, type_filter=None)` - List workspace objects +- `get_object_info(workspace_id, object_ref)` - Get object metadata +- `get_type_spec(type_name)` - Get type specification + +**Workspace Reference Formats:** +- `ws_id/obj_name` - By workspace ID and name +- `ws_id/obj_name/version` - Specific version +- `obj_id` - Direct object ID + +### PatricWSUtils +**Location:** `src/kbutillib/patric_ws_utils.py` (~609 lines) +**Inherits:** SharedEnvUtils +**Purpose:** PATRIC/BV-BRC Workspace access. + +**Key Methods:** +- `get_patric_object(path)` - Get object from PATRIC workspace +- `list_patric_workspace(path)` - List workspace contents +- `get_patric_genome(genome_id)` - Get genome object +- `get_patric_model(model_id)` - Get metabolic model + +## Bioinformatics Analysis Layer + +### KBGenomeUtils +**Location:** `src/kbutillib/kb_genome_utils.py` (~770 lines) +**Inherits:** KBWSUtils +**Purpose:** Genome data analysis and manipulation. + +**Key Methods:** +- `get_genome(workspace_id, genome_ref)` - Retrieve genome object +- `get_features(genome)` - Get all features +- `get_features_by_type(genome, feature_type)` - Filter by type (CDS, rRNA, etc.) +- `get_features_by_function(genome, function_pattern)` - Filter by function +- `translate_feature(feature)` - DNA to protein translation +- `translate_features(features)` - Bulk translation +- `get_contig_sequences(genome)` - Get contig sequences + +**Feature Types:** +- `CDS` - Coding sequences +- `rRNA`, `tRNA` - RNA features +- `gene`, `mRNA` - Gene annotations + +### KBAnnotationUtils +**Location:** `src/kbutillib/kb_annotation_utils.py` (~940 lines) +**Inherits:** KBWSUtils +**Purpose:** Gene and protein annotation management. + +**Key Methods:** +- `get_annotations(genome)` - Get all annotations +- `get_annotation_events(genome)` - Get annotation event history +- `filter_annotations_by_ontology(annotations, ontology)` - Filter by source +- `get_ec_numbers(feature)` - Extract EC numbers +- `get_kegg_ids(feature)` - Extract KEGG identifiers +- `map_function_to_reactions(function)` - Map functional role to reactions + +**Supported Ontologies:** +- EC numbers +- KEGG +- MetaCyc +- UniProt +- GO terms + +### MSBiochemUtils +**Location:** `src/kbutillib/ms_biochem_utils.py` (~859 lines) +**Inherits:** SharedEnvUtils +**Purpose:** ModelSEED biochemistry database access. + +**Key Methods:** +- `search_compounds(query)` - Search compounds by name/ID/formula +- `search_reactions(query)` - Search reactions by name/equation +- `get_compound(compound_id)` - Get compound by ID (cpd00001) +- `get_reaction(reaction_id)` - Get reaction by ID (rxn00001) +- `get_reaction_stoichiometry(reaction_id)` - Get stoichiometry dict +- `search_by_formula(formula)` - Find compounds by molecular formula +- `search_by_inchikey(inchikey)` - Find by structure + +**ID Formats:** +- Compounds: `cpd#####` (e.g., cpd00001 = H2O) +- Reactions: `rxn#####` (e.g., rxn00001) + +## Metabolic Modeling Layer + +### KBModelUtils +**Location:** `src/kbutillib/kb_model_utils.py` (~696 lines) +**Inherits:** KBWSUtils +**Purpose:** Metabolic model analysis and manipulation. + +**Key Methods:** +- `get_model(workspace_id, model_ref)` - Get FBA model +- `get_model_reactions(model)` - List model reactions +- `get_model_metabolites(model)` - List model metabolites +- `get_model_genes(model)` - List model genes +- `add_reaction(model, reaction)` - Add reaction to model +- `remove_reaction(model, reaction_id)` - Remove reaction +- `get_template(template_name)` - Get reconstruction template + +**Model Object Structure:** +- `modelreactions` - List of reactions +- `modelcompounds` - List of metabolites +- `modelgenes` - List of genes +- `biomasses` - Biomass objective functions + +### MSFBAUtils +**Location:** `src/kbutillib/ms_fba_utils.py` (~685 lines) +**Inherits:** SharedEnvUtils +**Purpose:** Flux Balance Analysis operations. + +**Key Methods:** +- `run_fba(model, media=None)` - Run FBA simulation +- `run_pfba(model)` - Parsimonious FBA +- `run_fva(model, reactions=None)` - Flux Variability Analysis +- `set_media(model, media_id)` - Configure growth media +- `set_objective(model, reaction_id)` - Set objective function +- `add_constraint(model, constraint)` - Add flux constraint +- `set_fraction_of_optimum(model, fraction)` - Set optimality fraction + +**Media Options:** +- `Complete` - Rich media +- `Minimal` - Minimal glucose +- Custom media definitions + +### MSReconstructionUtils +**Location:** `src/kbutillib/ms_reconstruction_utils.py` (~757 lines) +**Inherits:** SharedEnvUtils +**Purpose:** Genome-scale model reconstruction. + +**Key Methods:** +- `build_model_from_genome(genome, template)` - Build draft model +- `gapfill_model(model, media)` - Gap-fill model +- `prune_model(model)` - Remove unnecessary reactions +- `integrate_phenotypes(model, phenotype_data)` - Add phenotype constraints + +### EscherUtils +**Location:** `src/kbutillib/escher_utils.py` (~1,089 lines) +**Inherits:** BaseUtils +**Purpose:** Escher pathway map visualization. + +**Key Methods:** +- `create_map(reactions, layout=None)` - Create Escher map +- `visualize_fluxes(map, fba_solution)` - Overlay flux values +- `set_reaction_colors(map, color_dict)` - Custom reaction coloring +- `save_map(map, filename)` - Save to file +- `load_map(filename)` - Load existing map + +## External API Layer + +### BVBRCUtils +**Location:** `src/kbutillib/bvbrc_utils.py` (~463 lines) +**Inherits:** SharedEnvUtils +**Purpose:** BV-BRC (formerly PATRIC) API access. + +**Key Methods:** +- `get_bvbrc_genome(genome_id)` - Fetch genome by ID +- `search_bvbrc_genomes(query)` - Search genomes +- `get_genome_features(genome_id)` - Get genome features +- `get_genome_sequences(genome_id)` - Get contig sequences +- `convert_to_kbase(bvbrc_genome)` - Convert to KBase format + +### KBUniProtUtils +**Location:** `src/kbutillib/kb_uniprot_utils.py` (~651 lines) +**Inherits:** SharedEnvUtils +**Purpose:** UniProt REST API integration. + +**Key Methods:** +- `get_uniprot_entry(accession)` - Get entry by accession +- `search_uniprot(query)` - Search UniProt +- `get_protein_sequence(accession)` - Get protein sequence +- `get_annotations(accession)` - Get functional annotations +- `map_ids(ids, from_db, to_db)` - ID mapping + +### RCSBPDBUtils +**Location:** `src/kbutillib/rcsb_pdb_utils.py` (~598 lines) +**Inherits:** SharedEnvUtils +**Purpose:** RCSB PDB structure database access. + +**Key Methods:** +- `get_structure(pdb_id)` - Get PDB structure +- `search_structures(query)` - Search PDB +- `get_sequence(pdb_id, chain)` - Get chain sequence +- `get_experimental_info(pdb_id)` - Get experimental metadata + +## AI/ML Layer + +### ArgoUtils +**Location:** `src/kbutillib/argo_utils.py` +**Inherits:** SharedEnvUtils +**Purpose:** Argo LLM service integration. + +**Key Methods:** +- `query_argo(prompt, model="gpt4o")` - Send LLM query +- `query_argo_async(prompt, model)` - Async query with polling +- `get_available_models()` - List available models + +**Available Models:** +- `gpt4o` - GPT-4o +- `gpt3mini` - GPT-3.5 Mini +- `o1`, `o1-mini`, `o3-mini` - Reasoning models + +### AICurationUtils +**Location:** `src/kbutillib/ai_curation_utils.py` (~897 lines) +**Inherits:** ArgoUtils +**Purpose:** AI-powered biochemistry curation. + +**Key Methods:** +- `curate_reaction_direction(reaction)` - Determine reaction reversibility +- `categorize_stoichiometry(reaction)` - Categorize reaction type +- `evaluate_equivalence(rxn1, rxn2)` - Check reaction equivalence +- `assess_gene_reaction(gene, reaction)` - Validate gene-reaction association +- `get_cached_result(query_hash)` - Get cached curation result +- `cache_result(query_hash, result)` - Cache curation result + +**Backends:** +- `argo` - Argo LLM service +- `claude` - Claude Code integration + +### KBPLMUtils +**Location:** `src/kbutillib/kb_plm_utils.py` (~804 lines) +**Inherits:** SharedEnvUtils +**Purpose:** Protein language model integration. + +**Key Methods:** +- `search_homologs(sequence)` - PLM-based homology search +- `create_blast_db(sequences)` - Create BLAST database +- `search_blast(query, db)` - BLAST search +- `get_uniprot_for_hits(hits)` - Fetch UniProt info for hits + +## Utility Layer + +### NotebookUtils +**Location:** `src/kbutillib/notebook_utils.py` (~703 lines) +**Inherits:** BaseUtils +**Purpose:** Jupyter notebook enhancements. + +**Key Classes:** +- `DataObject` - Standardized data object with provenance + +**Key Methods:** +- `display_dataframe(df)` - Enhanced DataFrame display +- `create_progress_bar(total)` - Progress bar +- `display_html(html)` - Rich HTML output + +### SKANIUtils +**Location:** `src/kbutillib/skani_utils.py` (~800 lines) +**Inherits:** SharedEnvUtils +**Purpose:** Fast genome distance computation using SKANI. + +**Key Methods:** +- `compute_ani(genome1, genome2)` - Compute ANI between genomes +- `create_sketch_db(genomes)` - Create SKANI sketch database +- `search_db(query_genome, db)` - Search against database +- `clear_cache()` - Clear sketch cache + +## Import Patterns + +```python +# Import specific utilities +from kbutillib import KBWSUtils, KBGenomeUtils, MSBiochemUtils + +# Import all (optional dependencies may fail gracefully) +from kbutillib import * + +# Check what's available +import kbutillib +print(kbutillib.__all__) +``` + +## Composable Combinations + +```python +# Genomics workflow +class GenomicsTools(KBWSUtils, KBGenomeUtils, KBAnnotationUtils): + pass + +# Metabolic modeling workflow +class ModelingTools(KBModelUtils, MSFBAUtils, MSBiochemUtils): + pass + +# AI curation workflow +class CurationTools(AICurationUtils, MSBiochemUtils, NotebookUtils): + pass + +# Full analysis stack +class FullStack(KBGenomeUtils, KBAnnotationUtils, KBModelUtils, + MSBiochemUtils, MSFBAUtils, NotebookUtils): + pass +``` diff --git a/.claude/commands/kbutillib-expert/context/patterns.md b/.claude/commands/kbutillib-expert/context/patterns.md new file mode 100644 index 00000000..0178f4ea --- /dev/null +++ b/.claude/commands/kbutillib-expert/context/patterns.md @@ -0,0 +1,489 @@ +# KBUtilLib Common Patterns and Workflows + +Practical patterns and complete workflows for using KBUtilLib. + +## Pattern 1: Composable Class Design + +The core pattern in KBUtilLib is combining utility classes via multiple inheritance. + +### Basic Composition +```python +from kbutillib import KBWSUtils, KBGenomeUtils, MSBiochemUtils + +# Combine utilities you need +class MyAnalysisTools(KBWSUtils, KBGenomeUtils, MSBiochemUtils): + """Custom tool combining genome and biochemistry utilities.""" + + def my_custom_method(self, genome_ref): + """Custom method using inherited functionality.""" + genome = self.get_genome(self.workspace_id, genome_ref) + features = self.get_features_by_type(genome, "CDS") + + # Use MSBiochemUtils methods + for feature in features: + reactions = self.map_function_to_reactions(feature['function']) + return reactions + +# Use the combined class +tools = MyAnalysisTools(workspace_id=12345) +``` + +### Specialized Stacks +```python +# Genomics-focused stack +class GenomicsStack(KBWSUtils, KBGenomeUtils, KBAnnotationUtils, BVBRCUtils): + pass + +# Modeling-focused stack +class ModelingStack(KBWSUtils, KBModelUtils, MSFBAUtils, MSBiochemUtils): + pass + +# Curation-focused stack +class CurationStack(AICurationUtils, MSBiochemUtils, NotebookUtils): + pass + +# Full analysis stack +class FullStack(KBGenomeUtils, KBAnnotationUtils, KBModelUtils, + MSFBAUtils, MSBiochemUtils, NotebookUtils): + pass +``` + +## Pattern 2: Configuration Management + +### Standard Configuration Flow +```python +from kbutillib import SharedEnvUtils + +class MyTools(SharedEnvUtils): + pass + +# Option 1: Auto-detect configuration +tools = MyTools() # Loads from ~/kbutillib_config.yaml or defaults + +# Option 2: Explicit configuration file +tools = MyTools(config_file="/path/to/my_config.yaml") + +# Option 3: Runtime configuration +tools = MyTools() +tools.set_token("my_token", namespace="kbase") +``` + +### Configuration File Template +```yaml +# ~/kbutillib_config.yaml +kbase: + endpoint: https://kbase.us/services + workspace_url: https://kbase.us/services/ws + auth_service_url: https://kbase.us/services/auth + +argo: + endpoint: https://api.cels.anl.gov/argo/api/v1 + default_model: gpt4o + +modelseed: + database_path: ~/ModelSEEDDatabase + +logging: + level: INFO + format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + +# Custom settings +my_analysis: + output_dir: ~/analysis_results + cache_enabled: true +``` + +### Accessing Configuration +```python +# Dot-notation access +endpoint = tools.get_config_value("kbase.endpoint") +output_dir = tools.get_config_value("my_analysis.output_dir") + +# With defaults +cache = tools.get_config_value("my_analysis.cache_enabled", default=False) +``` + +## Pattern 3: Genome Analysis Workflow + +### Complete Genome Analysis Pipeline +```python +from kbutillib import KBWSUtils, KBGenomeUtils, KBAnnotationUtils + +class GenomeAnalyzer(KBWSUtils, KBGenomeUtils, KBAnnotationUtils): + pass + +analyzer = GenomeAnalyzer(workspace_id=12345) + +# Step 1: Retrieve genome +genome = analyzer.get_genome(12345, "MyGenome/1") +print(f"Genome: {genome['scientific_name']}") +print(f"Features: {len(genome['features'])}") + +# Step 2: Extract coding sequences +cds_features = analyzer.get_features_by_type(genome, "CDS") +print(f"CDS count: {len(cds_features)}") + +# Step 3: Translate to proteins +proteins = analyzer.translate_features(cds_features) +print(f"Translated {len(proteins)} proteins") + +# Step 4: Analyze annotations +annotations = analyzer.get_annotations(genome) +ec_annotations = analyzer.filter_annotations_by_ontology(annotations, "EC") +print(f"Features with EC numbers: {len(ec_annotations)}") + +# Step 5: Extract functional roles +for feature in cds_features[:10]: + ec_nums = analyzer.get_ec_numbers(feature) + if ec_nums: + reactions = analyzer.map_function_to_reactions(feature['function']) + print(f"{feature['id']}: {len(reactions)} reactions") +``` + +## Pattern 4: Metabolic Model Analysis + +### FBA Analysis Pipeline +```python +from kbutillib import KBWSUtils, KBModelUtils, MSFBAUtils, MSBiochemUtils + +class ModelAnalyzer(KBWSUtils, KBModelUtils, MSFBAUtils, MSBiochemUtils): + pass + +analyzer = ModelAnalyzer(workspace_id=12345) + +# Step 1: Get model +model = analyzer.get_model(12345, "MyModel/1") +print(f"Model has {len(analyzer.get_model_reactions(model))} reactions") + +# Step 2: Set up simulation +analyzer.set_media(model, "Complete") +analyzer.set_objective(model, "bio1") # Biomass reaction + +# Step 3: Run FBA +solution = analyzer.run_fba(model) +print(f"Growth rate: {solution.objective_value}") + +# Step 4: Analyze flux distribution +for reaction in solution.fluxes: + if abs(solution.fluxes[reaction]) > 0.1: + rxn_info = analyzer.get_reaction(reaction.split("_")[0]) + print(f"{reaction}: {solution.fluxes[reaction]:.2f}") + +# Step 5: Flux Variability Analysis +analyzer.set_fraction_of_optimum(model, 0.9) +fva = analyzer.run_fva(model) +for rxn, (min_flux, max_flux) in fva.items(): + if min_flux != max_flux: + print(f"{rxn}: [{min_flux:.2f}, {max_flux:.2f}]") +``` + +### Model Comparison +```python +# Compare two models +model1 = analyzer.get_model(12345, "Model1/1") +model2 = analyzer.get_model(12345, "Model2/1") + +rxns1 = set(r['id'] for r in analyzer.get_model_reactions(model1)) +rxns2 = set(r['id'] for r in analyzer.get_model_reactions(model2)) + +unique_to_1 = rxns1 - rxns2 +unique_to_2 = rxns2 - rxns1 +shared = rxns1 & rxns2 + +print(f"Shared reactions: {len(shared)}") +print(f"Unique to Model1: {len(unique_to_1)}") +print(f"Unique to Model2: {len(unique_to_2)}") +``` + +## Pattern 5: AI-Powered Curation + +### Reaction Curation Pipeline +```python +from kbutillib import AICurationUtils, MSBiochemUtils + +class CurationPipeline(AICurationUtils, MSBiochemUtils): + pass + +curator = CurationPipeline() + +# Step 1: Get reactions to curate +reactions_to_curate = curator.search_reactions("transport") + +# Step 2: Curate each reaction +results = [] +for rxn in reactions_to_curate[:10]: + # Check cache first + cached = curator.get_cached_result(rxn['id']) + if cached: + results.append(cached) + continue + + # Curate direction + direction = curator.curate_reaction_direction(rxn) + + # Categorize stoichiometry + category = curator.categorize_stoichiometry(rxn) + + result = { + 'reaction_id': rxn['id'], + 'direction': direction, + 'category': category + } + results.append(result) + + # Cache result + curator.cache_result(rxn['id'], result) + +# Step 3: Analyze results +reversible = sum(1 for r in results if r['direction'] == 'reversible') +print(f"Reversible reactions: {reversible}/{len(results)}") +``` + +### Gene-Reaction Validation +```python +# Validate gene-reaction associations +model = curator.get_model(12345, "MyModel/1") + +for reaction in model['modelreactions'][:10]: + for gene in reaction.get('genes', []): + assessment = curator.assess_gene_reaction(gene, reaction) + if assessment['confidence'] < 0.5: + print(f"Low confidence: {gene['id']} -> {reaction['id']}") + print(f" Reason: {assessment['reasoning']}") +``` + +## Pattern 6: External Database Integration + +### BV-BRC Genome Import +```python +from kbutillib import BVBRCUtils, KBWSUtils + +class GenomeImporter(BVBRCUtils, KBWSUtils): + pass + +importer = GenomeImporter(workspace_id=12345) + +# Step 1: Search for genomes +genomes = importer.search_bvbrc_genomes("Escherichia coli K-12") + +# Step 2: Fetch complete genome +bvbrc_genome = importer.get_bvbrc_genome(genomes[0]['genome_id']) + +# Step 3: Get features and sequences +features = importer.get_genome_features(genomes[0]['genome_id']) +sequences = importer.get_genome_sequences(genomes[0]['genome_id']) + +# Step 4: Convert to KBase format +kb_genome = importer.convert_to_kbase(bvbrc_genome) + +# Step 5: Save to KBase workspace +importer.save_object( + workspace_id=12345, + obj_type="KBaseGenomes.Genome", + data=kb_genome, + name="EcoliK12_imported" +) +``` + +### UniProt Annotation Enhancement +```python +from kbutillib import KBUniProtUtils, KBGenomeUtils + +class AnnotationEnhancer(KBUniProtUtils, KBGenomeUtils): + pass + +enhancer = AnnotationEnhancer() + +# Get genome features +genome = enhancer.get_genome(12345, "MyGenome/1") +features = enhancer.get_features_by_type(genome, "CDS") + +# Enhance with UniProt data +for feature in features[:10]: + # Search UniProt by sequence + sequence = enhancer.translate_feature(feature) + uniprot_hits = enhancer.search_uniprot(f"sequence:{sequence[:50]}") + + if uniprot_hits: + entry = enhancer.get_uniprot_entry(uniprot_hits[0]['accession']) + print(f"{feature['id']}: {entry['proteinDescription']}") +``` + +## Pattern 7: Notebook-Friendly Analysis + +### Interactive Analysis Session +```python +from kbutillib import (KBWSUtils, KBGenomeUtils, KBModelUtils, + MSBiochemUtils, NotebookUtils) + +class InteractiveAnalysis(KBWSUtils, KBGenomeUtils, KBModelUtils, + MSBiochemUtils, NotebookUtils): + pass + +tools = InteractiveAnalysis(workspace_id=12345) + +# Create tracked data object +genome = tools.get_genome(12345, "MyGenome/1") +genome_data = tools.DataObject( + name="my_genome", + data=genome, + source="kbase_workspace", + params={"workspace": 12345, "ref": "MyGenome/1"} +) + +# Display DataFrame with features +import pandas as pd +features_df = pd.DataFrame(tools.get_features(genome)) +tools.display_dataframe(features_df) + +# Progress bar for long operations +reactions = tools.search_reactions("metabolism") +with tools.create_progress_bar(total=len(reactions)) as pbar: + for rxn in reactions: + # Process reaction + pbar.update(1) +``` + +## Pattern 8: Provenance Tracking + +### Tracking Method Calls +```python +from kbutillib import BaseUtils + +class TrackedAnalysis(BaseUtils): + def analyze_data(self, data): + # Initialize call tracking + self.initialize_call("analyze_data", {"data_size": len(data)}) + + # Perform analysis + result = self._do_analysis(data) + + # Log progress + self.log_info(f"Analyzed {len(data)} items") + + return result + + def save_results(self, results, filename): + self.initialize_call("save_results", {"filename": filename}) + self.save_util_data(filename, results) + self.log_info(f"Saved results to {filename}") + +# Use and check provenance +analyzer = TrackedAnalysis() +result = analyzer.analyze_data(my_data) +analyzer.save_results(result, "analysis_results.json") + +# View provenance +print(analyzer.provenance) +# [{"method": "analyze_data", "params": {...}, "timestamp": ...}, ...] +``` + +## Pattern 9: Error Handling + +### Robust API Calls +```python +from kbutillib import KBWSUtils + +ws = KBWSUtils() + +def safe_get_object(workspace_id, object_ref): + """Safely retrieve object with error handling.""" + try: + return ws.get_object(workspace_id, object_ref) + except ValueError as e: + ws.log_error(f"Object not found: {object_ref}") + return None + except ConnectionError as e: + ws.log_error(f"Connection failed: {e}") + raise + except Exception as e: + ws.log_error(f"Unexpected error: {e}") + raise + +# Use with fallback +genome = safe_get_object(12345, "MyGenome/1") +if genome is None: + genome = safe_get_object(12345, "MyGenome_backup/1") +``` + +### Batch Processing with Recovery +```python +def process_batch(object_refs, workspace_id): + """Process multiple objects with error recovery.""" + results = [] + failed = [] + + for ref in object_refs: + try: + obj = ws.get_object(workspace_id, ref) + result = process_object(obj) + results.append(result) + except Exception as e: + ws.log_error(f"Failed to process {ref}: {e}") + failed.append({"ref": ref, "error": str(e)}) + + ws.log_info(f"Processed {len(results)}/{len(object_refs)} objects") + if failed: + ws.log_warning(f"Failed: {len(failed)} objects") + + return results, failed +``` + +## Pattern 10: Caching Results + +### File-Based Caching +```python +import os +import json +import hashlib + +class CachedAnalysis(BaseUtils): + def __init__(self, cache_dir="~/.kbutillib_cache"): + super().__init__() + self.cache_dir = os.path.expanduser(cache_dir) + os.makedirs(self.cache_dir, exist_ok=True) + + def _cache_key(self, method, params): + """Generate cache key from method and params.""" + key_str = f"{method}:{json.dumps(params, sort_keys=True)}" + return hashlib.md5(key_str.encode()).hexdigest() + + def _get_cached(self, key): + """Retrieve cached result.""" + cache_file = os.path.join(self.cache_dir, f"{key}.json") + if os.path.exists(cache_file): + return self.load_util_data(cache_file) + return None + + def _set_cached(self, key, data): + """Store result in cache.""" + cache_file = os.path.join(self.cache_dir, f"{key}.json") + self.save_util_data(cache_file, data) + + def cached_operation(self, method_func, params): + """Run operation with caching.""" + key = self._cache_key(method_func.__name__, params) + + cached = self._get_cached(key) + if cached: + self.log_debug(f"Cache hit: {key}") + return cached + + self.log_debug(f"Cache miss: {key}") + result = method_func(**params) + self._set_cached(key, result) + return result +``` + +## Example Notebooks Reference + +| Notebook | Purpose | Key Patterns | +|----------|---------|--------------| +| `ConfigureEnvironment.ipynb` | Initial setup | Configuration, tokens | +| `BVBRCGenomeConversion.ipynb` | Import genomes | External API, conversion | +| `AssemblyUploadDownload.ipynb` | Assembly handling | Workspace operations | +| `SKANIGenomeDistance.ipynb` | Genome similarity | External tools, caching | +| `ProteinLanguageModels.ipynb` | PLM analysis | AI/ML integration | +| `StoichiometryAnalysis.ipynb` | Reaction analysis | Biochemistry operations | +| `AICuration.ipynb` | AI curation | LLM integration, caching | +| `KBaseWorkspaceUtilities.ipynb` | Workspace ops | Type discovery, metadata | diff --git a/.claude/commands/modelseeddb-expert.md b/.claude/commands/modelseeddb-expert.md new file mode 100644 index 00000000..8c1067d4 --- /dev/null +++ b/.claude/commands/modelseeddb-expert.md @@ -0,0 +1,234 @@ +# ModelSEED Database Expert + +You are an expert on the ModelSEED Database - the comprehensive biochemistry database used for metabolic model reconstruction. You have deep knowledge of: + +1. **Data Formats** - Compound and reaction TSV/JSON schemas, field definitions, and conventions +2. **BiochemPy Library** - Python API for loading and manipulating compounds and reactions +3. **Data Editing Workflows** - How to add, update, validate, and maintain biochemistry data +4. **Database Structure** - Directory organization, aliases, structures, thermodynamics, and provenance + +## Related Expert Skills + +For questions outside ModelSEED Database's scope, suggest these specialized skills: +- `/modelseedpy-expert` - ModelSEEDpy for metabolic modeling, FBA, gapfilling +- `/msmodelutl-expert` - MSModelUtil class for model manipulation +- `/fbapkg-expert` - FBA packages and constraint systems + +## Knowledge Loading + +Before answering, read relevant documentation based on the question: + +**Primary References (read based on topic):** +- `/Users/chenry/Dropbox/Projects/ModelSEEDDatabase/Biochemistry/COMPOUNDS.md` - Compound field documentation +- `/Users/chenry/Dropbox/Projects/ModelSEEDDatabase/Biochemistry/REACTIONS.md` - Reaction field documentation +- `/Users/chenry/Dropbox/Projects/ModelSEEDDatabase/Biochemistry/README.md` - Overall structure +- `/Users/chenry/Dropbox/Projects/ModelSEEDDatabase/CDM_Schema.md` - Database schema diagram + +**BiochemPy Library (for API questions):** +- `/Users/chenry/Dropbox/Projects/ModelSEEDDatabase/Libs/Python/BiochemPy/Compounds.py` +- `/Users/chenry/Dropbox/Projects/ModelSEEDDatabase/Libs/Python/BiochemPy/Reactions.py` + +**Scripts (for maintenance/editing workflows):** +- `/Users/chenry/Dropbox/Projects/ModelSEEDDatabase/Scripts/README.md` +- `/Users/chenry/Dropbox/Projects/ModelSEEDDatabase/Scripts/Biochemistry/` + +## Quick Reference: Database Structure + +``` +ModelSEEDDatabase/ +├── Biochemistry/ # Main data files +│ ├── compound_00.tsv..compound_35.tsv # ~45,756 compounds +│ ├── reaction_00.tsv..reaction_60.tsv # ~56,070 reactions +│ ├── Aliases/ # External database mappings +│ ├── Structures/ # SMILES, InChI, pKa data +│ ├── Thermodynamics/ # ΔG calculations +│ ├── Provenance/ # Source data (KEGG, MetaCyc, etc.) +│ └── Curation/ # Manual corrections +├── Libs/Python/BiochemPy/ # Python library +├── Scripts/ # Maintenance scripts +├── Annotations/ # Roles and complexes +├── Media/ # Growth media definitions +└── Ontologies/ # Ontology translations +``` + +## Quick Reference: Compound Schema (21 fields) + +| Field | Description | Example | +|-------|-------------|---------| +| id | Unique ID `cpdNNNNN` | cpd00001 | +| name | Compound name | H2O | +| formula | Chemical formula (Hill system) | H2O | +| mass | Molecular weight | 18.0 | +| charge | Electric charge | 0 | +| inchikey | IUPAC InChI Key | XLYOFNOQVPJJNP-UHFFFAOYSA-N | +| deltag | Free energy (kcal/mol) | -37.54 | +| deltagerr | Free energy error | 0.5 | +| is_cofactor | Cofactor flag | 1 or 0 | +| pka | Acid dissociation constants | fragment:atom:value | +| pkb | Base dissociation constants | fragment:atom:value | +| aliases | External IDs | "KEGG:C00001;BiGG:h2o" | +| smiles | SMILES structure | O | + +## Quick Reference: Reaction Schema (22 fields) + +| Field | Description | Example | +|-------|-------------|---------| +| id | Unique ID `rxnNNNNN` | rxn00001 | +| name | Reaction name | diphosphate phosphohydrolase | +| equation | Balanced equation | (1) cpd00001[0] + ... | +| code | Pre-protonation equation | (1) cpd00001 + ... | +| stoichiometry | Detailed stoichiometry | n:cpdid:m:i:"name" | +| status | Validation status | OK, MI, CI, HB, EMPTY | +| is_transport | Transport reaction flag | 0 or 1 | +| reversibility | Direction | >, <, =, ? | +| ec_numbers | EC classifications | 3.6.1.1 | +| deltag | Free energy change | -5.2 | +| aliases | External IDs | "KEGG:R00004;BiGG:ATPM" | +| pathways | Pathway associations | "KEGG:map00010" | +| compound_ids | Compounds involved | cpd00001;cpd00012;cpd00009 | + +**Status Values:** +- `OK` - Valid and balanced +- `MI:element:diff` - Mass imbalance (e.g., MI:C:-1) +- `CI:value` - Charge imbalance (e.g., CI:2) +- `HB` - Hydrogen-balanced only +- `EMPTY` - Reactants cancel out +- `CPDFORMERROR` - Invalid compound formulas + +## Quick Reference: BiochemPy API + +```python +from BiochemPy import Compounds, Reactions + +# Load all compounds +cpds_helper = Compounds() +cpds_dict = cpds_helper.loadCompounds() # Returns dict keyed by ID + +# Access a compound +water = cpds_dict["cpd00001"] +print(water["name"], water["formula"], water["charge"]) + +# Load all reactions +rxns_helper = Reactions() +rxns_dict = rxns_helper.loadReactions() + +# Access a reaction +rxn = rxns_dict["rxn00001"] +print(rxn["name"], rxn["equation"], rxn["status"]) + +# Load aliases +cpd_aliases = cpds_helper.loadMSAliases() +rxn_aliases = rxns_helper.loadMSAliases() + +# Parse reaction equation +stoich = rxns_helper.parseEquation(rxn["equation"]) +# Returns: {compound_id: coefficient, ...} +``` + +## Common Workflows + +### 1. Adding a New Compound +```bash +# 1. Create entry in Biochemistry/Curation/New_Compounds/ +# 2. Run: python Scripts/Biochemistry/Update_DB/Add_New_Compounds.py +# 3. Validate: python Scripts/Biochemistry/Reprint_Biochemistry.py +``` + +### 2. Adding a New Reaction +```bash +# 1. Create entry in Biochemistry/Curation/New_Reactions/ +# 2. Run: python Scripts/Biochemistry/Update_DB/Add_New_Reactions.py +# 3. Rebalance: python Scripts/Biochemistry/Refresh_DB_after_Changes/Rebalance_Reactions.py +# 4. Validate: python Scripts/Biochemistry/Reprint_Biochemistry.py +``` + +### 3. Validating Data +```bash +# The key validation script - no output = success +python Scripts/Biochemistry/Reprint_Biochemistry.py +``` + +### 4. Finding Compounds/Reactions +```python +from BiochemPy import Compounds, Reactions + +cpds = Compounds() +cpds_dict = cpds.loadCompounds() + +# Search by name +for cpd_id, cpd in cpds_dict.items(): + if "glucose" in cpd["name"].lower(): + print(cpd_id, cpd["name"]) + +# Search by alias +aliases = cpds.loadMSAliases() +for cpd_id, alias_dict in aliases.items(): + if "KEGG" in alias_dict and "C00031" in alias_dict["KEGG"]: + print(f"Found: {cpd_id}") +``` + +## Key Design Principles + +1. **TSV is Master Format** - JSON files are derived; edit TSV files only +2. **Data Partitioning** - Files split into numbered segments (compound_00.tsv, etc.) for manageability +3. **Protonation at pH 7** - Formulas standardized using Marvin chemicalize +4. **Comprehensive Aliasing** - Every entity linked to external databases (KEGG, BiGG, MetaCyc, ChEBI) +5. **Validation-First** - Always run Reprint_Biochemistry.py after changes + +## Common Mistakes to Avoid + +1. **Editing JSON files directly** - Always edit TSV; JSON is auto-generated +2. **Forgetting to validate** - Run Reprint_Biochemistry.py after any changes +3. **Invalid formula format** - Must follow Hill system notation +4. **Missing protonation** - Use Marvin for standardized protonation +5. **Duplicate aliases** - Check existing aliases before adding new ones + +## Guidelines for Responding + +When helping users: + +1. **Be specific** - Reference exact file paths, field names, and valid values +2. **Show examples** - Provide working code or data snippets +3. **Explain the workflow** - Which scripts to run, in what order +4. **Warn about validation** - Remind users to run Reprint_Biochemistry.py +5. **Read the docs first** - Consult COMPOUNDS.md and REACTIONS.md for accurate field info + +## Response Format + +### For data format questions: +``` +### Field: `field_name` + +**Type:** string/number/boolean +**Required:** Yes/No +**Example:** value + +**Description:** What this field represents + +**Valid values:** List of acceptable values (if applicable) + +**Notes:** Any special considerations +``` + +### For "how do I" questions: +``` +### Approach + +Brief explanation of the workflow. + +**Step 1:** Description +```bash +command or code +``` + +**Step 2:** Description +```bash +command or code +``` + +**Validation:** How to verify success +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/modelseeddb-expert/context/biochempy-api.md b/.claude/commands/modelseeddb-expert/context/biochempy-api.md new file mode 100644 index 00000000..ec6a5e90 --- /dev/null +++ b/.claude/commands/modelseeddb-expert/context/biochempy-api.md @@ -0,0 +1,448 @@ +# BiochemPy Library API Reference + +The BiochemPy library provides Python interfaces for loading and manipulating ModelSEED biochemistry data. + +**Location:** `/Libs/Python/BiochemPy/` + +**Setup:** +```python +import sys +sys.path.append("/path/to/ModelSEEDDatabase/Libs/Python") +from BiochemPy import Compounds, Reactions +``` + +## Compounds Class + +### Constructor + +```python +Compounds(biochem_root='../../../Biochemistry/', cpds_file='compound_00.tsv') +``` + +**Attributes:** +- `BiochemRoot` - Path to Biochemistry directory +- `CpdsFile` - Path to first compound TSV file +- `AliasFile` - Path to compound aliases file +- `NameFile` - Path to compound names file +- `StructRoot` - Path to Structures directory +- `Headers` - List of TSV column headers + +### Loading Methods + +#### loadCompounds() +Load all compounds from JSON files. + +```python +cpds_helper = Compounds() +cpds_dict = cpds_helper.loadCompounds() + +# Returns: dict keyed by compound ID +# cpds_dict["cpd00001"] = { +# "id": "cpd00001", +# "name": "H2O", +# "formula": "H2O", +# "charge": 0, +# "mass": 18.0, +# ... +# } +``` + +#### loadCompounds_tsv() +Load compounds from TSV file (legacy, loads only one file). + +```python +cpds_dict = cpds_helper.loadCompounds_tsv() +# WARNING: Only loads first file - use loadCompounds() instead +``` + +#### loadMSAliases(sources_array=[]) +Load compound aliases from external databases. + +```python +# Load all aliases +aliases = cpds_helper.loadMSAliases() + +# Load specific sources +aliases = cpds_helper.loadMSAliases(["KEGG", "BiGG"]) + +# Returns: dict[compound_id][source] = [alias_list] +# aliases["cpd00001"]["KEGG"] = ["C00001"] +``` + +#### loadSourceAliases() +Load aliases indexed by source then external ID. + +```python +source_aliases = cpds_helper.loadSourceAliases() + +# Returns: dict[source][external_id] = [modelseed_ids] +# source_aliases["KEGG"]["C00001"] = ["cpd00001"] +``` + +#### loadNames() +Load compound names. + +```python +names = cpds_helper.loadNames() + +# Returns: dict[compound_id] = [name_list] +# names["cpd00001"] = ["H2O", "Water", ...] +``` + +#### loadStructures(sources_array=[], db_array=[], unique=True) +Load molecular structures. + +```python +# Default: SMILES, InChIKey, InChI from KEGG and MetaCyc +structs = cpds_helper.loadStructures() + +# Specific sources +structs = cpds_helper.loadStructures( + sources_array=["SMILE", "InChIKey"], + db_array=["KEGG"] +) + +# ModelSEED consolidated structures +structs = cpds_helper.loadStructures(db_array=["ModelSEED"]) +``` + +### Saving Methods + +#### saveCompounds(compounds_dict) +Save compounds to partitioned TSV and JSON files. + +```python +# After modifying compounds +cpds_helper.saveCompounds(cpds_dict) +# Creates compound_00.tsv, compound_00.json, compound_01.tsv, ... +``` + +#### saveAliases(alias_dict) +Save compound aliases. + +```python +cpds_helper.saveAliases(aliases) +``` + +#### saveNames(names_dict) +Save compound names. + +```python +cpds_helper.saveNames(names) +``` + +### Static Utility Methods + +#### Compounds.searchname(name) +Generate search variations of a compound name. + +```python +variations = Compounds.searchname("D-Glucose") +# Returns: ["D-Glucose", "d-glucose", "dglucose", ...] +``` + +#### Compounds.parseFormula(formula) +Parse chemical formula into atom counts. + +```python +atoms = Compounds.parseFormula("C6H12O6") +# Returns: {"C": 6, "H": 12, "O": 6} + +atoms = Compounds.parseFormula("H2O") +# Returns: {"H": 2, "O": 1} +``` + +#### Compounds.mergeFormula(formula) +Parse and normalize complex formulas. + +```python +formula, notes = Compounds.mergeFormula("Mg(Al,Fe)Si4O10(OH).4H2O") +# Returns: ("Al2H10MgO16Si4", "PO") # PO = polymeric formula note +``` + +#### Compounds.buildFormula(atoms_dict) +Build formula string from atom counts (Hill sorted). + +```python +formula = Compounds.buildFormula({"C": 6, "H": 12, "O": 6}) +# Returns: "C6H12O6" +``` + +#### Compounds.hill_sorted(atoms) +Generator yielding atoms in Hill order (C, H, then alphabetical). + +```python +sorted_atoms = list(Compounds.hill_sorted(["O", "C", "H", "N"])) +# Returns: ["C", "H", "N", "O"] +``` + +--- + +## Reactions Class + +### Constructor + +```python +Reactions(biochem_root='../../../Biochemistry/', rxns_file='reaction_00.tsv') +``` + +**Attributes:** +- `BiochemRoot` - Path to Biochemistry directory +- `RxnsFile` - Path to first reaction TSV file +- `AliasFile` - Path to reaction aliases file +- `NameFile` - Path to reaction names file +- `PwyFile` - Path to pathways file +- `ECFile` - Path to EC numbers file +- `Headers` - List of TSV column headers +- `CompoundsHelper` - Compounds instance for compound lookups +- `Compounds_Dict` - Loaded compounds dictionary + +### Loading Methods + +#### loadReactions() +Load all reactions from JSON files. + +```python +rxns_helper = Reactions() +rxns_dict = rxns_helper.loadReactions() + +# Returns: dict keyed by reaction ID +# rxns_dict["rxn00001"] = { +# "id": "rxn00001", +# "name": "diphosphate phosphohydrolase", +# "equation": "(1) cpd00001[0] + (1) cpd00012[0] <=> ...", +# "status": "OK", +# ... +# } +``` + +#### loadMSAliases(sources_array=[]) +Load reaction aliases. + +```python +aliases = rxns_helper.loadMSAliases() +aliases = rxns_helper.loadMSAliases(["KEGG", "MetaCyc"]) +``` + +#### loadNames() +Load reaction names. + +```python +names = rxns_helper.loadNames() +``` + +#### loadPathways() +Load pathway associations. + +```python +pathways = rxns_helper.loadPathways() +# Returns: dict[rxn_id][source] = [pathway_list] +``` + +#### loadECs() +Load EC number associations. + +```python +ecs = rxns_helper.loadECs() +# Returns: dict[rxn_id] = [ec_list] +# ecs["rxn00001"] = ["3.6.1.1"] +``` + +### Parsing Methods + +#### parseEquation(equation_string) +Parse reaction equation into reagent array. + +```python +equation = "(1) cpd00001[0] + (1) cpd00012[0] <=> (2) cpd00009[0] + (1) cpd00067[0]" +reagents = rxns_helper.parseEquation(equation) + +# Returns: list of dicts +# [ +# {"compound": "cpd00001", "compartment": 0, "coefficient": -1, +# "name": "H2O", "formula": "H2O", "charge": 0}, +# {"compound": "cpd00012", "compartment": 0, "coefficient": -1, ...}, +# {"compound": "cpd00009", "compartment": 0, "coefficient": 2, ...}, +# {"compound": "cpd00067", "compartment": 0, "coefficient": 1, ...} +# ] +``` + +#### parseStoich(stoichiometry) +Parse stoichiometry string into reagent array. + +```python +stoich = "-1:cpd00001:0:0:\"H2O\";-1:cpd00012:0:0:\"PPi\";2:cpd00009:0:0:\"Phosphate\"" +reagents = rxns_helper.parseStoich(stoich) +``` + +### Reaction Manipulation Methods + +#### balanceReaction(rgts_array, all_structures=False) +Check mass and charge balance of a reaction. + +```python +rgts = rxns_helper.parseEquation(equation) +status = rxns_helper.balanceReaction(rgts) + +# Returns status string: +# "OK" - balanced +# "MI:C:-1/H:-4" - mass imbalance +# "CI:2" - charge imbalance +# "EMPTY" - reactants cancel out +# "CPDFORMERROR" - invalid compound formula +# "Duplicate reagents" - same compound appears twice +``` + +#### adjustCompound(rxn_cpds_array, compound, adjustment, compartment=0) +Adjust coefficient of a compound in reaction. + +```python +# Add 2 protons to right side (positive adjustment = subtract from left) +rxns_helper.adjustCompound(rgts, "cpd00067", 2, compartment=0) +``` + +#### replaceCompound(rxn_cpds_array, old_compound, new_compound) +Replace one compound with another. + +```python +success = rxns_helper.replaceCompound(rgts, "cpd00001", "cpd00002") +# Returns: True if found and replaced, False otherwise +``` + +#### rebuildReaction(reaction_dict, stoichiometry=None) +Rebuild equation/code/definition from stoichiometry. + +```python +rxn = rxns_dict["rxn00001"] +# After modifying stoichiometry... +rxns_helper.rebuildReaction(rxn) +# Updates: code, equation, definition, compound_ids fields +``` + +### Code Generation Methods + +#### generateCode(rxn_cpds_array) +Generate unique reaction code for matching. + +```python +rgts = rxns_helper.parseEquation(equation) +code = rxns_helper.generateCode(rgts) +# Returns: string like "cpd00001_0:1|cpd00012_0:1|=|cpd00009_0:2|cpd00067_0:1" +``` + +#### generateCodes(rxns_dict, check_obsolete=True) +Generate codes for all reactions. + +```python +codes = rxns_helper.generateCodes(rxns_dict) +# Returns: dict[code] = {rxn_id: 1, ...} +``` + +### Static Utility Methods + +#### Reactions.isTransport(rxn_cpds_array) +Check if reaction is a transport reaction. + +```python +is_transport = Reactions.isTransport(rgts) +# Returns: 1 if multiple compartments, 0 otherwise +``` + +#### Reactions.buildStoich(rxn_cpds_array) +Build stoichiometry string from reagent array. + +```python +stoich_string = Reactions.buildStoich(rgts) +# Returns: "-1:cpd00001:0:0:\"H2O\";-1:cpd00012:0:0:\"PPi\";..." +``` + +#### Reactions.removeCpdRedundancy(rgts_array) +Remove duplicate compounds by summing coefficients. + +```python +cleaned_rgts = Reactions.removeCpdRedundancy(rgts) +``` + +### Saving Methods + +#### saveReactions(reactions_dict) +Save reactions to partitioned TSV and JSON files. + +```python +rxns_helper.saveReactions(rxns_dict) +``` + +#### saveAliases(alias_dict) +Save reaction aliases. + +```python +rxns_helper.saveAliases(aliases) +``` + +#### saveNames(names_dict) +Save reaction names. + +```python +rxns_helper.saveNames(names) +``` + +#### saveECs(ecs_dict) +Save EC number associations. + +```python +rxns_helper.saveECs(ecs) +``` + +--- + +## Common Patterns + +### Search for a compound by KEGG ID + +```python +cpds = Compounds() +source_aliases = cpds.loadSourceAliases() + +kegg_id = "C00031" # D-Glucose +if "KEGG" in source_aliases and kegg_id in source_aliases["KEGG"]: + ms_ids = source_aliases["KEGG"][kegg_id] + print(f"ModelSEED IDs: {ms_ids}") +``` + +### Validate all reactions + +```python +rxns = Reactions() +rxns_dict = rxns.loadReactions() + +for rxn_id, rxn in rxns_dict.items(): + rgts = rxns.parseStoich(rxn["stoichiometry"]) + status = rxns.balanceReaction(rgts) + if status != "OK": + print(f"{rxn_id}: {status}") +``` + +### Find reactions containing a compound + +```python +rxns = Reactions() +rxns_dict = rxns.loadReactions() + +target_cpd = "cpd00027" # D-Glucose +for rxn_id, rxn in rxns_dict.items(): + if target_cpd in rxn["compound_ids"]: + print(f"{rxn_id}: {rxn['name']}") +``` + +### Modify and save a compound + +```python +cpds = Compounds() +cpds_dict = cpds.loadCompounds() + +# Modify +cpds_dict["cpd00001"]["deltag"] = -37.5 + +# Save all +cpds.saveCompounds(cpds_dict) +``` diff --git a/.claude/commands/modelseeddb-expert/context/data-formats.md b/.claude/commands/modelseeddb-expert/context/data-formats.md new file mode 100644 index 00000000..6cbf5edc --- /dev/null +++ b/.claude/commands/modelseeddb-expert/context/data-formats.md @@ -0,0 +1,226 @@ +# ModelSEED Database Data Formats + +## File Organization + +Data files are located in `/Biochemistry/`: +- **Compounds**: `compound_00.tsv` through `compound_35.tsv` (~45,756 total entries) +- **Reactions**: `reaction_00.tsv` through `reaction_60.tsv` (~56,070 total entries) +- Each has corresponding `.json` format (auto-generated from TSV) + +**Important**: TSV files are the master format. Never edit JSON files directly. + +## Compound Schema (21 fields) + +| # | Field | Type | Description | +|---|-------|------|-------------| +| 1 | id | string | Unique ID format `cpdNNNNN` (e.g., cpd00001) | +| 2 | abbreviation | string | Short name of compound | +| 3 | name | string | Long descriptive name | +| 4 | formula | string | Chemical formula (Hill system, protonated form) | +| 5 | mass | float/null | Molecular weight or "null" | +| 6 | source | string | Source database (currently "Primary Database") | +| 7 | inchikey | string | IUPAC InChI Key identifier | +| 8 | charge | int | Electric charge of compound | +| 9 | is_core | bool | True if in core biochemistry (all true currently) | +| 10 | is_obsolete | bool | True if obsolete/replaced | +| 11 | linked_compound | string | Semicolon-separated related compound IDs or "null" | +| 12 | is_cofactor | bool | True if compound is a cofactor | +| 13 | deltag | float/null | Free energy change (kcal/mol) or "null" | +| 14 | deltagerr | float/null | Free energy error or "null" | +| 15 | pka | string | Acid dissociation constants (see format below) | +| 16 | pkb | string | Base dissociation constants (see format below) | +| 17 | abstract_compound | bool | Abstraction flag (all null currently) | +| 18 | comprised_of | string | Component info or "null" | +| 19 | aliases | string | Semicolon-separated alternative names (see format below) | +| 20 | smiles | string | SMILES structure representation | +| 21 | notes | string | Abbreviated notes (GC, EQ, EQU, etc.) | + +### pKa/pKb Format + +Format: `fragment:atom:value` + +- **fragment**: Molecular fragment index (usually 1) +- **atom**: Atom index within fragment +- **value**: Dissociation constant value + +Multiple values separated by semicolon. + +Example for NAD: +``` +1:17:1.8;1:18:2.56;1:6:12.32;1:25:11.56;1:35:13.12 +``` + +### Alias Format + +Format: `"source:value"` + +- **source**: Name of external database +- **value**: ID or name in that database + +Multiple aliases separated by semicolon. + +Example for Cobamide (cpd00181): +``` +"KEGG:C00210";"name:Cobamide";"searchname:cobamide";"ModelSEED:cpd00181";"KBase:kb|cpd.181" +``` + +Common alias sources: +- KEGG, BiGG, MetaCyc, ChEBI, HMDB +- name, searchname (normalized lowercase) +- ModelSEED, KBase + +### Example Compound Entry (cpd00001 - Water) + +``` +id: cpd00001 +abbreviation: H2O +name: H2O +formula: H2O +mass: 18.0 +charge: 0 +inchikey: XLYOFNOQVPJJNP-UHFFFAOYSA-N +deltag: -37.54 +is_cofactor: 0 +smiles: O +``` + +## Reaction Schema (22 fields) + +| # | Field | Type | Description | +|---|-------|------|-------------| +| 1 | id | string | Unique ID format `rxnNNNNN` (e.g., rxn00001) | +| 2 | abbreviation | string | Short reaction name | +| 3 | name | string | Long reaction name | +| 4 | code | string | Equation using compound IDs (pre-protonation) | +| 5 | stoichiometry | string | Detailed stoichiometry format | +| 6 | is_transport | bool | True if transport reaction | +| 7 | equation | string | Equation using compound IDs (post-protonation) | +| 8 | definition | string | Equation using compound names | +| 9 | reversibility | string | Direction: ">", "<", "=", "?" | +| 10 | direction | string | Direction: ">", "<", "=" | +| 11 | abstract_reaction | bool | Abstraction flag (all null currently) | +| 12 | pathways | string | Semicolon-separated pathway associations | +| 13 | aliases | string | Alternative names (same format as compounds) | +| 14 | ec_numbers | string | Enzyme Commission numbers | +| 15 | deltag | float | Free energy change or 10000000 if unknown | +| 16 | deltagerr | float | Free energy error or 10000000 if unknown | +| 17 | compound_ids | string | Semicolon-separated compound IDs involved | +| 18 | status | string | Validation status (see below) | +| 19 | is_obsolete | bool | True if obsolete/replaced | +| 20 | linked_reaction | string | Related reaction IDs or "null" | +| 21 | notes | string | Abbreviated notes | +| 22 | source | string | Source database | + +### Equation Format (code/equation fields) + +Format: `(n) cpdid[m]` + +- **n**: Coefficient +- **cpdid**: Compound ID +- **m**: Compartment index (0=cytosol, 1=extracellular, etc.) + +Compounds separated by `+`, sides separated by direction symbol (`<=>`, `=>`, `<=`). + +Example (rxn00001): +``` +(1) cpd00001[0] + (1) cpd00012[0] <=> (2) cpd00009[0] + (1) cpd00067[0] +``` + +### Definition Format (compound names) + +Same format but with compound names instead of IDs: +``` +(1) H2O[0] + (1) PPi[0] <=> (2) Phosphate[0] + (1) H+[0] +``` + +### Stoichiometry Format + +Format: `n:cpdid:m:i:"cpdname"` + +- **n**: Coefficient (negative=reactant, positive=product) +- **cpdid**: Compound ID +- **m**: Compartment index +- **i**: Community index (legacy, usually 0) +- **cpdname**: Compound name + +Compounds separated by semicolon. + +Example (rxn00001): +``` +-1:cpd00001:0:0:"H2O";-1:cpd00012:0:0:"PPi";2:cpd00009:0:0:"Phosphate";1:cpd00067:0:0:"H+" +``` + +### Status Field Values + +Multiple values separated by `|` character. + +| Status | Meaning | +|--------|---------| +| OK | Reaction is valid and balanced | +| MI:element:diff | Mass imbalance (e.g., MI:C:-1 = 1 extra C on left) | +| CI:value | Charge imbalance (positive = right side larger) | +| HB | Hydrogen-balanced (H added to balance) | +| EMPTY | Reactants cancel out completely | +| CPDFORMERROR | Compound has no/invalid formula | + +**Mass Imbalance Example** (rxn00277): +``` +(1) Glycine[0] <=> (1) HCN[0] +Status: MI:C:-1/H:-4/O:-2 +``` +(1 extra C, 4 extra H, 2 extra O on left side) + +**Charge Imbalance Example** (rxn00008): +``` +(2) H2O[0] <=> (1) H2O2[0] + (2) H+[0] +Status: CI:2 +``` +(Right side has +2 charge imbalance) + +### Example Reaction Entry (rxn00001) + +``` +id: rxn00001 +name: diphosphate phosphohydrolase +code: (1) cpd00001 + (1) cpd00012 <=> (2) cpd00009 + (1) cpd00067 +equation: (1) cpd00001[0] + (1) cpd00012[0] <=> (2) cpd00009[0] + (1) cpd00067[0] +definition: (1) H2O[0] + (1) PPi[0] <=> (2) Phosphate[0] + (1) H+[0] +stoichiometry: -1:cpd00001:0:0:"H2O";-1:cpd00012:0:0:"PPi";2:cpd00009:0:0:"Phosphate";1:cpd00067:0:0:"H+" +status: OK +is_transport: 0 +reversibility: = +direction: = +ec_numbers: 3.6.1.1 +compound_ids: cpd00001;cpd00009;cpd00012;cpd00067 +``` + +## Compartment Indices + +| Index | Compartment | +|-------|-------------| +| 0 | Cytosol (c0) | +| 1 | Extracellular (e0) | +| 2+ | Other compartments (model-specific) | + +## Supporting Data Directories + +### Aliases/ Directory +- `Unique_ModelSEED_Compound_Aliases.txt` - Compound to external ID mappings +- `Unique_ModelSEED_Reaction_Aliases.txt` - Reaction to external ID mappings +- `Unique_ModelSEED_Reaction_Pathways.txt` - Pathway associations +- `Unique_ModelSEED_Reaction_ECs.txt` - EC number mappings +- `Source_Classifiers.txt` - Three-tier source classification + +### Structures/ Directory +- SMILES, InChI, InChIKey from multiple sources +- pKa/pKb calculations from Marvin +- Charged vs Original structure variants + +### Thermodynamics/ Directory +- Group contribution calculations +- eQuilibrator estimates +- Delta G values and errors + +### Provenance/ Directory +- Original source files from KEGG, MetaCyc, Rhea, ChEBI +- MetaNetX mapping files diff --git a/.claude/commands/modelseeddb-expert/context/workflows.md b/.claude/commands/modelseeddb-expert/context/workflows.md new file mode 100644 index 00000000..da28fa07 --- /dev/null +++ b/.claude/commands/modelseeddb-expert/context/workflows.md @@ -0,0 +1,400 @@ +# ModelSEED Database Maintenance Workflows + +## Environment Setup + +```bash +# 1. Activate conda environment +conda activate msd-env + +# 2. Set PYTHONPATH +export PYTHONPATH=$PYTHONPATH:/path/to/ModelSEEDDatabase/Libs/Python/ + +# 3. Change to Scripts/Biochemistry directory +cd /path/to/ModelSEEDDatabase/Scripts/Biochemistry/ +``` + +## Validation Workflow + +### Reprint_Biochemistry.py + +The primary validation script. If no changes occur after running, the database is valid. + +```bash +./Reprint_Biochemistry.py +git status -s +# If no changes, validation passed +``` + +**What it does:** +1. Loads all compounds from JSON +2. Saves compounds (regenerates TSV and JSON) +3. Loads and saves compound aliases and names +4. Loads all reactions from JSON +5. Rebuilds each reaction (recalculates equation, code, definition, compound_ids) +6. Saves reactions (regenerates TSV and JSON) +7. Loads and saves reaction aliases, names, and EC numbers + +**Use after:** Any data modification to validate consistency. + +--- + +## Adding New Compounds + +### Input File Format + +Create a TSV file with columns: +``` +id names formula charge mass inchi inchikey smiles +``` + +- **id**: External database ID (e.g., KEGG C00031) +- **names**: Pipe-separated names (e.g., "D-Glucose|Glucose|Dextrose") +- **formula**: Chemical formula (Hill system) +- **charge**: Integer charge +- **mass**: Molecular weight +- **inchi/inchikey/smiles**: Molecular structures (optional but recommended) + +### Add_New_Compounds.py + +```bash +./Update_DB/Add_New_Compounds.py compounds.tsv DATABASE [-s] [-r] + +# Arguments: +# compounds.tsv - Input file with new compounds +# DATABASE - Source database name (e.g., "KEGG", "MetaCyc", "User") +# -s - Save changes (without this, dry run only) +# -r - Generate report file +``` + +**Matching logic:** +1. Check if external ID already exists as alias → match +2. Check if structure (InChI > InChIKey > SMILES) matches → match +3. Check if name matches (only if no structure match for compounds with structures) → match +4. No match → create new compound with next available cpdNNNNN ID + +**Example:** +```bash +# Dry run to see what would be matched/added +./Update_DB/Add_New_Compounds.py new_compounds.tsv KEGG -r + +# Actually save changes +./Update_DB/Add_New_Compounds.py new_compounds.tsv KEGG -s +``` + +### Post-Addition Scripts + +After adding compounds: +```bash +# Merge any duplicate formulas +./Update_DB/Merge_Formulas.py + +# Update aliases in database +./Refresh_DB_after_Changes/Update_Compound_Aliases_in_DB.py + +# If structures provided, update structure files +../Structures/List_ModelSEED_Structures.py +../Structures/Update_Compound_Structures_Formulas_Charge.py + +# Rebalance any reactions affected +./Refresh_DB_after_Changes/Rebalance_Reactions.py + +# Final validation +./Reprint_Biochemistry.py +``` + +--- + +## Adding New Reactions + +### Input File Format + +Create a TSV file with columns: +``` +id equation names ecs +``` + +- **id**: External database ID +- **equation**: Reaction equation using compound IDs + - Format: `(coeff) cpdid[compartment] + ... <=> (coeff) cpdid[compartment] + ...` + - Example: `(1) C00001[0] + (1) C00002[0] <=> (1) C00003[0]` +- **names**: Pipe-separated reaction names +- **ecs**: Pipe-separated EC numbers + +### Add_New_Reactions.py + +```bash +./Update_DB/Add_New_Reactions.py reactions.tsv CPD_DATABASE RXN_DATABASE [-s] [-r] + +# Arguments: +# reactions.tsv - Input file with new reactions +# CPD_DATABASE - Source for compound IDs (e.g., "KEGG", "ModelSEED") +# RXN_DATABASE - Source database name for reactions +# -s - Save changes +# -r - Generate report file +``` + +**Matching logic:** +1. Translate compound IDs to ModelSEED IDs +2. Generate reaction code (unique identifier based on stoichiometry) +3. Check if code matches existing reaction → match +4. Check if code matches after water adjustment → match +5. No match → create new reaction with next available rxnNNNNN ID + +### Post-Addition Scripts + +After adding reactions: +```bash +# Rebalance to check mass/charge balance +./Refresh_DB_after_Changes/Rebalance_Reactions.py + +# Adjust protons for balance +./Refresh_DB_after_Changes/Adjust_Reaction_Protons.py + +# Adjust water for balance +./Refresh_DB_after_Changes/Adjust_Reaction_Water.py + +# Merge any duplicate reactions (may occur after adjustments) +./Refresh_DB_after_Changes/Merge_Reactions.py + +# Update aliases +./Refresh_DB_after_Changes/Update_Reaction_Aliases_in_DB.py + +# Final validation +./Reprint_Biochemistry.py +``` + +--- + +## Maintenance Scripts + +### Scripts/Biochemistry/Maintain/ + +| Script | Purpose | +|--------|---------| +| `Check_Charges.py` | Verify compound charges | +| `Check_Formulas.py` | Verify formula validity | +| `Check_Links.py` | Verify linked_compound/linked_reaction references | +| `Check_Transport.py` | Verify is_transport flags | +| `Check_Template_Reactions.py` | Check reactions against templates | +| `Fix_Compound_Obsolescence.py` | Handle obsolete compound transitions | +| `Fix_Values.py` | Fix common data issues | +| `Manual_Update_Links.py` | Manually update links between entities | +| `Remove_Duplicate_Aliases.py` | Clean up duplicate alias entries | +| `Update_Obsolete_Compounds_in_Reactions.py` | Update reactions using obsolete compounds | + +### Scripts/Biochemistry/Refresh_DB_after_Changes/ + +| Script | Purpose | +|--------|---------| +| `Rebalance_Reactions.py` | Recalculate mass/charge balance status | +| `Rebuild_Reactions.py` | Regenerate equation/code/definition fields | +| `Rebuild_Stoichiometry.py` | Regenerate stoichiometry field | +| `Adjust_Reaction_Protons.py` | Add/remove H+ to balance charges | +| `Adjust_Reaction_Water.py` | Add/remove H2O to balance mass | +| `Merge_Reactions.py` | Merge duplicate reactions | +| `Merge_Obsolete_Aliases.py` | Consolidate aliases for merged entities | +| `Remove_Newly_Obsolescent_Compounds.py` | Remove newly obsolete compounds | +| `Remove_Newly_Obsolescent_Reactions.py` | Remove newly obsolete reactions | +| `Update_Compound_Aliases_in_DB.py` | Sync aliases into compound records | +| `Update_Reaction_Aliases_in_DB.py` | Sync aliases into reaction records | +| `Update_Source_Column.py` | Update source field values | + +--- + +## Common Editing Patterns + +### Modify a Single Compound + +```python +import sys +sys.path.append('/path/to/ModelSEEDDatabase/Libs/Python') +from BiochemPy import Compounds + +cpds = Compounds() +cpds_dict = cpds.loadCompounds() + +# Modify +cpds_dict["cpd00001"]["deltag"] = -37.5 +cpds_dict["cpd00001"]["deltagerr"] = 0.5 + +# Save +cpds.saveCompounds(cpds_dict) +``` + +Then validate: +```bash +./Reprint_Biochemistry.py +git diff +``` + +### Modify a Single Reaction + +```python +from BiochemPy import Reactions + +rxns = Reactions() +rxns_dict = rxns.loadReactions() + +# Modify +rxns_dict["rxn00001"]["reversibility"] = ">" +rxns_dict["rxn00001"]["direction"] = ">" + +# Rebuild equation strings +rxns.rebuildReaction(rxns_dict["rxn00001"]) + +# Save +rxns.saveReactions(rxns_dict) +``` + +### Fix a Mass Imbalance + +```python +from BiochemPy import Reactions + +rxns = Reactions() +rxns_dict = rxns.loadReactions() + +rxn = rxns_dict["rxn12345"] +rgts = rxn["stoichiometry"] + +# Check current balance +status = rxns.balanceReaction(rgts) +print(f"Current status: {status}") # e.g., "MI:H:-2/O:-1" + +# Add water to balance (positive adjustment adds to right side) +rxns.adjustCompound(rgts, "cpd00001", -1) # Add 1 H2O to left side + +# Recheck +new_status = rxns.balanceReaction(rgts) +print(f"New status: {new_status}") + +# Rebuild and save +rxns.rebuildReaction(rxn, rgts) +rxns.saveReactions(rxns_dict) +``` + +### Add an Alias to a Compound + +```python +from BiochemPy import Compounds + +cpds = Compounds() +aliases = cpds.loadMSAliases() + +# Add new alias +cpd_id = "cpd00027" +source = "ChEBI" +new_alias = "CHEBI:4167" + +if cpd_id not in aliases: + aliases[cpd_id] = {} +if source not in aliases[cpd_id]: + aliases[cpd_id][source] = [] +aliases[cpd_id][source].append(new_alias) + +# Save +cpds.saveAliases(aliases) + +# Then sync to database +# Run: ./Refresh_DB_after_Changes/Update_Compound_Aliases_in_DB.py +``` + +### Search for Compounds by Name + +```python +from BiochemPy import Compounds + +cpds = Compounds() +cpds_dict = cpds.loadCompounds() +names = cpds.loadNames() + +search_term = "glucose" +matches = [] + +for cpd_id, name_list in names.items(): + for name in name_list: + if search_term.lower() in name.lower(): + matches.append((cpd_id, name, cpds_dict.get(cpd_id, {}).get("formula", ""))) + +for cpd_id, name, formula in matches: + print(f"{cpd_id}: {name} ({formula})") +``` + +### Find Imbalanced Reactions + +```python +from BiochemPy import Reactions + +rxns = Reactions() +rxns_dict = rxns.loadReactions() + +imbalanced = [] +for rxn_id, rxn in rxns_dict.items(): + if rxn["status"] != "OK" and "CPDFORMERROR" not in rxn["status"]: + imbalanced.append((rxn_id, rxn["name"], rxn["status"])) + +print(f"Found {len(imbalanced)} imbalanced reactions:") +for rxn_id, name, status in imbalanced[:20]: # Show first 20 + print(f" {rxn_id}: {name} - {status}") +``` + +--- + +## Git Workflow + +### Before Making Changes +```bash +git status +git checkout -b feature/my-changes +``` + +### After Making Changes +```bash +# Validate +./Reprint_Biochemistry.py +git status -s + +# Review changes +git diff Biochemistry/ + +# Commit +git add Biochemistry/ +git commit -m "Description of changes" +``` + +### Reverting Changes +```bash +# Revert all biochemistry changes +./Reset_Biochemistry_in_Git.sh + +# Or manually +git checkout -- Biochemistry/ +``` + +--- + +## Troubleshooting + +### "CPDFORMERROR" Status +- Compound has no formula or invalid formula +- Check compound record for formula field +- May need to add formula from external source + +### "MI" (Mass Imbalance) +- Atoms don't balance between left and right sides +- Run `Rebalance_Reactions.py` to identify issues +- May need `Adjust_Reaction_Water.py` or manual compound adjustment + +### "CI" (Charge Imbalance) +- Charges don't balance +- Often fixed by `Adjust_Reaction_Protons.py` +- May indicate incorrect compound charges + +### Duplicate Compounds +- Run `Remove_Duplicate_Aliases.py` +- Consider merging via obsolescence workflow + +### Missing Compound in Reaction +- Ensure compound exists in database +- Check compound alias mapping +- May need to add compound first diff --git a/.claude/commands/modelseedpy-expert.md b/.claude/commands/modelseedpy-expert.md new file mode 100644 index 00000000..4bc7c6ce --- /dev/null +++ b/.claude/commands/modelseedpy-expert.md @@ -0,0 +1,221 @@ +# ModelSEEDpy Expert + +You are an expert on ModelSEEDpy - a Python package for metabolic model reconstruction, analysis, and gapfilling. You have comprehensive knowledge of: + +1. **Overall Architecture** - How the modules connect and interact +2. **Core Workflows** - Model building, gapfilling, FBA, community modeling +3. **Module Selection** - Which classes/functions to use for specific tasks +4. **Integration Patterns** - How ModelSEEDpy integrates with COBRApy and KBase + +## Related Expert Skills + +For deep dives into specific areas, use these specialized skills: +- `/msmodelutl-expert` - Deep expertise on MSModelUtil (central model wrapper) +- `/fbapkg-expert` - Deep expertise on FBA packages and constraint systems + +## Knowledge Loading + +Before answering, read relevant documentation based on the question: + +**Architecture Overview:** +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/__init__.py` + +**For specific modules, read the source:** +- Core: `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/core/` +- FBA Packages: `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/fbapkg/` +- Community: `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/community/` +- Biochemistry: `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/biochem/` + +## Quick Reference: Module Map + +``` +ModelSEEDpy +│ +├── core/ # Core model utilities +│ ├── msmodelutl.py # MSModelUtil - Central model wrapper ⭐ +│ ├── msgapfill.py # MSGapfill - Gapfilling algorithms +│ ├── msfba.py # MSFBA - FBA execution +│ ├── msatpcorrection.py # MSATPCorrection - ATP analysis +│ ├── msmedia.py # MSMedia - Growth media definitions +│ ├── mstemplate.py # MSTemplate - Model templates +│ ├── msbuilder.py # MSBuilder - Model construction +│ ├── msgrowthphenotypes.py # Growth phenotype testing +│ ├── msminimalmedia.py # Minimal media computation +│ ├── fbahelper.py # FBAHelper - Low-level FBA utilities +│ └── msgenome.py # MSGenome - Genome handling +│ +├── fbapkg/ # FBA constraint packages +│ ├── mspackagemanager.py # MSPackageManager - Package registry ⭐ +│ ├── basefbapkg.py # BaseFBAPkg - Base class for packages +│ ├── gapfillingpkg.py # GapfillingPkg - Gapfilling constraints +│ ├── kbasemediapkg.py # KBaseMediaPkg - Media constraints +│ ├── flexiblebiomasspkg.py # FlexibleBiomassPkg - Biomass flexibility +│ ├── simplethermopkg.py # SimpleThermoPkg - Thermodynamic constraints +│ └── [15+ more packages] +│ +├── community/ # Community/multi-species modeling +│ ├── mscommunity.py # MSCommunity - Community models +│ ├── mssteadycom.py # MSSteadyCom - SteadyCom algorithm +│ └── mscommfitting.py # Community fitting +│ +├── biochem/ # ModelSEED biochemistry database +│ ├── modelseed_biochem.py # ModelSEEDBiochem - Reaction/compound DB +│ └── modelseed_reaction.py # Reaction utilities +│ +└── multiomics/ # Multi-omics integration + └── [omics integration tools] +``` + +## Common Workflows + +### Workflow 1: Load and Analyze a Model +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.core.msmedia import MSMedia + +# Load model +mdlutl = MSModelUtil.from_cobrapy("model.json") + +# Set media and run FBA +media = MSMedia.from_dict({"EX_cpd00027_e0": 10}) # Glucose +mdlutl.add_missing_exchanges(media) +mdlutl.set_media(media) +solution = mdlutl.model.optimize() +``` + +### Workflow 2: Gapfill a Model +```python +from modelseedpy.core.msgapfill import MSGapfill + +# Create gapfiller +gapfill = MSGapfill(mdlutl, default_target="bio1") + +# Run gapfilling +solution = gapfill.run_gapfilling(media, target="bio1") + +# Integrate solution +mdlutl.add_gapfilling(solution) +``` + +### Workflow 3: Build Model from Genome +```python +from modelseedpy.core.msbuilder import MSBuilder + +# Build draft model from genome +builder = MSBuilder(genome, template) +model = builder.build() +``` + +### Workflow 4: Community Modeling +```python +from modelseedpy.community.mscommunity import MSCommunity + +# Create community from member models +community = MSCommunity(member_models=[model1, model2]) +community.run_fba() +``` + +## Task → Module Routing + +| Task | Primary Module | Secondary | +|------|---------------|-----------| +| Load/wrap a model | `MSModelUtil` | - | +| Find metabolites/reactions | `MSModelUtil` | - | +| Set growth media | `MSModelUtil` + `KBaseMediaPkg` | `MSMedia` | +| Run FBA | `mdlutl.model.optimize()` | `MSFBA` | +| Gapfill a model | `MSGapfill` | `GapfillingPkg` | +| Test growth conditions | `MSModelUtil` | - | +| ATP correction | `MSATPCorrection` | - | +| Add custom constraints | `fbapkg` classes | `BaseFBAPkg` | +| Community modeling | `MSCommunity` | `MSSteadyCom` | +| Build model from genome | `MSBuilder` | `MSTemplate` | +| Access biochemistry DB | `ModelSEEDBiochem` | - | + +## Key Design Patterns + +### Singleton Caching +Both `MSModelUtil` and `MSPackageManager` use singleton patterns: +```python +# These return the same instance +mdlutl1 = MSModelUtil.get(model) +mdlutl2 = MSModelUtil.get(model) + +pkgmgr1 = MSPackageManager.get_pkg_mgr(model) +pkgmgr2 = MSPackageManager.get_pkg_mgr(model) +``` + +### Model Wrapping +All high-level classes accept either `model` or `MSModelUtil`: +```python +# Both work: +gapfill = MSGapfill(model) +gapfill = MSGapfill(mdlutl) +``` + +### Package System +FBA constraints are modular through packages: +```python +# Get or create a package +pkg = mdlutl.pkgmgr.getpkg("GapfillingPkg") + +# Packages add variables/constraints to the model +pkg.build_package(parameters) +``` + +## Guidelines for Responding + +1. **Route to specialized skills** when questions go deep: + - MSModelUtil details → suggest `/msmodelutl-expert` + - FBA package details → suggest `/fbapkg-expert` + +2. **Start with the right module** - Help users find where to begin + +3. **Show integration** - How modules work together + +4. **Provide working examples** - Complete, runnable code + +5. **Explain COBRApy relationship** - ModelSEEDpy wraps and extends COBRApy + +## Response Format + +### For "how do I" questions: +``` +### Approach + +Brief explanation of which modules to use and why. + +**Modules involved:** +- `Module1` - Purpose +- `Module2` - Purpose + +**Example:** +```python +# Complete working code +``` + +**For deeper information:** Use `/specialized-skill` +``` + +### For architecture questions: +``` +### Overview + +Explanation of the component/concept. + +### Key Classes + +- `ClassName` (module) - Purpose +- `ClassName` (module) - Purpose + +### How They Connect + +Explanation of relationships. + +### Example + +Working example showing integration. +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/modelseedpy-expert/context/architecture.md b/.claude/commands/modelseedpy-expert/context/architecture.md new file mode 100644 index 00000000..bf105a86 --- /dev/null +++ b/.claude/commands/modelseedpy-expert/context/architecture.md @@ -0,0 +1,285 @@ +# ModelSEEDpy Architecture + +## Overview + +ModelSEEDpy is a Python package for metabolic model reconstruction, gapfilling, and analysis. It builds on COBRApy and integrates with the ModelSEED biochemistry database and KBase platform. + +## Module Hierarchy + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ User Code │ +└─────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌───────────────┐ ┌───────────────┐ +│ MSGapfill │ │ MSCommunity │ │ MSBuilder │ +│ (Gapfilling) │ │ (Community) │ │ (Model build)│ +└───────────────┘ └───────────────┘ └───────────────┘ + │ │ │ + └─────────────────────┼─────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ MSModelUtil │ +│ (Central Model Wrapper - core/msmodelutl.py) │ +│ │ +│ • Wraps cobra.Model │ +│ • Provides metabolite/reaction search │ +│ • Manages media, exchanges, tests │ +│ • Coordinates with other components │ +└─────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌───────────────┐ ┌───────────────┐ +│MSPackageManager│ │ MSATPCorrection│ │ModelSEEDBiochem│ +│ (FBA Packages) │ │ (ATP Analysis) │ │ (Biochem DB) │ +└───────────────┘ └───────────────┘ └───────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ FBA Packages (fbapkg/) │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │GapfillingPkg │ │KBaseMediaPkg │ │FlexBiomassPkg│ ... │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +│ │ +│ All inherit from BaseFBAPkg │ +│ Add variables/constraints to model.solver │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ COBRApy Model │ +│ (cobra.Model object) │ +│ │ +│ • Reactions, Metabolites, Genes │ +│ • model.solver (optlang) │ +│ • model.optimize() │ +└─────────────────────────────────────────────────────────────────┘ +``` + +## Core Modules (modelseedpy/core/) + +### MSModelUtil (msmodelutl.py) ~2000 lines +**The central hub for model operations.** + +Key responsibilities: +- Wrap and extend cobra.Model +- Metabolite/reaction search and lookup +- Exchange and transport management +- Media configuration +- FBA testing and condition management +- Gapfilling support methods +- Integration with all other components + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +mdlutl = MSModelUtil.get(model) # Singleton access +``` + +### MSGapfill (msgapfill.py) ~1200 lines +**Automated model gapfilling.** + +Key features: +- Multi-media gapfilling +- ATP-aware gapfilling +- Binary/linear reaction filtering +- Solution testing and validation + +```python +from modelseedpy.core.msgapfill import MSGapfill +gapfill = MSGapfill(mdlutl, default_target="bio1") +solution = gapfill.run_gapfilling(media, target="bio1") +``` + +### MSATPCorrection (msatpcorrection.py) +**ATP production analysis and correction.** + +Prevents models from producing ATP without valid biochemistry. + +```python +atputl = mdlutl.get_atputl(core_template=template) +atp_tests = mdlutl.get_atp_tests() +``` + +### MSFBA (msfba.py) +**Higher-level FBA execution with reporting.** + +```python +from modelseedpy.core.msfba import MSFBA +fba = MSFBA(mdlutl) +result = fba.run_fba() +``` + +### MSMedia (msmedia.py) +**Growth media definitions.** + +```python +from modelseedpy.core.msmedia import MSMedia +media = MSMedia.from_dict({"EX_cpd00027_e0": 10}) +media = MSMedia.from_file("media.tsv") +``` + +### MSBuilder (msbuilder.py) +**Model construction from genome annotations.** + +```python +from modelseedpy.core.msbuilder import MSBuilder +builder = MSBuilder(genome, template) +model = builder.build() +``` + +### MSTemplate (mstemplate.py) +**Model templates for reconstruction.** + +Templates define which reactions can be added during reconstruction and their properties. + +### MSGrowthPhenotypes (msgrowthphenotypes.py) +**Phenotype testing and comparison.** + +Test model predictions against experimental growth data. + +## FBA Packages (modelseedpy/fbapkg/) + +### MSPackageManager +**Central registry for FBA packages.** + +```python +from modelseedpy.fbapkg import MSPackageManager +pkgmgr = MSPackageManager.get_pkg_mgr(model) # Singleton + +# List available packages +pkgmgr.list_available_packages() + +# Get or create a package +pkg = pkgmgr.getpkg("GapfillingPkg") +``` + +### BaseFBAPkg +**Base class for all FBA packages.** + +All packages inherit from this and implement: +- `build_package(params)` - Add constraints/variables +- `clear()` - Remove constraints/variables + +### Key Packages + +| Package | Purpose | +|---------|---------| +| `GapfillingPkg` | Gapfilling MILP formulation | +| `KBaseMediaPkg` | Media exchange constraints | +| `FlexibleBiomassPkg` | Flexible biomass composition | +| `SimpleThermoPkg` | Simple thermodynamic constraints | +| `FullThermoPkg` | Full thermodynamic constraints | +| `ReactionUsePkg` | Binary reaction usage variables | +| `RevBinPkg` | Reversibility binary variables | +| `ObjectivePkg` | Objective function management | +| `TotalFluxPkg` | Total flux minimization | +| `BilevelPkg` | Bilevel optimization | + +## Community Module (modelseedpy/community/) + +### MSCommunity (mscommunity.py) +**Multi-species community modeling.** + +```python +from modelseedpy.community.mscommunity import MSCommunity +community = MSCommunity(member_models=[m1, m2, m3]) +``` + +### MSSteadyCom (mssteadycom.py) +**SteadyCom algorithm for community FBA.** + +Computes steady-state community compositions. + +## Biochemistry Module (modelseedpy/biochem/) + +### ModelSEEDBiochem (modelseed_biochem.py) +**Access to ModelSEED reaction/compound database.** + +```python +from modelseedpy.biochem import ModelSEEDBiochem +biochem = ModelSEEDBiochem.get() +reaction = biochem.get_reaction("rxn00001") +compound = biochem.get_compound("cpd00001") +``` + +## Key Design Patterns + +### 1. Singleton/Cache Pattern +Used by MSModelUtil, MSPackageManager, ModelSEEDBiochem: + +```python +# Same instance returned for same model +mdlutl1 = MSModelUtil.get(model) +mdlutl2 = MSModelUtil.get(model) +assert mdlutl1 is mdlutl2 +``` + +### 2. Model/Utility Acceptance +All high-level classes accept either raw model or utility: + +```python +def __init__(self, model_or_mdlutl): + self.mdlutl = MSModelUtil.get(model_or_mdlutl) + self.model = self.mdlutl.model +``` + +### 3. Package Registration +FBA packages self-register with MSPackageManager: + +```python +class MyPkg(BaseFBAPkg): + def __init__(self, model): + super().__init__(model, "MyPkg", ...) + # BaseFBAPkg.__init__ calls pkgmgr.addpkgobj(self) +``` + +### 4. Lazy Loading +Heavy components loaded on demand: + +```python +# MSATPCorrection created only when needed +atputl = mdlutl.get_atputl() # Creates if missing +``` + +## Data Flow Example: Gapfilling + +``` +User Request: "Gapfill model on glucose media" + │ + ▼ + ┌───────────────┐ + │ MSGapfill │ + │ │ + │ 1. Get media │ + │ 2. Setup FBA │ + │ 3. Run MILP │ + │ 4. Filter │ + └───────────────┘ + │ + ┌───────────┼───────────┐ + │ │ │ + ▼ ▼ ▼ +┌─────────────┐ ┌─────────┐ ┌─────────────┐ +│MSModelUtil │ │GapfillPkg│ │KBaseMediaPkg│ +│ │ │ │ │ │ +│set_media() │ │build_pkg │ │build_pkg │ +│test_soln() │ │(MILP) │ │(bounds) │ +└─────────────┘ └─────────┘ └─────────────┘ + │ │ │ + └───────────┼───────────┘ + │ + ▼ + ┌───────────────┐ + │ cobra.Model │ + │ │ + │ .solver │ + │ .optimize() │ + └───────────────┘ +``` diff --git a/.claude/commands/modelseedpy-expert/context/workflows.md b/.claude/commands/modelseedpy-expert/context/workflows.md new file mode 100644 index 00000000..dd9b4fef --- /dev/null +++ b/.claude/commands/modelseedpy-expert/context/workflows.md @@ -0,0 +1,316 @@ +# ModelSEEDpy Common Workflows + +## Workflow 1: Load and Analyze an Existing Model + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.core.msmedia import MSMedia + +# Load model from file +mdlutl = MSModelUtil.from_cobrapy("model.json") +# Or wrap an existing COBRApy model +# mdlutl = MSModelUtil.get(cobra_model) + +# Inspect model +print(f"Reactions: {len(mdlutl.model.reactions)}") +print(f"Metabolites: {len(mdlutl.model.metabolites)}") + +# Find specific metabolites +glucose_list = mdlutl.find_met("glucose", "c0") +if glucose_list: + glucose = glucose_list[0] + print(f"Found glucose: {glucose.id}") + +# Set up media +media = MSMedia.from_dict({ + "EX_cpd00027_e0": 10, # Glucose + "EX_cpd00001_e0": 1000, # Water + "EX_cpd00009_e0": 1000, # Phosphate + # ... other nutrients +}) + +# Ensure exchanges exist +mdlutl.add_missing_exchanges(media) +mdlutl.set_media(media) + +# Run FBA +solution = mdlutl.model.optimize() +print(f"Growth rate: {solution.objective_value}") +``` + +## Workflow 2: Gapfill a Non-Growing Model + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.core.msgapfill import MSGapfill +from modelseedpy.core.msmedia import MSMedia + +# Load model +mdlutl = MSModelUtil.from_cobrapy("draft_model.json") + +# Define media +media = MSMedia.from_dict({"EX_cpd00027_e0": 10}) +mdlutl.add_missing_exchanges(media) + +# Check if model grows (probably not if draft) +mdlutl.set_media(media) +sol = mdlutl.model.optimize() +print(f"Pre-gapfill growth: {sol.objective_value}") + +# Create gapfiller +gapfill = MSGapfill( + mdlutl, + default_target="bio1", + minimum_obj=0.1 # Minimum required growth +) + +# Run gapfilling +solution = gapfill.run_gapfilling( + media=media, + target="bio1" +) + +print(f"Gapfilling solution: {solution}") + +# Test which reactions are truly needed +unneeded = mdlutl.test_solution( + solution, + targets=["bio1"], + medias=[media], + thresholds=[0.1], + remove_unneeded_reactions=True +) + +# Record the gapfilling +mdlutl.add_gapfilling(solution) + +# Verify growth +sol = mdlutl.model.optimize() +print(f"Post-gapfill growth: {sol.objective_value}") + +# Save model +mdlutl.save_model("gapfilled_model.json") +``` + +## Workflow 3: ATP-Aware Gapfilling + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.core.msgapfill import MSGapfill +from modelseedpy.core.mstemplate import MSTemplateBuilder + +# Load model +mdlutl = MSModelUtil.from_cobrapy("model.json") + +# Get core template for ATP tests +template = MSTemplateBuilder.build_core_template() + +# Get ATP test conditions (prevents ATP loops) +atp_tests = mdlutl.get_atp_tests(core_template=template) + +# Create gapfiller with ATP constraints +gapfill = MSGapfill(mdlutl, default_target="bio1") + +# Run ATP-constrained gapfilling +solution = gapfill.run_gapfilling( + media=media, + target="bio1", + atp_tests=atp_tests # Prevents solutions that produce free ATP +) +``` + +## Workflow 4: Test Growth on Multiple Media + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.core.msmedia import MSMedia + +mdlutl = MSModelUtil.from_cobrapy("model.json") + +# Define test conditions +conditions = [ + { + "media": MSMedia.from_dict({"EX_cpd00027_e0": 10}), # Glucose + "objective": "bio1", + "is_max_threshold": False, # Must grow ABOVE threshold + "threshold": 0.1 + }, + { + "media": MSMedia.from_dict({"EX_cpd00029_e0": 10}), # Acetate + "objective": "bio1", + "is_max_threshold": False, + "threshold": 0.05 + }, + { + "media": MSMedia.from_dict({"EX_cpd00036_e0": 10}), # Succinate + "objective": "bio1", + "is_max_threshold": False, + "threshold": 0.05 + } +] + +# Add missing exchanges for all media +for cond in conditions: + mdlutl.add_missing_exchanges(cond["media"]) + +# Test all conditions +results = {} +for i, cond in enumerate(conditions): + passed = mdlutl.test_single_condition(cond) + media_name = f"condition_{i}" + results[media_name] = passed + print(f"{media_name}: {'PASS' if passed else 'FAIL'}") + +# Or use batch testing +all_passed = mdlutl.test_condition_list(conditions) +print(f"All conditions passed: {all_passed}") +``` + +## Workflow 5: Build Model from Genome + +```python +from modelseedpy.core.msbuilder import MSBuilder +from modelseedpy.core.msgenome import MSGenome +from modelseedpy.core.mstemplate import MSTemplateBuilder + +# Load genome +genome = MSGenome.from_fasta("genome.fasta") +# Or from annotation +# genome = MSGenome.from_rast(annotation_data) + +# Get template +template = MSTemplateBuilder.build_template("GramNegative") + +# Build model +builder = MSBuilder(genome, template) +model = builder.build() + +# Wrap in MSModelUtil for further operations +mdlutl = MSModelUtil.get(model) +print(f"Built model with {len(model.reactions)} reactions") +``` + +## Workflow 6: Community Modeling + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.community.mscommunity import MSCommunity + +# Load individual models +model1 = MSModelUtil.from_cobrapy("species1.json").model +model2 = MSModelUtil.from_cobrapy("species2.json").model +model3 = MSModelUtil.from_cobrapy("species3.json").model + +# Create community +community = MSCommunity( + member_models=[model1, model2, model3], + ids=["sp1", "sp2", "sp3"] +) + +# Run community FBA +result = community.run_fba() + +# Get individual contributions +for member in community.members: + print(f"{member.id}: {member.growth_rate}") +``` + +## Workflow 7: Add Custom FBA Constraints + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.fbapkg import MSPackageManager + +mdlutl = MSModelUtil.from_cobrapy("model.json") +pkgmgr = mdlutl.pkgmgr + +# Get reaction use package (binary variables for reaction on/off) +rxn_use_pkg = pkgmgr.getpkg("ReactionUsePkg") +rxn_use_pkg.build_package({ + "reaction_list": mdlutl.model.reactions +}) + +# Get total flux package (minimize total flux) +total_flux_pkg = pkgmgr.getpkg("TotalFluxPkg") +total_flux_pkg.build_package() + +# Get thermodynamic package +thermo_pkg = pkgmgr.getpkg("SimpleThermoPkg") +thermo_pkg.build_package() + +# Run FBA with all constraints active +solution = mdlutl.model.optimize() +``` + +## Workflow 8: Flexible Biomass Analysis + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.fbapkg import MSPackageManager + +mdlutl = MSModelUtil.from_cobrapy("model.json") + +# Get flexible biomass package +flex_bio_pkg = mdlutl.pkgmgr.getpkg("FlexibleBiomassPkg") + +# Build with flexibility parameters +flex_bio_pkg.build_package({ + "bio_rxn_id": "bio1", + "flex_coefficient": 0.1, # Allow 10% flexibility + "use_rna_class": True, + "use_protein_class": True +}) + +# Now biomass composition can vary within bounds +solution = mdlutl.model.optimize() +``` + +## Workflow 9: Compare Multiple Solutions + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.core.msmedia import MSMedia + +mdlutl = MSModelUtil.from_cobrapy("model.json") + +# Run FBA on different media and collect solutions +solutions = {} + +media_glucose = MSMedia.from_dict({"EX_cpd00027_e0": 10}) +mdlutl.add_missing_exchanges(media_glucose) +mdlutl.set_media(media_glucose) +solutions["glucose"] = mdlutl.model.optimize() + +media_acetate = MSMedia.from_dict({"EX_cpd00029_e0": 10}) +mdlutl.add_missing_exchanges(media_acetate) +mdlutl.set_media(media_acetate) +solutions["acetate"] = mdlutl.model.optimize() + +# Export comparison to CSV +mdlutl.print_solutions(solutions, "flux_comparison.csv") +``` + +## Workflow 10: Debugging - Find Unproducible Biomass Components + +```python +from modelseedpy.core.msmodelutl import MSModelUtil + +mdlutl = MSModelUtil.from_cobrapy("model.json") + +# Set up media +mdlutl.set_media(media) + +# Find biomass components that can't be produced +unproducible = mdlutl.find_unproducible_biomass_compounds( + target_rxn="bio1" +) + +for met in unproducible: + print(f"Cannot produce: {met.id} - {met.name}") + +# Check sensitivity to specific reaction knockouts +ko_results = mdlutl.find_unproducible_biomass_compounds( + target_rxn="bio1", + ko_list=[["rxn00001_c0", ">"], ["rxn00002_c0", "<"]] +) +``` diff --git a/.claude/commands/msmodelutl-expert.md b/.claude/commands/msmodelutl-expert.md new file mode 100644 index 00000000..6059df81 --- /dev/null +++ b/.claude/commands/msmodelutl-expert.md @@ -0,0 +1,175 @@ +# MSModelUtil Expert + +You are an expert on the MSModelUtil class from ModelSEEDpy. You have deep knowledge of: + +1. **The MSModelUtil API** - All 55+ methods, their parameters, return values, and usage +2. **Integration patterns** - How MSModelUtil connects with MSGapfill, MSFBA, MSPackageManager, etc. +3. **Best practices** - Efficient ways to use the API, common pitfalls to avoid +4. **Debugging** - How to diagnose issues in code using MSModelUtil + +## Related Expert Skills + +For questions outside MSModelUtil's scope, suggest these specialized skills: +- `/modelseedpy-expert` - General ModelSEEDpy overview, module routing, workflows +- `/fbapkg-expert` - Deep dive on FBA packages (GapfillingPkg, KBaseMediaPkg, etc.) + +## Knowledge Loading + +Before answering, read the current MSModelUtil documentation: + +**Primary Reference (always read):** +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/agent-io/docs/msmodelutl-developer-guide.md` + +**Source Code (read when needed for implementation details):** +- `/Users/chenry/Dropbox/Projects/ModelSEEDpy/modelseedpy/core/msmodelutl.py` + +## Quick Reference: Essential Patterns + +### Pattern 1: Safe Instance Access +```python +# Always use get() for consistent instance access +mdlutl = MSModelUtil.get(model) # Works with model or mdlutl + +# Functions should accept either +def my_function(model_or_mdlutl): + mdlutl = MSModelUtil.get(model_or_mdlutl) + model = mdlutl.model +``` + +### Pattern 2: Find and Operate on Metabolites +```python +# Always handle empty results +mets = mdlutl.find_met("glucose", "c0") +if mets: + glucose = mets[0] + # Do something with glucose +else: + # Handle not found +``` + +### Pattern 3: Add Exchanges for Media +```python +# Before setting media, ensure exchanges exist +missing = mdlutl.add_missing_exchanges(media) +if missing: + print(f"Added exchanges for: {missing}") +mdlutl.set_media(media) +``` + +### Pattern 4: Test Growth Conditions +```python +condition = { + "media": media, + "objective": "bio1", + "is_max_threshold": True, # True = must be BELOW threshold + "threshold": 0.1 +} +mdlutl.apply_test_condition(condition) +passed = mdlutl.test_single_condition(condition, apply_condition=False) +``` + +### Pattern 5: Gapfill and Validate +```python +# After gapfilling +solution = gapfiller.run_gapfilling(media, target="bio1") + +# Test which reactions are actually needed +unneeded = mdlutl.test_solution( + solution, + targets=["bio1"], + medias=[media], + thresholds=[0.1], + remove_unneeded_reactions=True +) +``` + +## Common Mistakes to Avoid + +1. **Not using get()**: Creating multiple MSModelUtil instances for same model +2. **Ignoring empty find_met results**: Always check if list is empty +3. **Forgetting build_metabolite_hash()**: Called automatically by find_met, but cached +4. **Wrong threshold interpretation**: is_max_threshold=True means FAIL if >= threshold +5. **Not adding exchanges before setting media**: Use add_missing_exchanges() first + +## Integration Map + +``` +MSModelUtil ↔ MSGapfill +- MSGapfill takes MSModelUtil in constructor +- Sets mdlutl.gfutl = self for bidirectional access +- Uses mdlutl.test_solution() for solution validation +- Uses mdlutl.reaction_expansion_test() for minimal solutions + +MSModelUtil ↔ MSPackageManager +- Created automatically: self.pkgmgr = MSPackageManager.get_pkg_mgr(model) +- Used for media: self.pkgmgr.getpkg("KBaseMediaPkg").build_package(media) +- All FBA packages access model through MSPackageManager + +MSModelUtil ↔ MSATPCorrection +- Lazy-loaded via get_atputl() +- Sets self.atputl for caching +- Uses ATP tests for gapfilling constraints + +MSModelUtil ↔ ModelSEEDBiochem +- Used in add_ms_reaction() for reaction data +- Used in assign_reliability_scores_to_reactions() for scoring + +MSModelUtil ↔ MSFBA +- MSFBA wraps model_or_mdlutl input +- Uses MSModelUtil for consistent access +``` + +## Guidelines for Responding + +When helping users: + +1. **Be specific** - Reference exact method names, parameters, and return types +2. **Show examples** - Provide working code snippets +3. **Explain integration** - Show how methods connect to other ModelSEEDpy components +4. **Warn about pitfalls** - Mention common mistakes and how to avoid them +5. **Read the docs first** - Always consult the developer guide for accurate information + +## Response Format + +### For API questions: +``` +### Method: `method_name(params)` + +**Purpose:** Brief description + +**Parameters:** +- `param1` (type): Description +- `param2` (type, optional): Description + +**Returns:** Description of return value + +**Example:** +```python +# Working example +``` + +**Related methods:** List of related methods +``` + +### For "how do I" questions: +``` +### Approach + +Brief explanation of the approach. + +**Step 1:** Description +```python +code +``` + +**Step 2:** Description +```python +code +``` + +**Notes:** Any important considerations +``` + +## User Request + +$ARGUMENTS diff --git a/.claude/commands/msmodelutl-expert/context/api-summary.md b/.claude/commands/msmodelutl-expert/context/api-summary.md new file mode 100644 index 00000000..86669d97 --- /dev/null +++ b/.claude/commands/msmodelutl-expert/context/api-summary.md @@ -0,0 +1,121 @@ +# MSModelUtil API Quick Reference + +## Core Concepts + +- **Singleton pattern**: Use `MSModelUtil.get(model)` to get/create instances +- **Wraps cobra.Model**: Access via `mdlutl.model` +- **Integrates with MSPackageManager**: Access via `mdlutl.pkgmgr` +- **Location**: `modelseedpy/core/msmodelutl.py` (~2,000 lines) + +## Essential Methods + +### Factory/Initialization +| Method | Description | +|--------|-------------| +| `MSModelUtil.get(model)` | Get or create instance (PREFERRED) | +| `MSModelUtil.from_cobrapy(filename)` | Load from file | +| `MSModelUtil(model)` | Direct construction | + +### Metabolite Search +| Method | Description | +|--------|-------------| +| `find_met(name, compartment=None)` | Find metabolites by name/ID | +| `msid_hash()` | Get ModelSEED ID to metabolite mapping | +| `metabolite_msid(met)` [static] | Extract ModelSEED ID from metabolite | +| `build_metabolite_hash()` | Build internal lookup caches | + +### Reaction Operations +| Method | Description | +|--------|-------------| +| `rxn_hash()` | Get stoichiometry to reaction mapping | +| `find_reaction(stoichiometry)` | Find reaction by stoichiometry | +| `exchange_list()` | Get exchange reactions | +| `exchange_hash()` | Metabolite to exchange mapping | +| `is_core(rxn)` | Check if reaction is core metabolism | + +### Exchange/Transport +| Method | Description | +|--------|-------------| +| `add_exchanges_for_metabolites(cpds, uptake, excretion)` | Add exchanges | +| `add_transport_and_exchange_for_metabolite(met, direction)` | Add transport | +| `add_missing_exchanges(media)` | Fill media gaps | + +### Media/FBA +| Method | Description | +|--------|-------------| +| `set_media(media)` | Configure growth media | +| `apply_test_condition(condition)` | Apply test constraints | +| `test_single_condition(condition)` | Run single test | +| `test_condition_list(conditions)` | Run multiple tests | + +### Gapfilling Support +| Method | Description | +|--------|-------------| +| `test_solution(solution, targets, medias, thresholds)` | Validate solutions | +| `add_gapfilling(solution)` | Record integrated gapfilling | +| `reaction_expansion_test(rxn_list, conditions)` | Find minimal sets | + +### ATP Correction +| Method | Description | +|--------|-------------| +| `get_atputl()` | Get ATP correction utility | +| `get_atp_tests()` | Get ATP test conditions | + +### Model Editing +| Method | Description | +|--------|-------------| +| `add_ms_reaction(rxn_dict)` | Add ModelSEED reactions | +| `add_atp_hydrolysis(compartment)` | Add ATP hydrolysis | +| `get_attributes()` / `save_attributes()` | Model metadata | + +### Analysis +| Method | Description | +|--------|-------------| +| `assign_reliability_scores_to_reactions()` | Score reactions | +| `find_unproducible_biomass_compounds()` | Biomass sensitivity | +| `analyze_minimal_reaction_set(solution, label)` | Alternative analysis | + +### I/O +| Method | Description | +|--------|-------------| +| `save_model(filename, format)` | Save model to file | +| `printlp(filename)` | Write LP for debugging | +| `print_solutions(solution_hash, filename)` | Export solutions to CSV | + +## Key Instance Attributes + +```python +self.model # The wrapped cobra.Model +self.pkgmgr # MSPackageManager for this model +self.atputl # MSATPCorrection instance (lazy-loaded) +self.gfutl # MSGapfill reference (set by gapfiller) +self.metabolite_hash # Metabolite lookup cache +self.test_objective # Current test objective value +self.reaction_scores # Gapfilling reaction scores +self.integrated_gapfillings # List of integrated solutions +self.attributes # Model metadata dictionary +``` + +## Condition Dictionary Format + +```python +condition = { + "media": MSMedia, # Media object + "objective": "bio1", # Objective reaction ID + "is_max_threshold": True, # True = FAIL if value >= threshold + "threshold": 0.1 # Threshold value +} +``` + +## Solution Dictionary Format + +```python +solution = { + "new": {"rxn00001_c0": ">"}, # Newly added reactions + "reversed": {"rxn00002_c0": "<"}, # Direction-reversed reactions + "media": media, # Media used + "target": "bio1", # Target reaction + "minobjective": 0.1, # Minimum objective + "binary_check": True # Binary filtering done +} +``` diff --git a/.claude/commands/msmodelutl-expert/context/integration.md b/.claude/commands/msmodelutl-expert/context/integration.md new file mode 100644 index 00000000..6f99490c --- /dev/null +++ b/.claude/commands/msmodelutl-expert/context/integration.md @@ -0,0 +1,239 @@ +# MSModelUtil Integration Map + +## Module Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ MSModelUtil │ +│ (Central Model Wrapper) │ +└───────────────────────────┬─────────────────────────────────────┘ + │ + ┌───────────────────┼───────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌───────────────┐ ┌───────────────────────┐ +│ MSFBA │ │ MSGapfill │ │ MSPackageManager │ +│ (FBA runner) │ │ (Gapfilling) │ │ (Constraint pkgs) │ +└───────────────┘ └───────────────┘ └───────────────────────┘ + │ │ │ + └───────────────────┼───────────────────┘ + │ + ┌───────────────────┼───────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌───────────────┐ ┌───────────────────────┐ +│ MSMedia │ │MSATPCorrection│ │ ModelSEEDBiochem │ +│ (Media def) │ │ (ATP tests) │ │ (Reaction database) │ +└───────────────┘ └───────────────┘ └───────────────────────┘ +``` + +## Key Relationships + +### MSModelUtil ↔ MSGapfill + +**Connection:** +- MSGapfill takes MSModelUtil in constructor +- Sets `mdlutl.gfutl = self` for bidirectional access + +**Methods Used:** +- `mdlutl.test_solution()` - Validates gapfilling solutions +- `mdlutl.reaction_expansion_test()` - Finds minimal reaction sets +- `mdlutl.add_gapfilling()` - Records integrated solutions +- `mdlutl.assign_reliability_scores_to_reactions()` - Scores reactions for gapfilling + +**Example:** +```python +from modelseedpy.core.msgapfill import MSGapfill + +# MSGapfill stores reference to mdlutl +gapfill = MSGapfill(mdlutl, default_target="bio1") +# Now: mdlutl.gfutl == gapfill + +# Use mdlutl methods for solution validation +solution = gapfill.run_gapfilling(media, target="bio1") +unneeded = mdlutl.test_solution(solution, ["bio1"], [media], [0.1]) +``` + +### MSModelUtil ↔ MSPackageManager + +**Connection:** +- Created automatically in `__init__`: `self.pkgmgr = MSPackageManager.get_pkg_mgr(model)` +- Provides FBA constraint packages + +**Methods Used:** +- `mdlutl.pkgmgr.getpkg("KBaseMediaPkg").build_package(media)` - Apply media constraints +- `mdlutl.pkgmgr.getpkg("ObjectivePkg")` - Set objectives +- All FBA packages access model through MSPackageManager + +**Example:** +```python +# MSModelUtil uses pkgmgr internally for set_media() +mdlutl.set_media(media) +# Equivalent to: +# mdlutl.pkgmgr.getpkg("KBaseMediaPkg").build_package(media) +``` + +### MSModelUtil ↔ MSATPCorrection + +**Connection:** +- Lazy-loaded via `get_atputl()` +- Sets `self.atputl` for caching +- Used for ATP production tests during gapfilling + +**Methods Used:** +- `mdlutl.get_atputl()` - Get or create MSATPCorrection +- `mdlutl.get_atp_tests()` - Get ATP test conditions +- ATP tests are used as constraints during gapfilling + +**Example:** +```python +from modelseedpy.core.mstemplate import MSTemplateBuilder + +template = MSTemplateBuilder.build_core_template() +atputl = mdlutl.get_atputl(core_template=template) +tests = mdlutl.get_atp_tests(core_template=template) + +# Tests are condition dicts that can be used with test_single_condition +for test in tests: + passed = mdlutl.test_single_condition(test) +``` + +### MSModelUtil ↔ ModelSEEDBiochem + +**Connection:** +- Used for reaction/compound database lookups +- Not stored as instance attribute (imported when needed) + +**Methods Used:** +- `mdlutl.add_ms_reaction()` - Adds reactions from ModelSEED database +- `mdlutl.assign_reliability_scores_to_reactions()` - Uses biochemistry data for scoring + +**Example:** +```python +# Add ModelSEED reactions by ID +reactions = mdlutl.add_ms_reaction({ + "rxn00001": "c0", # Reaction ID -> compartment + "rxn00002": "c0" +}) +``` + +### MSModelUtil ↔ MSFBA + +**Connection:** +- MSFBA wraps `model_or_mdlutl` input +- Uses `MSModelUtil.get()` for consistent access + +**Example:** +```python +from modelseedpy.core.msfba import MSFBA + +# MSFBA internally calls MSModelUtil.get() +fba = MSFBA(mdlutl) +# or +fba = MSFBA(model) # Will create/get MSModelUtil +``` + +### MSModelUtil ↔ MSMedia + +**Connection:** +- MSMedia objects are passed to `set_media()` +- Used in test conditions + +**Example:** +```python +from modelseedpy.core.msmedia import MSMedia + +# Create media +media = MSMedia.from_dict({"EX_cpd00027_e0": 10}) + +# Apply to model +mdlutl.add_missing_exchanges(media) +mdlutl.set_media(media) +``` + +## Dependency Chain + +``` +User Code + │ + ▼ +MSGapfill / MSFBA / MSCommunity + │ + ▼ +MSModelUtil ◄──────────────────┐ + │ │ + ├── MSPackageManager ───────┤ + │ │ │ + │ ▼ │ + │ FBA Packages │ + │ │ + ├── MSATPCorrection ────────┤ + │ │ + └── ModelSEEDBiochem │ + │ │ + └───────────────────┘ +``` + +## Instance Attributes Set by Other Modules + +| Attribute | Set By | Purpose | +|-----------|--------|---------| +| `mdlutl.gfutl` | MSGapfill | Reference to gapfiller | +| `mdlutl.atputl` | get_atputl() | Cached ATP correction utility | +| `mdlutl.pkgmgr` | __init__ | Package manager for constraints | +| `mdlutl.reaction_scores` | MSGapfill | Gapfilling reaction scores | + +## Cross-Module Workflows + +### Gapfilling Workflow + +```python +# 1. Create MSModelUtil +mdlutl = MSModelUtil.get(model) + +# 2. Create MSGapfill (sets mdlutl.gfutl) +gapfill = MSGapfill(mdlutl) + +# 3. Get ATP tests (creates mdlutl.atputl) +atp_tests = mdlutl.get_atp_tests(core_template=template) + +# 4. Run gapfilling (uses pkgmgr internally) +solution = gapfill.run_gapfilling(media, target="bio1") + +# 5. Validate solution (uses test_solution) +unneeded = mdlutl.test_solution(solution, ["bio1"], [media], [0.1]) + +# 6. Record gapfilling +mdlutl.add_gapfilling(solution) +``` + +### FBA Workflow + +```python +# 1. Create MSModelUtil +mdlutl = MSModelUtil.get(model) + +# 2. Set media (uses pkgmgr) +mdlutl.add_missing_exchanges(media) +mdlutl.set_media(media) + +# 3. Run FBA (through cobra.Model) +solution = mdlutl.model.optimize() + +# 4. Analyze results +print(f"Growth: {solution.objective_value}") +``` + +### Community Modeling Workflow + +```python +from modelseedpy.community.mscommunity import MSCommunity + +# MSCommunity creates MSModelUtil for each member +community = MSCommunity(model=community_model, member_models=[m1, m2]) + +# Each member has its own MSModelUtil +for member in community.members: + mdlutl = member.model_util + # Work with individual member +``` diff --git a/.claude/commands/msmodelutl-expert/context/patterns.md b/.claude/commands/msmodelutl-expert/context/patterns.md new file mode 100644 index 00000000..34baf3ce --- /dev/null +++ b/.claude/commands/msmodelutl-expert/context/patterns.md @@ -0,0 +1,257 @@ +# Common MSModelUtil Patterns + +## Pattern 1: Safe Instance Access + +```python +from modelseedpy.core.msmodelutl import MSModelUtil + +# Always use get() for consistent instance access +mdlutl = MSModelUtil.get(model) # Works with model or mdlutl + +# Multiple calls return same instance +mdlutl1 = MSModelUtil.get(model) +mdlutl2 = MSModelUtil.get(model) +assert mdlutl1 is mdlutl2 # True + +# Functions should accept either model or mdlutl +def my_function(model_or_mdlutl): + mdlutl = MSModelUtil.get(model_or_mdlutl) + model = mdlutl.model + # ... rest of function +``` + +## Pattern 2: Load and Analyze a Model + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +from modelseedpy.core.msmedia import MSMedia + +# Load model +mdlutl = MSModelUtil.from_cobrapy("my_model.json") + +# Set media +media = MSMedia.from_dict({"EX_cpd00027_e0": 10}) # Glucose +mdlutl.set_media(media) + +# Run FBA +solution = mdlutl.model.optimize() +print(f"Growth: {solution.objective_value}") +``` + +## Pattern 3: Find and Operate on Metabolites + +```python +# Always handle empty results +mets = mdlutl.find_met("glucose", "c0") +if mets: + glucose = mets[0] + # Do something with glucose +else: + print("Glucose not found in model") + +# Find by ModelSEED ID +mets = mdlutl.find_met("cpd00001") # Water +mets = mdlutl.find_met("cpd00001", "c0") # Cytosolic water + +# Get all ModelSEED IDs +id_hash = mdlutl.msid_hash() +# id_hash["cpd00001"] = [, ] +``` + +## Pattern 4: Add Exchanges for Media + +```python +from modelseedpy.core.msmedia import MSMedia + +# Before setting media, ensure exchanges exist +missing = mdlutl.add_missing_exchanges(media) +if missing: + print(f"Added exchanges for: {missing}") + +# Now set the media +mdlutl.set_media(media) + +# Alternative: add specific exchanges +mets_e0 = mdlutl.find_met("glucose", "e0") +if mets_e0: + mdlutl.add_exchanges_for_metabolites(mets_e0, uptake=10, excretion=0) +``` + +## Pattern 5: Test Growth Conditions + +```python +# Define condition +condition = { + "media": media, + "objective": "bio1", + "is_max_threshold": True, # True = FAIL if value >= threshold + "threshold": 0.1 +} + +# Apply and test (two-step) +mdlutl.apply_test_condition(condition) +passed = mdlutl.test_single_condition(condition, apply_condition=False) + +# Or test directly (one-step) +passed = mdlutl.test_single_condition(condition, apply_condition=True) + +# Test multiple conditions +all_passed = mdlutl.test_condition_list([cond1, cond2, cond3]) +``` + +## Pattern 6: Gapfill and Validate + +```python +from modelseedpy.core.msgapfill import MSGapfill + +# Create gapfiller +gapfill = MSGapfill(mdlutl, default_target="bio1") + +# Run gapfilling +solution = gapfill.run_gapfilling(media, target="bio1") + +# Test which reactions are actually needed +unneeded = mdlutl.test_solution( + solution, + targets=["bio1"], + medias=[media], + thresholds=[0.1], + remove_unneeded_reactions=True # Actually remove them +) + +# Record the gapfilling +mdlutl.add_gapfilling(solution) +``` + +## Pattern 7: ATP Correction + +```python +from modelseedpy.core.mstemplate import MSTemplateBuilder + +# Get core template +template = MSTemplateBuilder.build_core_template() + +# Get ATP tests +tests = mdlutl.get_atp_tests(core_template=template) + +# Run tests +for test in tests: + passed = mdlutl.test_single_condition(test) + print(f"{test['media'].id}: {'PASS' if passed else 'FAIL'}") +``` + +## Pattern 8: Find and Add Reactions + +```python +# Find a metabolite +glucose_list = mdlutl.find_met("glucose", "c0") +if glucose_list: + glucose = glucose_list[0] + + # Add exchange if missing + if glucose not in mdlutl.exchange_hash(): + mdlutl.add_exchanges_for_metabolites([glucose], uptake=10, excretion=0) + +# Add a transport reaction +mdlutl.add_transport_and_exchange_for_metabolite(glucose, direction=">") + +# Add ModelSEED reactions +reactions = mdlutl.add_ms_reaction({ + "rxn00001": "c0", + "rxn00002": "c0" +}) +``` + +## Pattern 9: Debug FBA Issues + +```python +import logging + +# Enable debug logging +logging.getLogger("modelseedpy.core.msmodelutl").setLevel(logging.DEBUG) + +# Print LP file for solver issues +mdlutl.printlp(print=True, filename="debug_problem") +# Creates debug_problem.lp in current directory + +# Check metabolite hash is built +if mdlutl.metabolite_hash is None: + mdlutl.build_metabolite_hash() + +# Verify MSModelUtil caching +print(f"Cached models: {len(MSModelUtil.mdlutls)}") + +# Find unproducible biomass compounds +unproducible = mdlutl.find_unproducible_biomass_compounds("bio1") +for met in unproducible: + print(f"Cannot produce: {met.id}") +``` + +## Pattern 10: Compare Multiple Solutions + +```python +# Run FBA under different conditions +solutions = {} + +# Glucose media +mdlutl.set_media(glucose_media) +solutions["glucose"] = mdlutl.model.optimize() + +# Acetate media +mdlutl.set_media(acetate_media) +solutions["acetate"] = mdlutl.model.optimize() + +# Export comparison +mdlutl.print_solutions(solutions, "flux_comparison.csv") +``` + +## Common Mistakes + +1. **Not using get()**: Creating multiple MSModelUtil instances for same model + ```python + # WRONG + mdlutl1 = MSModelUtil(model) + mdlutl2 = MSModelUtil(model) # Different instances! + + # RIGHT + mdlutl1 = MSModelUtil.get(model) + mdlutl2 = MSModelUtil.get(model) # Same instance + ``` + +2. **Ignoring empty find_met results**: Always check if list is empty + ```python + # WRONG + glucose = mdlutl.find_met("glucose")[0] # IndexError if not found! + + # RIGHT + mets = mdlutl.find_met("glucose") + if mets: + glucose = mets[0] + ``` + +3. **Wrong threshold interpretation**: is_max_threshold=True means FAIL if value >= threshold + ```python + # is_max_threshold=True means: + # - Test PASSES if objective < threshold + # - Test FAILS if objective >= threshold + # This is for testing "no ATP production" conditions + ``` + +4. **Not adding exchanges before setting media**: + ```python + # WRONG + mdlutl.set_media(media) # May fail if exchanges missing + + # RIGHT + mdlutl.add_missing_exchanges(media) + mdlutl.set_media(media) + ``` + +5. **Modifying model instead of mdlutl.model**: + ```python + # WRONG (if model was reassigned) + model.reactions.get_by_id("bio1").bounds = (0, 1000) + + # RIGHT (always use mdlutl.model) + mdlutl.model.reactions.get_by_id("bio1").bounds = (0, 1000) + ``` diff --git a/.claude/commands/run_headless.md b/.claude/commands/run_headless.md new file mode 100644 index 00000000..b3a272fd --- /dev/null +++ b/.claude/commands/run_headless.md @@ -0,0 +1,158 @@ +# Command: run_headless + +## Purpose + +Execute Claude Code commands in autonomous headless mode with comprehensive JSON output. This command enables Claude to run structured tasks without interactive terminal access, producing complete documentation of all actions taken. + +## Overview + +You are running in headless mode to execute structured commands. You will receive input that may include: +1. **Claude Commands**: One or more commands to be executed (e.g., create-prd, generate-tasks, doc-code-for-dev) +2. **User Prompt**: Description of the work to be done, which may: + - Reference an existing PRD by name (e.g., "user-profile-editing") + - Contain a complete new feature description that should be saved as a PRD +3. **PRD Reference Handling**: When a PRD name is referenced: + - Look for `agent-io/prds//humanprompt.md` + - Look for `agent-io/prds//fullprompt.md` if present + - These files provide the detailed context for the work +4. **PRD Storage**: When a user prompt is provided without a PRD name: + - Analyze the prompt to create a descriptive PRD name (use kebab-case) + - Save the user prompt to `agent-io/prds//humanprompt.md` + - Document the PRD name in your output for future reference + +Your job is to execute the command according to the instructions and produce a comprehensive JSON output file. + +## Critical Principles for Headless Operation + +### User Cannot See Terminal +- The user has NO access to your terminal output +- ALL relevant information MUST go in the JSON output file +- Do not assume the user saw anything you did +- Every action, decision, and result must be documented in `claude-output.json` + +### Autonomous Execution +- Execute tasks independently without asking for permission +- Only ask questions when genuinely ambiguous or missing critical information +- Make reasonable assumptions and document them in comments +- Complete as much work as possible before requesting user input +- Work proactively to accomplish the full scope of the command + +## Command Execution Flow + +Follow this process for all headless executions: + +### 1. Parse Input and Handle PRDs +- Parse the input to identify: + - Which Claude commands to execute + - The user prompt describing the work + - Whether a PRD name is referenced +- **If a PRD name is referenced**: + - Read the PRD files from `agent-io/prds//` + - Use humanprompt.md and fullprompt.md (if available) as context +- **If user prompt provided without PRD name**: + - Create a descriptive PRD name based on the prompt content (use kebab-case) + - Create directory `agent-io/prds//` + - Save the user prompt to `agent-io/prds//humanprompt.md` + - Document the PRD name in your output +- If resuming from a previous session, review the parent session context + +### 2. Execute Command +- Follow the instructions in the command file +- Apply the principles from the system prompt +- Work autonomously as much as possible +- Track all actions as you work + +### 3. Track Everything +- Track all actions in memory as you work +- Build up the JSON output structure continuously +- Document files created, modified, or deleted +- Record task progress and status changes +- Capture all decisions and assumptions + +### 4. Handle User Queries (if needed) +- If you need user input, prepare clear questions +- Format questions according to the JSON schema +- Save complete context for resumption +- Set status to "user_query" +- Ensure session_id is included for continuity + +### 5. Write JSON Output +- Write the complete JSON to `claude-output.json` +- Ensure all required fields are present +- Validate JSON structure before writing +- Include comprehensive session_summary + +## Example Headless Session + +### Example 1: New PRD Creation + +**Input:** +- Commands: `["create-prd"]` +- User prompt: "Add user profile editing feature with avatar upload and bio section" +- PRD name: Not provided + +**Execution Process:** +1. Parse input - no PRD name provided, so create one +2. Generate PRD name: "user-profile-editing" +3. Create directory: `agent-io/prds/user-profile-editing/` +4. Save user prompt to `agent-io/prds/user-profile-editing/humanprompt.md` +5. Ask clarifying questions (if needed) by setting status to "user_query" +6. Generate enhanced PRD content +7. Save to `agent-io/prds/user-profile-editing/fullprompt.md` +8. Create comprehensive JSON output with: + - Status: "complete" + - Session ID: (provided by Claude Code automatically) + - Parent session ID: null (this is a new session) + - Session summary explaining what was accomplished + - Files created: humanprompt.md, fullprompt.md, data.json + - PRD name documented in artifacts + - Any relevant comments, assumptions, or observations + +### Example 2: Using Existing PRD + +**Input:** +- Commands: `["generate-tasks"]` +- User prompt: "Generate implementation tasks for user-profile-editing" +- PRD name: "user-profile-editing" (referenced in prompt) + +**Execution Process:** +1. Parse input - PRD name "user-profile-editing" identified +2. Read `agent-io/prds/user-profile-editing/humanprompt.md` +3. Read `agent-io/prds/user-profile-editing/fullprompt.md` (if exists) +4. Use PRD context to generate detailed task list +5. Save tasks to `agent-io/prds/user-profile-editing/data.json` +6. Create comprehensive JSON output with task list and references + +### The user workflow: +- User reads `claude-output.json` to understand everything you did +- User can review created files based on paths in JSON +- User can resume work by creating new session with parent_session_id + +### If clarification is needed: +- Set status to "user_query" +- Include session_id in output +- Add queries_for_user array with clear, specific questions +- When user provides answers in a new session, that session will have parent_session_id pointing to this session +- Claude Code uses the session chain to maintain full context + +## Output Requirements + +Always output to: `claude-output.json` in the working directory + +The JSON must include: +- All required fields for the command type and status +- Complete file tracking (created, modified, deleted) +- Task progress if applicable +- Session information for continuity +- Comments explaining decisions and assumptions +- Any errors or warnings encountered + +## Best Practices for Headless Execution + +- **Be Specific**: Include file paths, line numbers, function names +- **Be Complete**: Don't leave out details assuming the user knows them +- **Be Clear**: Write for someone who wasn't watching you work +- **Be Actionable**: Comments should help the user understand next steps +- **Be Honest**: If something is incomplete or uncertain, say so +- **Be Thorough**: Document every action taken, no matter how small +- **Be Proactive**: Complete as much work as possible before asking questions diff --git a/.claude/settings.local.json b/.claude/settings.local.json new file mode 100644 index 00000000..274bcb49 --- /dev/null +++ b/.claude/settings.local.json @@ -0,0 +1,12 @@ +{ + "permissions": { + "allow": [ + "Bash(grep:*)", + "Bash(find:*)", + "Bash(tree:*)", + "Bash(mkdir:*)", + "Skill(modelseedpy-expert)", + "Skill(modelseedpy-expert:*)" + ] + } +} diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml index 87de0099..ffde9f98 100644 --- a/.github/workflows/pre-commit.yml +++ b/.github/workflows/pre-commit.yml @@ -3,6 +3,8 @@ name: Run Pre-Commit on: pull_request: {} push: + paths-ignore: + - 'examples/**' branches: - dev - main @@ -13,7 +15,7 @@ jobs: strategy: matrix: os: [ubuntu-latest] - python-version: ['3.8', '3.9', '3.10'] + python-version: ['3.9', '3.10', '3.11'] steps: - uses: actions/checkout@v2 - uses: actions/setup-python@v3 diff --git a/.github/workflows/tox.yml b/.github/workflows/tox.yml new file mode 100644 index 00000000..c3d816d0 --- /dev/null +++ b/.github/workflows/tox.yml @@ -0,0 +1,28 @@ +name: Run Tox + +on: + pull_request: {} + push: + branches: [main] + +jobs: + build: + runs-on: ${{ matrix.os }} + strategy: + matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + python-version: ['3.9', '3.10', '3.11'] + steps: + - uses: actions/checkout@v3 + - name: Set up Python + uses: actions/setup-python@v3 + with: + python-version: ${{ matrix.python-version }} + - name: Install dependencies + run: | + python -m pip install --upgrade pip setuptools wheel build + python -m pip install tox tox-gh-actions + - name: Test with tox + run: | + tox + python -m build . diff --git a/.gitignore b/.gitignore index 6390162b..591c53c2 100644 --- a/.gitignore +++ b/.gitignore @@ -5,10 +5,8 @@ __pycache__/ *.py[cod] *$py.class - # C extensions *.so - # Distribution / packaging .Python build/ @@ -29,17 +27,14 @@ share/python-wheels/ .installed.cfg *.egg MANIFEST - # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec - # Installer logs pip-log.txt pip-delete-this-directory.txt - # Unit test / coverage reports htmlcov/ .tox/ @@ -53,81 +48,70 @@ coverage.xml *.py,cover .hypothesis/ .pytest_cache/ - # Translations *.mo *.pot - # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal - # Flask stuff: instance/ .webassets-cache - # Scrapy stuff: .scrapy - # Sphinx documentation docs/_build/ - # PyBuilder target/ - # Jupyter Notebook .ipynb_checkpoints .idea - # IPython profile_default/ ipython_config.py - # pyenv .python-version - # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock - # PEP 582; used by e.g. github.com/David-OConnor/pyflow __pypackages__/ - # Celery stuff celerybeat-schedule celerybeat.pid - # SageMath parsed files *.sage.py - # Environments .env .venv +activate.sh env/ venv/ ENV/ env.bak/ venv.bak/ - # Spyder project settings .spyderproject .spyproject - # Rope project settings .ropeproject - # mkdocs documentation /site - # mypy .mypy_cache/ .dmypy.json dmypy.json - # Pyre type checker .pyre/ +.pydevproject +.settings/* +*data/* +*.lp + +# Cursor workspace files +*.code-workspace diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 04cde634..325706ab 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -21,7 +21,9 @@ repos: args: - --pytest-test-first - id: check-json + exclude: examples/ - id: pretty-format-json + exclude: examples/ args: - --autofix - --top-keys=_id diff --git a/.travis.yml b/.travis.yml index e72cfaff..75b2eb81 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,8 +1,8 @@ language: python python: - - 3.6 - - 3.7 - - 3.8 + - 3.9 + - 3.10 + - 3.11 before_install: - python --version - pip install -U pip diff --git a/README.rst b/README.rst index 6f380d9a..3d491ec3 100644 --- a/README.rst +++ b/README.rst @@ -25,6 +25,10 @@ ________________________________________________________________________ :target: https://pepy.tech/project/modelseedpy :alt: Downloads +.. image:: https://img.shields.io/badge/code%20style-black-000000.svg + :target: https://github.com/ambv/black + :alt: Black + Metabolic modeling is an pivotal method for computational research in synthetic biology and precision medicine. The metabolic models, such as the constrint-based flux balance analysis (FBA) algorithm, are improved with comprehensive datasets that capture more metabolic chemistry in the model and improve the accuracy of simulation predictions. We therefore developed ModelSEEDpy as a comprehensive suite of packages that bootstrap metabolic modeling with the ModelSEED Database (`Seaver et al., 2021 `_ ). These packages parse and manipulate (e.g. gapfill missing reactions or calculated chemical properties of metabolites), constrain (with kinetic, thermodynamics, and nutrient uptake), and simulate cobrakbase models (both individual models and communities). This is achieved by standardizing COBRA models through the ``cobrakbase`` module into a form that is amenable with the KBase/ModelSEED ecosystem. These functionalities are exemplified in `Python Notebooks `_ . Please submit errors, inquiries, or suggestions as `GitHub issues `_ where they can be addressed by our developers. @@ -33,11 +37,11 @@ Metabolic modeling is an pivotal method for computational research in synthetic Installation ---------------------- -ModelSEEDpy will soon be installable via the ``PyPI`` channel:: +PIP (latest stable version 0.4.0):: pip install modelseedpy -but, until then, the repository must cloned:: +GitHub dev build (latest working version):: git clone https://github.com/ModelSEED/ModelSEEDpy.git @@ -51,8 +55,3 @@ The associated ModelSEED Database, which is required for a few packages, is simp git clone https://github.com/ModelSEED/ModelSEEDDatabase.git and the path to this repository is passed as an argument to the corresponding packages. - -**Windows users** must separately install the ``pyeda`` module: 1) download the appropriate wheel for your Python version from `this website `_ ; and 2) install the wheel through the following commands in a command prompt/powershell console:: - - cd path/to/pyeda/wheel - pip install pyeda_wheel_name.whl diff --git a/agent-io/docs/msmodelutl-developer-guide.md b/agent-io/docs/msmodelutl-developer-guide.md new file mode 100644 index 00000000..af82ff21 --- /dev/null +++ b/agent-io/docs/msmodelutl-developer-guide.md @@ -0,0 +1,712 @@ +# MSModelUtil Developer Guide + +## Overview + +`MSModelUtil` is the central utility wrapper class in ModelSEEDpy that encapsulates a COBRApy `Model` object and provides extensive model manipulation, analysis, and FBA-related functionality. It serves as the primary bridge between COBRApy models and ModelSEED-specific functionality. + +**Location:** `modelseedpy/core/msmodelutl.py` (~2,000 lines) + +## Architecture + +### Design Pattern + +MSModelUtil uses a **singleton-like caching pattern** where instances are cached by model object: + +```python +class MSModelUtil: + mdlutls = {} # Static cache of MSModelUtil instances + + @staticmethod + def get(model, create_if_missing=True): + """Get or create MSModelUtil for a model""" + if isinstance(model, MSModelUtil): + return model + if model in MSModelUtil.mdlutls: + return MSModelUtil.mdlutls[model] + elif create_if_missing: + MSModelUtil.mdlutls[model] = MSModelUtil(model) + return MSModelUtil.mdlutls[model] + return None +``` + +This means you can safely call `MSModelUtil.get(model)` multiple times and always get the same instance. + +### Core Dependencies + +``` +MSModelUtil + ├── cobra.Model (wrapped object) + ├── MSPackageManager (FBA constraint packages) + ├── ModelSEEDBiochem (reaction/compound database) + ├── FBAHelper (FBA utility functions) + └── MSATPCorrection (lazy-loaded for ATP analysis) +``` + +### Key Instance Attributes + +```python +self.model # The wrapped cobra.Model +self.pkgmgr # MSPackageManager for this model +self.wsid # KBase workspace ID (if applicable) +self.atputl # MSATPCorrection instance (lazy-loaded) +self.gfutl # MSGapfill reference (set by gapfiller) +self.metabolite_hash # Metabolite lookup cache +self.search_metabolite_hash # Fuzzy search cache +self.test_objective # Current test objective value +self.reaction_scores # Gapfilling reaction scores +self.integrated_gapfillings # List of integrated gapfilling solutions +self.attributes # Model metadata dictionary +self.atp_tests # Cached ATP test conditions +self.reliability_scores # Reaction reliability scores +``` + +--- + +## API Reference + +### Initialization & Factory Methods + +#### `MSModelUtil(model)` +Create a new MSModelUtil wrapping a cobra.Model. + +```python +from modelseedpy.core.msmodelutl import MSModelUtil +import cobra + +model = cobra.io.load_json_model("my_model.json") +mdlutl = MSModelUtil(model) +``` + +#### `MSModelUtil.get(model, create_if_missing=True)` [static] +Get or create an MSModelUtil instance. Preferred method for obtaining instances. + +```python +# These are equivalent and return the same instance: +mdlutl1 = MSModelUtil.get(model) +mdlutl2 = MSModelUtil.get(model) +assert mdlutl1 is mdlutl2 # True + +# Also accepts MSModelUtil directly (returns it unchanged) +mdlutl3 = MSModelUtil.get(mdlutl1) +assert mdlutl3 is mdlutl1 # True +``` + +#### `MSModelUtil.from_cobrapy(filename)` [static] +Load a model from a file and wrap it in MSModelUtil. + +```python +# Supports .json and .xml/.sbml files +mdlutl = MSModelUtil.from_cobrapy("model.json") +mdlutl = MSModelUtil.from_cobrapy("model.xml") +``` + +#### `MSModelUtil.build_from_kbase_json_file(filename, kbaseapi)` [static] +Load a model from KBase JSON format. + +```python +from modelseedpy.core.kbaseapi import KBaseAPI +kbaseapi = KBaseAPI() +mdlutl = MSModelUtil.build_from_kbase_json_file("kbase_model.json", kbaseapi) +``` + +--- + +### I/O Methods + +#### `save_model(filename, format="json")` +Save the model to a file. + +```python +mdlutl.save_model("output.json", format="json") +mdlutl.save_model("output.xml", format="xml") +``` + +#### `printlp(model=None, path="", filename="debug", print=False)` +Write the LP formulation to a file for debugging. + +```python +mdlutl.printlp(print=True) # Writes debug.lp +mdlutl.printlp(path="/tmp", filename="mymodel", print=True) +``` + +#### `print_solutions(solution_hash, filename="reaction_solutions.csv")` +Export multiple FBA solutions to CSV. + +```python +solutions = { + "glucose": model.optimize(), + "acetate": model.optimize() # after changing media +} +mdlutl.print_solutions(solutions, "flux_comparison.csv") +``` + +--- + +### Metabolite Search & Lookup + +#### `find_met(name, compartment=None)` +Find metabolites by name, ID, or annotation. Returns a list of matching metabolites. + +```python +# Find by ModelSEED ID +mets = mdlutl.find_met("cpd00001") # Water + +# Find by name +mets = mdlutl.find_met("glucose") + +# Find in specific compartment +mets = mdlutl.find_met("cpd00001", "c0") # Cytosolic water +mets = mdlutl.find_met("cpd00001", "e0") # Extracellular water +``` + +#### `metabolite_msid(metabolite)` [static] +Extract the ModelSEED compound ID from a metabolite. + +```python +msid = MSModelUtil.metabolite_msid(met) # Returns "cpd00001" or None +``` + +#### `reaction_msid(reaction)` [static] +Extract the ModelSEED reaction ID from a reaction. + +```python +msid = MSModelUtil.reaction_msid(rxn) # Returns "rxn00001" or None +``` + +#### `msid_hash()` +Create a dictionary mapping ModelSEED IDs to metabolite lists. + +```python +id_hash = mdlutl.msid_hash() +# id_hash["cpd00001"] = [, ] +``` + +#### `build_metabolite_hash()` +Build internal metabolite lookup caches. Called automatically by `find_met()`. + +```python +mdlutl.build_metabolite_hash() +# Now mdlutl.metabolite_hash and mdlutl.search_metabolite_hash are populated +``` + +#### `search_name(name)` [static] +Normalize a name for fuzzy searching (lowercase, strip compartment suffix, remove non-alphanumeric). + +```python +MSModelUtil.search_name("D-Glucose_c0") # Returns "dglucose" +``` + +--- + +### Reaction Search & Analysis + +#### `rxn_hash()` +Create a dictionary mapping reaction stoichiometry strings to reactions. + +```python +rxn_hash = mdlutl.rxn_hash() +# rxn_hash["cpd00001_c0+cpd00002_c0=cpd00003_c0"] = [, 1] +``` + +#### `find_reaction(stoichiometry)` +Find a reaction by its stoichiometry. + +```python +stoich = {met1: -1, met2: -1, met3: 1} +result = mdlutl.find_reaction(stoich) +# Returns [reaction, direction] or None +``` + +#### `stoichiometry_to_string(stoichiometry)` [static] +Convert stoichiometry dict to canonical string representation. + +```python +strings = MSModelUtil.stoichiometry_to_string(rxn.metabolites) +# Returns ["reactants=products", "products=reactants"] +``` + +#### `exchange_list()` +Get all exchange reactions (EX_ or EXF prefixed). + +```python +exchanges = mdlutl.exchange_list() +``` + +#### `exchange_hash()` +Create a dictionary mapping metabolites to their exchange reactions. + +```python +ex_hash = mdlutl.exchange_hash() +# ex_hash[] = +``` + +#### `nonexchange_reaction_count()` +Count non-exchange reactions that have non-zero bounds. + +```python +count = mdlutl.nonexchange_reaction_count() +``` + +#### `is_core(rxn)` +Check if a reaction is a core metabolic reaction. + +```python +if mdlutl.is_core("rxn00001_c0"): + print("This is a core reaction") +``` + +--- + +### Exchange & Transport Management + +#### `add_exchanges_for_metabolites(cpds, uptake=0, excretion=0, prefix="EX_", prefix_name="Exchange for ")` +Add exchange reactions for metabolites. + +```python +# Add uptake-only exchanges +mdlutl.add_exchanges_for_metabolites(mets, uptake=1000, excretion=0) + +# Add bidirectional exchanges +mdlutl.add_exchanges_for_metabolites(mets, uptake=1000, excretion=1000) +``` + +#### `add_transport_and_exchange_for_metabolite(met, direction="=", prefix="trans", override=False)` +Add a charge-balanced transport reaction and corresponding exchange. + +```python +# Add bidirectional transport for a cytosolic metabolite +transport = mdlutl.add_transport_and_exchange_for_metabolite( + met_c0, direction="=", prefix="trans" +) +``` + +#### `add_missing_exchanges(media)` +Add exchange reactions for media compounds that don't have them. + +```python +missing = mdlutl.add_missing_exchanges(my_media) +# Returns list of compound IDs that needed exchanges added +``` + +--- + +### Media & FBA Configuration + +#### `set_media(media)` +Set the model's growth media. + +```python +from modelseedpy.core.msmedia import MSMedia + +# From MSMedia object +mdlutl.set_media(my_media) + +# From dictionary +mdlutl.set_media({"cpd00001": 1000, "cpd00007": 1000}) +``` + +#### `set_objective_from_phenotype(phenotype, missing_transporters=[], create_missing_compounds=False)` +Configure the model objective based on a phenotype type. + +```python +# For growth phenotypes, sets biomass objective +# For uptake/excretion phenotypes, sets appropriate exchange objectives +obj_str = mdlutl.set_objective_from_phenotype(phenotype) +``` + +--- + +### FBA Testing & Condition Management + +#### `apply_test_condition(condition, model=None)` +Apply a test condition (media, objective, direction) to the model. + +```python +condition = { + "media": my_media, + "objective": "bio1", + "is_max_threshold": True, + "threshold": 0.1 +} +mdlutl.apply_test_condition(condition) +``` + +#### `test_single_condition(condition, apply_condition=True, model=None, report_atp_loop_reactions=False, analyze_failures=False, rxn_list=[])` +Test if a model meets a condition's threshold. + +```python +passed = mdlutl.test_single_condition(condition) +# Returns True if threshold is NOT exceeded (for is_max_threshold=True) +``` + +#### `test_condition_list(condition_list, model=None, positive_growth=[], rxn_list=[])` +Test multiple conditions. Returns True only if ALL pass. + +```python +all_passed = mdlutl.test_condition_list(conditions) +``` + +--- + +### Gapfilling Support + +#### `test_solution(solution, targets, medias, thresholds=[0.1], remove_unneeded_reactions=False, do_not_remove_list=[])` +Test if gapfilling solution reactions are needed. + +```python +# Solution format: {"new": {rxn_id: direction}, "reversed": {rxn_id: direction}} +# Or: list of [rxn_id, direction, label] +unneeded = mdlutl.test_solution( + solution, + targets=["bio1"], + medias=[glucose_media], + thresholds=[0.1] +) +``` + +#### `add_gapfilling(solution)` +Record an integrated gapfilling solution. + +```python +mdlutl.add_gapfilling({ + "new": {"rxn00001_c0": ">"}, + "reversed": {"rxn00002_c0": "<"}, + "media": media, + "target": "bio1", + "minobjective": 0.1, + "binary_check": True +}) +``` + +#### `convert_solution_to_list(solution)` +Convert dictionary solution format to list format. + +```python +solution_list = mdlutl.convert_solution_to_list(solution) +# Returns [[rxn_id, direction, "new"|"reversed"], ...] +``` + +--- + +### Reaction Expansion Testing + +These methods are used for binary/linear search to find minimal reaction sets. + +#### `reaction_expansion_test(reaction_list, condition_list, binary_search=True, attribute_label="gf_filter", positive_growth=[], resort_by_score=True, active_reaction_sets=[])` +Test which reactions in a list can be removed while still meeting conditions. + +```python +filtered = mdlutl.reaction_expansion_test( + reaction_list=[[rxn, ">"], [rxn2, "<"], ...], + condition_list=conditions, + binary_search=True +) +# Returns reactions that were filtered out +``` + +#### `binary_expansion_test(reaction_list, condition, currmodel, depth=0, positive_growth=[])` +Binary search variant of expansion testing. + +#### `linear_expansion_test(reaction_list, condition, currmodel, positive_growth=[])` +Linear (one-by-one) variant of expansion testing. + +--- + +### ATP Correction + +#### `get_atputl(atp_media_filename=None, core_template=None, gapfilling_delta=0, max_gapfilling=0, forced_media=[], remake_atputil=False)` +Get or create the MSATPCorrection utility. + +```python +atputl = mdlutl.get_atputl(core_template=template) +``` + +#### `get_atp_tests(core_template=None, atp_media_filename=None, recompute=False, remake_atputil=False)` +Get ATP test conditions. + +```python +tests = mdlutl.get_atp_tests(core_template=template) +# Returns list of condition dicts with media, threshold, objective +``` + +--- + +### Reliability Scoring + +#### `assign_reliability_scores_to_reactions(active_reaction_sets=[])` +Calculate reliability scores for all reactions based on biochemistry data. + +```python +scores = mdlutl.assign_reliability_scores_to_reactions() +# scores[rxn_id][">"] = forward score +# scores[rxn_id]["<"] = reverse score +``` + +Scoring considers: +- Mass/charge balance status +- Delta G values +- Compound completeness +- ATP production direction +- Transported charge + +--- + +### Biomass Analysis + +#### `evaluate_biomass_reaction_mass(biomass_rxn_id, normalize=False)` +Calculate the mass balance of a biomass reaction. + +```python +result = mdlutl.evaluate_biomass_reaction_mass("bio1") +# Returns {"ATP": atp_coefficient, "Total": total_mass} +``` + +#### `find_unproducible_biomass_compounds(target_rxn="bio1", ko_list=None)` +Find biomass compounds that cannot be produced. + +```python +# Without knockouts +unproducible = mdlutl.find_unproducible_biomass_compounds() + +# With knockouts to test sensitivity +ko_results = mdlutl.find_unproducible_biomass_compounds( + ko_list=[["rxn00001_c0", ">"], ["rxn00002_c0", "<"]] +) +``` + +--- + +### Minimal Reaction Set Analysis + +#### `analyze_minimal_reaction_set(solution, label, print_output=True)` +Analyze a minimal reaction set for alternatives and coupled reactions. + +```python +output = mdlutl.analyze_minimal_reaction_set(fba_solution, "my_analysis") +# Writes CSV to nboutput/rxn_analysis/