Enable Parallel Development With Custom Minion Directories

by Admin 59 views
Enable Parallel Development With Custom Minion Directories

Hey there, code wizards! Ever wished you could juggle multiple projects, branches, or even entire repositories simultaneously? Well, you're in luck! This article is all about introducing a game-changing feature: custom working directories for minions. This enhancement will supercharge your workflow, making parallel development a breeze and opening doors to complex, multi-repo projects. Get ready to level up your coding game! Let's dive in, shall we?

The Core Problem: Current Minion Limitations

Currently, our minions are a bit like loyal but limited sidekicks. They all share the same working directory, which is inherited from the main project. This setup creates a bottleneck, especially when you're working with git worktrees or juggling multiple branches.

Current Minion Behavior

  • Shared Working Directory: All minions in a project are stuck in the same directory. This means, no isolation.
  • Inherited Directory: The working directory is inherited from the parent project. No flexibility here.
  • No Customization: You can't assign different directories to different minions. This is a major limitation.
  • No Creation Options: There's no way to specify a custom directory when you create a minion. This severely limits parallel development scenarios.

The Workflow Roadblock

With the current setup, you're facing limitations galore:

  • Git Worktree Paralysis: You can't have minions working in different git worktrees. This hinders parallel feature development.
  • Branch Switching Blues: You're stuck switching branches manually, which can be time-consuming and error-prone.
  • Repo Coordination Chaos: You can't have minions working in different but related repositories simultaneously, which blocks integration testing across different projects.
  • Sequential Workflow Sucks: You're forced to stick to a sequential workflow instead of embracing the power of parallelism.

The Bright Future: Expected Behavior

Now, imagine a world where your minions are independent, adaptable, and ready to tackle any task. That's the power of custom working directories!

Unleashing the Power of Flexible Directories

  • Individual Directories: Each minion gets its own dedicated working directory. This means true isolation.
  • Creation Flexibility: You can specify the working directory during minion creation. This is where the magic happens!
  • Default Compatibility: The default behavior will remain unchanged: minions will inherit the project directory, so it won't break anything.
  • Custom Override: You'll have the option to specify any valid filesystem path. Go wild!
  • No Restrictions: There's no need for the custom directory to be a subdirectory of the project. This opens up endless possibilities.

Enabled Workflows: Your Coding Superpowers

With this feature, you'll be able to unlock these amazing workflows:

  • Feature-Rich Collaboration: Minion A can work in the main worktree on feature-x branch.
  • Independent Development: Minion B can work in a separate worktree on feature-y branch.
  • Cross-Repo Integration: Minion C can work in a related repository for seamless integration testing. That's the real deal.
  • Seamless Coordination: All minions can coordinate through Legion comms.

Impact: Why This Matters

Let's be real: this feature is a game-changer. It'll transform your workflow and unlock new levels of productivity. Let's dig in a bit more on the effects it'll have.

The Pain Points (Current Limitations)

  • Parallelism Blocked: You can't parallelize work across branches or worktrees, which means you have to wait.
  • Manual Branching: You're forced to manually switch branches or use multiple projects. What a pain!
  • Conflict Risks: There's no safe parallel development, which raises the risk of conflicts and errors.
  • Testing Roadblock: You can't test integrations across repositories, which makes testing harder.
  • Sequential Stranglehold: You're stuck in a sequential workflow instead of enjoying the benefits of parallel development.

The Superpowers (Enabled Capabilities)

  • Concurrent Branching: Multiple minions can work on different branches simultaneously.
  • Isolated Feature Development: You can develop features in parallel and in isolation. Amazing!
  • Cross-Repo Testing: You can perform integration testing across different repositories.
  • Git Worktree Mastery: You can finally use git worktree-based workflows, which are common in large projects.
  • Multi-Repo Magic: You can coordinate complex multi-repo projects with ease.

Frequency: Who Benefits the Most?

This feature is high-value for these advanced workflows:

  • Large Teams: Large teams using git worktrees and working on multiple projects at the same time.
  • Multi-Repo Projects: Projects that require coordination across multiple repositories.
  • Parallel Development: Teams that want to do parallel feature development.
  • Integration Testing: Anyone doing integration testing scenarios.
  • Experimentation: Teams that want to run different experiments and different approaches simultaneously.

User Perspective: The Developer's Dream

Let's put ourselves in the shoes of a developer and see how this feature will impact their daily life.

User Story

As a developer using git worktrees for parallel development, I want to assign different minions to different worktrees, So that multiple features can be developed simultaneously without conflicts. It's that simple!

User Journey

  1. Project Setup: The user has a project with the main worktree at /repo/main.
  2. Worktree Creation: The user creates worktrees for features: /repo/feature-a, /repo/feature-b.
  3. Minion A Creation: The user creates Minion A with the working directory /repo/feature-a.
  4. Minion B Creation: The user creates Minion B with the working directory /repo/feature-b.
  5. Current Scenario: Currently, both minions share /repo/main, which isn't ideal.
  6. Expected Outcome: With the new feature, minions work independently in their assigned worktrees.
  7. Parallel Development: Both minions develop features in parallel, without any conflicts. Success!

Real-World Example: Parallel Feature Development

Let's look at a concrete example:

Scenario:

Project: web-application
├── /home/user/projects/web-app (main worktree)
├── /home/user/projects/web-app-auth-feature (worktree)
└── /home/user/projects/web-app-ui-redesign (worktree)

Legion Setup:
- Minion "auth-dev" → /home/user/projects/web-app-auth-feature
- Minion "ui-dev" → /home/user/projects/web-app-ui-redesign
- Minion "integration-test" → /home/user/projects/web-app (main)

Workflow:
1.  auth-dev implements authentication in isolation
2.  ui-dev redesigns UI in isolation
3.  integration-test verifies both features work together
4.  No branch switching, no conflicts, full parallelism

Quality Standards: Making It Rock Solid

We're not just throwing features out there; we're building quality into every line of code.

The Pillars of Quality

  • Backward Compatibility: It must support existing minions, inheriting the project directory. The default behavior should be unchanged to avoid breaking existing setups.
  • Data Persistence: The custom working directory must persist in the minion state, surviving application restarts.
  • Performance: There should be no performance impact, and file operations should be scoped to the minion's directory.
  • Security: It should validate directory paths, prevent access outside allowed paths, and provide clear error messages for invalid paths.

Acceptance Criteria: How We Know We're Done

Here's how we'll measure success:

Definition of Done

  • Minion creation UI includes an optional working directory field.
  • Users can specify a custom directory when creating a minion manually.
  • The spawn_minion MCP tool accepts an optional working_directory parameter.
  • Minions use the custom directory if specified; otherwise, they use the project directory.
  • The working directory persists in the minion state (state.json).
  • Directory validation is in place on minion creation (exists, is accessible).
  • Error handling is provided for invalid directories.
  • The minion SDK is initialized with the correct working directory.
  • Documentation is updated with working directory usage.

User Interface Updates

Create Minion Modal:

This is what the new UI will look like:

┌─────────────────────────────────────┐
│ Create Minion                       │
├─────────────────────────────────────┤
│ Name: [________________]            │
│                                     │
│ Role: [________________]            │
│                                     │
│ Working Directory (optional):       │
│ [/home/user/projects/feature-a ]    │
│ [Browse...] 📁                      │
│                                     │
│ ⚠️ Leave empty to use project       │
│    directory                        │
│                                     │
│ Capabilities: [________________]    │
│                                     │
│         [Cancel]  [Create Minion]   │
└─────────────────────────────────────┘

Browse Button:

The browse button will open the existing FolderBrowserModal component, allowing users to select any directory on the filesystem without restrictions.

MCP Tool Updates

spawn_minion Tool:

{
  "name": "spawn_minion",
  "parameters": {
    "name": "string (required)",
    "role": "string (required)",
    "capabilities": "array (optional)",
    "working_directory": "string (optional)"  # NEW
  },
  "description": "Spawn a new minion...\n\nworking_directory: Optional custom working directory. If not specified, inherits project directory. Can be any valid filesystem path."
}

Example Usage:

# Spawn minion in different worktree
spawn_minion(
  name="feature-auth-dev",
  role="Implement authentication system",
  working_directory="/home/user/repos/myapp-auth-worktree"
)

# Spawn minion in default project directory
spawn_minion(
  name="main-dev",
  role="Fix bugs in main branch"
  # No working_directory specified, uses project dir
)

Backend Updates

SessionInfo Extension (Already Has working_directory):

@dataclass
class SessionInfo:
    # Existing fields...
    working_directory: Path  # Already exists, now can differ from project

Minion Creation Logic:

async def create_minion(
    name: str,
    role: str,
    working_directory: Optional[str] = None,  # NEW parameter
    # Other parameters...
):
    # Use custom directory if provided, otherwise project directory
    minion_cwd = working_directory or project.working_directory

    # Validate directory exists and is accessible
    if not Path(minion_cwd).exists():
        raise ValueError(f"Working directory does not exist: {minion_cwd}")

    # Create minion session with custom working directory
    session = await session_manager.create_session(
        working_directory=minion_cwd,
        # Other session params...
    )

Validation Requirements:

  • The path must exist.
  • The path must be readable and writable.
  • The path should be absolute (convert relative paths).
  • Clear error messages should be provided for validation failures.

Test Scenarios: Putting It to the Test

Here's how we'll test the new feature:

  1. UI Creation: User creates a minion with a custom directory via the UI.
  2. Default Behavior: User creates a minion without a custom directory via the UI.
  3. MCP Tool Usage: Minion spawns a child with a custom directory via MCP.
  4. Invalid Directory: An invalid directory is specified.
  5. Restart Persistence: The working directory persists across restarts.
  6. Multiple Directories: Multiple minions are in different directories.
  7. Git Operations: Git operations in separate worktrees are tested.

Edge Cases: The Devil's in the Details

We'll also consider these edge cases:

  • Working directory is deleted while the minion is running.
  • Working directory has no read/write permissions.
  • Relative paths vs. absolute paths.
  • Network paths or remote filesystems.
  • Symlinks or junctions.
  • Very long path names.
  • Special characters in paths.
  • Cross-platform path handling (Windows vs. Unix).

Out of Scope: What We're Not Doing (Yet)

These items are out of scope for this specific issue:

  • Automatic git worktree creation.
  • Directory permission management.
  • Sandboxing or security isolation between minions.
  • Quotas or disk usage limits per minion.
  • Automatic cleanup of working directories.
  • Template directories or initialization scripts.
  • Synchronization between minion directories.

Additional Context: Deep Dive

Let's explore some extra details that will help you fully grasp the potential of this feature.

Git Worktree Workflow: The Key to Parallelism

What are Git Worktrees?

Git worktrees allow multiple working directories for the same repository, each on different branches. This enables:

  • Parallel development without branch switching.
  • Isolated testing environments.
  • Simultaneous work on multiple features.

Example Setup:

# Main repository
cd /home/user/myproject
git worktree add ../myproject-feature-a feature-a
git worktree add ../myproject-feature-b feature-b

# Now have 3 directories:
# /home/user/myproject (main branch)
# /home/user/myproject-feature-a (feature-a branch)
# /home/user/myproject-feature-b (feature-b branch)

Legion Integration:

With this feature, assign one minion per worktree for true parallel development.

Multi-Repo Workflows: Orchestrating Microservices

Microservices Example:

Legion Project: "Backend Services Integration"

Repositories:
- /repos/auth-service (Minion "auth-dev")
- /repos/payment-service (Minion "payment-dev")
- /repos/user-service (Minion "user-dev")
- /repos/integration-tests (Minion "integration-test")

Each minion works in its service repository, coordination via comms.

Current SessionInfo Structure: The Building Blocks

SessionInfo already has a working_directory field, so the backend may already support this partially. We need to:

  1. Expose it in the UI for manual minion creation.
  2. Add it to the MCP tool for spawn_minion.
  3. Ensure the SDK properly uses the custom directory.

Related Features: The Ecosystem

  • Existing: SessionInfo.working_directory (backend support) and FolderBrowserModal (UI for directory selection).
  • This Feature: UI integration for minion creation, MCP tool parameter addition, and validation logic.
  • Future (Out of Scope): Automatic worktree management, directory templates, and resource quotas per directory.

Security Considerations: Keeping It Safe

Path Validation:

  • Prevent directory traversal attacks (/../../../etc).
  • Validate paths before use.
  • Check read/write permissions.

Open Question:

Should there be restrictions on which directories minions can access?

  • Option A: No restrictions (trust model).
  • Option B: Whitelist approach (only allowed paths).
  • Option C: Validate but don't restrict (warn about unusual paths).

Recommendation: Option A for the MVP (trust model), Option B for future security hardening.

Implementation Complexity: How Hard Is It?

  • Low-Medium Complexity: The backend already supports a custom working_directory. We need UI updates (form field, validation), an MCP tool parameter addition, validation logic, and testing across multiple directories.
  • Estimated Effort: Medium (approximately 4-8 hours).

Conclusion

This feature unlocks advanced multi-minion workflows that are critical for scaling Legion to complex, real-world development scenarios. It is particularly valuable for teams using git worktrees or multi-repo architectures. Implementation builds on existing infrastructure (SessionInfo already tracks working_directory), primarily adding UI exposure and MCP tool integration.