# AGENTS.md - Requirements & Decisions

This document records the requirements, architecture decisions, and rationale for the S3 to HF Bucket Importer.

## Requirements

### Functional

1. **OAuth Login**: Users sign in with Hugging Face OAuth. Scopes: `manage-repos` (create buckets), `jobs` (run Jobs), plus default `openid` + `profile`.

2. **S3 Configuration**: Users provide AWS credentials (Access Key ID, Secret Access Key), region, bucket name, optional endpoint URL (for S3-compatible services like MinIO), and optional source prefix.

3. **File Browser**: Display S3 bucket contents in a tree view with:
   - Lazy loading (only load folder contents on expand)
   - Checkboxes for file/folder selection
   - Select all / Deselect all
   - File count and size statistics
   - CORS fallback mode if browser can't access S3 directly

4. **Destination Configuration**: User provides:
   - Bucket name (`bucket` or `namespace/bucket` format)
   - Optional destination prefix
   - Auto-create bucket if it doesn't exist

5. **Import Execution**: Launch an HF Job that:
   - Installs `huggingface_hub[s3]` from branch `cursor/s3-to-hf-bucket-ingestion-f144`
   - Runs `hf buckets import` with appropriate arguments
   - Passes S3 credentials as encrypted secrets
   - Uses CPU hardware (I/O-bound task)

6. **No Local Storage**: Nothing persists in the browser. Page refresh = start over. All credentials are in-memory only.

### Non-Functional

- Professional, confidence-inspiring design
- Dark theme inspired by HF Buckets announcement
- Responsive (works on mobile)
- Clear error messages and graceful degradation

## Architecture Decisions

### Decision 1: No Build Step

**Choice**: Vanilla HTML/CSS/JS with ES modules from CDN (esm.sh)

**Alternatives considered**:
- Vite + vanilla JS (tree-shaking, npm packages)
- React/Vue/Svelte (component model)

**Rationale**: 
- Simplest deployment: just upload files to a static Space
- No `node_modules`, no `package.json`, no build config
- CDN imports are cached by the browser across sessions
- The app is a single page with ~4 states - doesn't warrant a framework
- AWS SDK v3 is ~500KB from CDN but cached effectively
- Easy for anyone to maintain: just HTML/CSS/JS, no tooling knowledge needed

### Decision 2: CDN Provider (esm.sh)

**Choice**: esm.sh for ES module CDN

**Rationale**:
- Reliable ESM CDN that handles CommonJS -> ESM conversion
- Supports import maps natively
- Handles transitive dependencies for AWS SDK v3
- Alternative was jsdelivr+esm or unpkg, but esm.sh has better ESM support

### Decision 3: CORS Fallback Strategy

**Choice**: Try browser-side S3 listing, gracefully degrade to manual mode

**Alternatives considered**:
- Skip browser listing entirely (simpler but less interactive)
- Require CORS (better UX but higher friction)

**Rationale**:
- Most S3 buckets won't have CORS configured for our domain
- Attempting listing first gives the best UX for configured buckets
- Manual fallback (include/exclude patterns) lets everyone use the tool
- Clear CORS instructions help users enable browsing if they want

### Decision 4: Selection-to-Pattern Mapping

**Choice**: Compute include or exclude patterns, whichever set is smaller

**Rationale**:
- `hf buckets import` supports `--include` and `--exclude` with fnmatch patterns
- When most files are selected, use `--exclude` for the few deselected
- When few files are selected, use `--include` for the selected ones
- Folder-level patterns (e.g., `folder/*`) keep command lines short
- Shell-escaping prevents injection in the Job command

### Decision 5: Job Configuration

**Choice**: `python:3.12` Docker image with `pip install` + `bash -c`

**Alternatives considered**:
- Custom Docker image with huggingface_hub pre-installed
- `ghcr.io/astral-sh/uv` image with `uv run`

**Rationale**:
- `python:3.12` is a standard, well-maintained image
- `pip install` from git is simple and reliable
- `bash -c` allows chaining install + import in one command
- Once `hf buckets import` is released, change to `pip install 'huggingface_hub[s3]>=X.Y.Z'`
- CPU-only hardware (`cpu-basic`) is sufficient - this is purely I/O-bound

### Decision 6: Bucket Creation

**Choice**: Call `POST /api/repos/create` with `type: "bucket"` before submitting the Job

**Rationale**:
- The Job needs the bucket to exist before it can write to it
- Creating from the browser (pre-Job) ensures the user has proper permissions
- Handle 409 (already exists) gracefully

### Decision 7: Token Handling

**Choice**: Pass OAuth token as Job secret (`HF_TOKEN`), S3 credentials as secrets too

**Rationale**:
- Secrets are encrypted server-side by the Jobs API
- Injected as environment variables at runtime
- Never appear in logs or the HF UI
- The `hf` CLI automatically uses `HF_TOKEN` for authentication
- boto3/s3fs automatically uses `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`

## OAuth Scopes

| Scope | Purpose |
|-------|---------|
| `openid` | ID token (always included) |
| `profile` | Username, avatar (always included) |
| `manage-repos` | Create and manage buckets |
| `jobs` | Create and monitor HF Jobs |

## API Endpoints Used

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/api/whoami-v2` | Get user info + orgs |
| POST | `/api/repos/create` | Create destination bucket |
| POST | `/api/jobs/{namespace}` | Submit import Job |
| GET | `/api/jobs/{namespace}/{id}` | Poll Job status |

## Security Model

1. **S3 credentials**: Only transmitted from browser to HF Jobs API as encrypted secrets. Never stored in localStorage, cookies, or any persistent storage.

2. **OAuth token**: Held in JavaScript memory only. Not persisted. Passed to Job as encrypted secret for HF API authentication.

3. **Shell injection prevention**: All user inputs interpolated into the Job command are escaped using single-quote wrapping with internal quote escaping.

4. **No backend**: The static site has no server-side code. All API calls are made directly from the browser to S3 and HF APIs.

## File Structure

```
s3-importer/
├── README.md       # Space YAML frontmatter + project docs
├── AGENTS.md       # This file
├── index.html      # HTML structure + import maps
├── style.css       # Dark theme (CSS custom properties)
└── app.js          # Application logic (ES module, ~500 lines)
```

## Future Improvements

- [ ] Show Job logs inline (stream from `/api/jobs/{namespace}/{id}/logs`)
- [ ] Support AWS session tokens (for temporary credentials / assumed roles)
- [ ] Remember recent imports (optional, opt-in localStorage)
- [ ] Support importing to existing bucket with merge strategy
- [ ] Add a "dry run" option that shows what would be imported
- [ ] Update `huggingface_hub` install to stable release once available