mirror of
https://github.com/The-Low-Code-Foundation/OpenNoodl.git
synced 2026-01-12 15:22:55 +01:00
Finished node canvas UI tweaks. Failed to add connection highlighting
This commit is contained in:
Richard Osborne
@@ -0,0 +1,156 @@
|
||||
# TASK-007 Changelog
|
||||
|
||||
## Overview
|
||||
|
||||
This changelog tracks the implementation of the Integrated Local Backend feature, enabling zero-configuration full-stack development in Nodegex.
|
||||
|
||||
### Implementation Phases
|
||||
|
||||
1. **Phase A**: LocalSQL Adapter - SQLite-based CloudStore implementation
|
||||
2. **Phase B**: Backend Server - Express server with REST/WebSocket
|
||||
3. **Phase C**: Workflow Runtime - Visual workflow execution
|
||||
4. **Phase D**: Launcher Integration - Backend management UI
|
||||
5. **Phase E**: Migration & Export - Tools for moving to production
|
||||
6. **Phase F**: Standalone Deployment - Bundling backend with apps
|
||||
|
||||
---
|
||||
|
||||
## [Date TBD] - Task Created
|
||||
|
||||
### Summary
|
||||
|
||||
Task documentation created for Integrated Local Backend feature (TASK-007). This feature addresses the #1 friction point for new users by providing a zero-configuration backend that runs alongside the editor.
|
||||
|
||||
### Task Documents Created
|
||||
|
||||
- `README.md` - Full task specification (~1800 lines)
|
||||
- `CHECKLIST.md` - Implementation checklist (~400 items)
|
||||
- `CHANGELOG.md` - This file
|
||||
- `NOTES.md` - Working notes template
|
||||
|
||||
### Key Architectural Decisions
|
||||
|
||||
1. **Hybrid adapter approach**: Keep Parse support while adding local SQLite adapter
|
||||
2. **Same visual paradigm**: Backend workflows use identical node system as frontend
|
||||
3. **Launcher-managed backends**: Backends exist independently of projects, can be shared
|
||||
4. **WebSocket realtime**: Change notifications via WebSocket for SSE/WS node support
|
||||
5. **Bundled deployment**: Backend can be packaged with Electron apps
|
||||
|
||||
### Strategic Context
|
||||
|
||||
This feature transforms Nodegex from a frontend-focused visual editor into a true full-stack development platform. The zero-config experience removes the biggest barrier to adoption while maintaining clear migration paths to production backends.
|
||||
|
||||
---
|
||||
|
||||
## Progress Summary
|
||||
|
||||
| Phase | Status | Date Started | Date Completed | Hours |
|
||||
|-------|--------|--------------|----------------|-------|
|
||||
| Phase A: LocalSQL Adapter | Not Started | - | - | - |
|
||||
| Phase B: Backend Server | Not Started | - | - | - |
|
||||
| Phase C: Workflow Runtime | Not Started | - | - | - |
|
||||
| Phase D: Launcher Integration | Not Started | - | - | - |
|
||||
| Phase E: Migration & Export | Not Started | - | - | - |
|
||||
| Phase F: Standalone Deployment | Not Started | - | - | - |
|
||||
|
||||
---
|
||||
|
||||
## Session Log
|
||||
|
||||
### Template for Session Entries
|
||||
|
||||
```markdown
|
||||
## [YYYY-MM-DD] - Session N: [Phase Name]
|
||||
|
||||
### Summary
|
||||
[Brief description of what was accomplished]
|
||||
|
||||
### Files Created
|
||||
- `path/to/file.ts` - [Purpose]
|
||||
|
||||
### Files Modified
|
||||
- `path/to/file.ts` - [What changed and why]
|
||||
|
||||
### Technical Notes
|
||||
- [Key decisions made]
|
||||
- [Patterns discovered]
|
||||
- [Gotchas encountered]
|
||||
|
||||
### Testing Notes
|
||||
- [What was tested]
|
||||
- [Any edge cases discovered]
|
||||
|
||||
### Performance Notes
|
||||
- [Any performance observations]
|
||||
|
||||
### Next Steps
|
||||
- [What needs to be done next]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Blockers Log
|
||||
|
||||
_Track any blockers encountered during implementation_
|
||||
|
||||
| Date | Blocker | Resolution | Time Lost |
|
||||
|------|---------|------------|-----------|
|
||||
| - | - | - | - |
|
||||
|
||||
---
|
||||
|
||||
## Technical Decisions Log
|
||||
|
||||
_Record significant technical decisions made during implementation_
|
||||
|
||||
| Date | Decision | Rationale | Alternatives Considered |
|
||||
|------|----------|-----------|------------------------|
|
||||
| - | SQLite over embedded Postgres | Lightweight, zero-config, single file | PostgreSQL, MongoDB |
|
||||
| - | WebSocket for realtime | Native browser support, bi-directional | SSE, polling |
|
||||
| - | Express over Fastify | Already in codebase, team familiarity | Fastify, Koa |
|
||||
|
||||
---
|
||||
|
||||
## API Changes Log
|
||||
|
||||
_Track any changes to public APIs or interfaces_
|
||||
|
||||
| Date | Change | Migration Required |
|
||||
|------|--------|-------------------|
|
||||
| - | - | - |
|
||||
|
||||
---
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
_Record performance measurements during development_
|
||||
|
||||
| Date | Scenario | Measurement | Target | Status |
|
||||
|------|----------|-------------|--------|--------|
|
||||
| - | Query 10K records | - | <100ms | - |
|
||||
| - | WebSocket broadcast to 100 clients | - | <50ms | - |
|
||||
| - | Workflow hot reload | - | <1s | - |
|
||||
|
||||
---
|
||||
|
||||
## Known Issues
|
||||
|
||||
_Track known issues discovered during implementation_
|
||||
|
||||
| Issue | Severity | Workaround | Fix Planned |
|
||||
|-------|----------|------------|-------------|
|
||||
| - | - | - | - |
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
_Ideas for future improvements discovered during implementation_
|
||||
|
||||
1. External adapter for Supabase
|
||||
2. External adapter for PocketBase
|
||||
3. External adapter for Directus
|
||||
4. Visual schema editor in backend panel
|
||||
5. Query builder UI for data exploration
|
||||
6. Automated backup scheduling
|
||||
7. Multi-tenant support for deployed apps
|
||||
@@ -0,0 +1,435 @@
|
||||
# TASK-007: Integrated Local Backend - Implementation Checklist
|
||||
|
||||
## Pre-Implementation
|
||||
|
||||
- [ ] Review existing CloudStore implementation (`packages/noodl-runtime/src/api/cloudstore.js`)
|
||||
- [ ] Review existing CloudRunner implementation (`packages/noodl-viewer-cloud/src/`)
|
||||
- [ ] Review existing cloud-function-server (`packages/noodl-editor/src/main/src/cloud-function-server.js`)
|
||||
- [ ] Create feature branch: `feature/007-integrated-local-backend`
|
||||
- [ ] Set up development environment with SQLite tools
|
||||
|
||||
---
|
||||
|
||||
## Phase A: Foundation - LocalSQL Adapter (16-20 hours)
|
||||
|
||||
### A.1: SQLite Integration (4 hours)
|
||||
|
||||
- [ ] Install `better-sqlite3` dependency in `noodl-runtime`
|
||||
- [ ] Create adapter directory structure:
|
||||
```
|
||||
packages/noodl-runtime/src/api/adapters/
|
||||
├── index.ts
|
||||
├── cloudstore-adapter.ts
|
||||
└── local-sql/
|
||||
├── LocalSQLAdapter.ts
|
||||
├── SQLiteConnection.ts
|
||||
├── QueryBuilder.ts
|
||||
├── SchemaManager.ts
|
||||
└── types.ts
|
||||
```
|
||||
- [ ] Define `CloudStoreAdapter` interface with all required methods
|
||||
- [ ] Implement `SQLiteConnection` wrapper class
|
||||
- [ ] Implement basic `LocalSQLAdapter` constructor and connection
|
||||
- [ ] Add WAL mode pragma for better concurrency
|
||||
- [ ] Test: Database file creation and basic connection
|
||||
|
||||
### A.2: Query Translation (4 hours)
|
||||
|
||||
- [ ] Implement `QueryBuilder.buildSelect()` for basic queries
|
||||
- [ ] Implement WHERE clause translation for operators:
|
||||
- [ ] `equalTo` / `notEqualTo`
|
||||
- [ ] `greaterThan` / `lessThan` / `greaterThanOrEqualTo` / `lessThanOrEqualTo`
|
||||
- [ ] `containedIn` / `notContainedIn`
|
||||
- [ ] `exists`
|
||||
- [ ] `contains` / `startsWith` / `endsWith`
|
||||
- [ ] `regex` (via GLOB)
|
||||
- [ ] Implement ORDER BY clause translation
|
||||
- [ ] Implement LIMIT/OFFSET translation
|
||||
- [ ] Implement `QueryBuilder.buildInsert()`
|
||||
- [ ] Implement `QueryBuilder.buildUpdate()`
|
||||
- [ ] Implement `QueryBuilder.buildDelete()`
|
||||
- [ ] Test: Complex queries with multiple operators
|
||||
- [ ] Test: Query performance with 10,000+ records
|
||||
|
||||
### A.3: Schema Management (4 hours)
|
||||
|
||||
- [ ] Define `TableSchema` and `ColumnDefinition` interfaces
|
||||
- [ ] Implement `SchemaManager.createTable()`
|
||||
- [ ] Map Noodl types to SQLite types:
|
||||
- [ ] String → TEXT
|
||||
- [ ] Number → REAL
|
||||
- [ ] Boolean → INTEGER (0/1)
|
||||
- [ ] Date → TEXT (ISO8601)
|
||||
- [ ] Object/Array → TEXT (JSON)
|
||||
- [ ] Pointer → TEXT (objectId reference)
|
||||
- [ ] Relation → Junction table
|
||||
- [ ] Implement `SchemaManager.addColumn()`
|
||||
- [ ] Implement `SchemaManager.exportSchema()`
|
||||
- [ ] Implement `SchemaManager.generatePostgresSQL()`
|
||||
- [ ] Implement `SchemaManager.generateSupabaseSQL()`
|
||||
- [ ] Create automatic indexes for `createdAt`, `updatedAt`
|
||||
- [ ] Test: Schema creation and migration
|
||||
- [ ] Test: Export to PostgreSQL format
|
||||
|
||||
### A.4: Adapter Registration & Selection (4 hours)
|
||||
|
||||
- [ ] Create `AdapterRegistry` singleton class
|
||||
- [ ] Implement `createAdapter()` factory method
|
||||
- [ ] Refactor existing `CloudStore` to use adapter pattern
|
||||
- [ ] Create `ParseAdapter` wrapper around existing code
|
||||
- [ ] Implement adapter switching based on project config
|
||||
- [ ] Add adapter lifecycle methods (connect/disconnect)
|
||||
- [ ] Test: Adapter switching between local and Parse
|
||||
- [ ] Test: Multiple simultaneous adapters
|
||||
|
||||
### Phase A Verification
|
||||
|
||||
- [ ] All query operators work correctly
|
||||
- [ ] Schema creation handles all Noodl data types
|
||||
- [ ] Export generates valid PostgreSQL SQL
|
||||
- [ ] No regressions in Parse adapter functionality
|
||||
- [ ] Performance benchmarks documented
|
||||
|
||||
---
|
||||
|
||||
## Phase B: Local Backend Server (12-16 hours)
|
||||
|
||||
### B.1: Server Architecture (4 hours)
|
||||
|
||||
- [ ] Create `local-backend` directory in `noodl-editor/src/main/src/`
|
||||
- [ ] Implement `LocalBackendServer` class
|
||||
- [ ] Set up Express middleware (CORS, JSON parsing)
|
||||
- [ ] Implement REST routes:
|
||||
- [ ] `GET /health` - Health check
|
||||
- [ ] `GET /api/_schema` - Get schema
|
||||
- [ ] `POST /api/_schema` - Update schema
|
||||
- [ ] `GET /api/_export` - Export data
|
||||
- [ ] `GET /api/:table` - Query records
|
||||
- [ ] `GET /api/:table/:id` - Fetch single record
|
||||
- [ ] `POST /api/:table` - Create record
|
||||
- [ ] `PUT /api/:table/:id` - Update record
|
||||
- [ ] `DELETE /api/:table/:id` - Delete record
|
||||
- [ ] `POST /functions/:name` - Execute workflow
|
||||
- [ ] `POST /api/_batch` - Batch operations
|
||||
- [ ] Test: All REST endpoints with curl/Postman
|
||||
|
||||
### B.2: WebSocket Realtime (3 hours)
|
||||
|
||||
- [ ] Add `ws` dependency
|
||||
- [ ] Implement WebSocket server on same HTTP server
|
||||
- [ ] Implement client subscription tracking
|
||||
- [ ] Implement `broadcast()` method for events
|
||||
- [ ] Wire adapter events to WebSocket broadcast
|
||||
- [ ] Handle client disconnection cleanup
|
||||
- [ ] Test: Subscribe and receive realtime updates
|
||||
- [ ] Test: Multiple clients receive broadcasts
|
||||
|
||||
### B.3: Backend Manager (4 hours)
|
||||
|
||||
- [ ] Create `BackendManager` singleton class
|
||||
- [ ] Implement backend storage structure:
|
||||
```
|
||||
~/.noodl/backends/{id}/
|
||||
├── config.json
|
||||
├── schema.json
|
||||
├── workflows/
|
||||
└── data/local.db
|
||||
```
|
||||
- [ ] Implement IPC handlers:
|
||||
- [ ] `backend:list` - List all backends
|
||||
- [ ] `backend:create` - Create new backend
|
||||
- [ ] `backend:delete` - Delete backend
|
||||
- [ ] `backend:start` - Start backend server
|
||||
- [ ] `backend:stop` - Stop backend server
|
||||
- [ ] `backend:status` - Get backend status
|
||||
- [ ] `backend:export-schema` - Export schema
|
||||
- [ ] `backend:export-data` - Export data
|
||||
- [ ] Implement automatic port allocation
|
||||
- [ ] Wire BackendManager into main process
|
||||
- [ ] Test: Create/start/stop backend from IPC
|
||||
|
||||
### B.4: Editor Integration (3 hours)
|
||||
|
||||
- [ ] Create `BackendModel` in editor
|
||||
- [ ] Implement `loadProjectBackend()` method
|
||||
- [ ] Add backend status to project model
|
||||
- [ ] Create backend indicator in editor header
|
||||
- [ ] Implement start/stop controls in UI
|
||||
- [ ] Test: Backend starts with project if autoStart enabled
|
||||
- [ ] Test: Backend stops when project closes
|
||||
|
||||
### Phase B Verification
|
||||
|
||||
- [ ] Backend server starts and responds to requests
|
||||
- [ ] Realtime WebSocket updates work
|
||||
- [ ] Multiple backends can run simultaneously
|
||||
- [ ] IPC commands work from renderer process
|
||||
- [ ] Editor shows correct backend status
|
||||
|
||||
---
|
||||
|
||||
## Phase C: Visual Workflow Runtime (12-16 hours)
|
||||
|
||||
### C.1: Runtime Adaptation (4 hours)
|
||||
|
||||
- [ ] Review existing `CloudRunner` implementation
|
||||
- [ ] Modify CloudRunner to accept local adapter
|
||||
- [ ] Remove Parse-specific dependencies from base runtime
|
||||
- [ ] Add adapter injection to runtime context
|
||||
- [ ] Ensure `sandbox.isolate.js` works in pure Node.js
|
||||
- [ ] Test: CloudRunner executes workflows with local adapter
|
||||
|
||||
### C.2: Database Nodes (4 hours)
|
||||
|
||||
- [ ] Create `noodl.local.query` node
|
||||
- [ ] Create `noodl.local.insert` node
|
||||
- [ ] Create `noodl.local.update` node
|
||||
- [ ] Create `noodl.local.delete` node
|
||||
- [ ] Create `noodl.local.transaction` node (optional)
|
||||
- [ ] Add nodes to cloud viewer node registry
|
||||
- [ ] Test: Each node type in isolation
|
||||
- [ ] Test: Nodes work in visual workflow
|
||||
|
||||
### C.3: Trigger Nodes (4 hours)
|
||||
|
||||
- [ ] Create `noodl.trigger.schedule` node (cron-based)
|
||||
- [ ] Create `noodl.trigger.dbChange` node
|
||||
- [ ] Create `noodl.trigger.webhook` node
|
||||
- [ ] Implement trigger registration system
|
||||
- [ ] Implement trigger cleanup on node deletion
|
||||
- [ ] Test: Schedule triggers at specified intervals
|
||||
- [ ] Test: DB change triggers fire correctly
|
||||
- [ ] Test: Webhook triggers receive HTTP requests
|
||||
|
||||
### C.4: Workflow Hot Reload (4 hours)
|
||||
|
||||
- [ ] Create `WorkflowCompiler` class
|
||||
- [ ] Implement debounced compilation on component change
|
||||
- [ ] Export workflows to backend workflows directory
|
||||
- [ ] Implement `backend:update-workflow` IPC handler
|
||||
- [ ] Implement `backend:reload-workflows` IPC handler
|
||||
- [ ] Test: Workflow changes reflect immediately
|
||||
- [ ] Test: No service interruption during reload
|
||||
|
||||
### Phase C Verification
|
||||
|
||||
- [ ] CloudRunner executes workflows with local database
|
||||
- [ ] All database node types work correctly
|
||||
- [ ] All trigger types fire at appropriate times
|
||||
- [ ] Hot reload works without restarting backend
|
||||
- [ ] No memory leaks in long-running workflows
|
||||
|
||||
---
|
||||
|
||||
## Phase D: Launcher Integration (8-10 hours)
|
||||
|
||||
### D.1: Backend List UI (3 hours)
|
||||
|
||||
- [ ] Create `BackendManager` directory in Launcher views
|
||||
- [ ] Create `BackendList.tsx` component
|
||||
- [ ] Create `BackendCard.tsx` component
|
||||
- [ ] Implement backend loading from IPC
|
||||
- [ ] Implement status indicators (running/stopped)
|
||||
- [ ] Add start/stop controls per backend
|
||||
- [ ] Add delete button with confirmation
|
||||
- [ ] Test: List displays all backends correctly
|
||||
|
||||
### D.2: Create Backend UI (2 hours)
|
||||
|
||||
- [ ] Create `CreateBackendDialog.tsx` component
|
||||
- [ ] Implement name input with validation
|
||||
- [ ] Call `backend:create` on confirmation
|
||||
- [ ] Refresh list after creation
|
||||
- [ ] Test: New backend appears in list
|
||||
|
||||
### D.3: Backend Selector (3 hours)
|
||||
|
||||
- [ ] Create `BackendSelector.tsx` dropdown component
|
||||
- [ ] Show in project card when editing settings
|
||||
- [ ] Implement backend association saving
|
||||
- [ ] Update project config on selection
|
||||
- [ ] Option to create new backend from selector
|
||||
- [ ] Test: Backend association persists
|
||||
- [ ] Test: Switching backends works correctly
|
||||
|
||||
### D.4: Backend Admin Panel (2 hours)
|
||||
|
||||
- [ ] Create basic data viewer component
|
||||
- [ ] Show tables and record counts
|
||||
- [ ] Allow basic CRUD operations
|
||||
- [ ] Show recent activity/logs
|
||||
- [ ] Test: Data displays correctly
|
||||
- [ ] Test: CRUD operations work
|
||||
|
||||
### Phase D Verification
|
||||
|
||||
- [ ] Launcher shows all backends with status
|
||||
- [ ] Can create/delete backends from launcher
|
||||
- [ ] Can start/stop backends independently
|
||||
- [ ] Projects can be associated with backends
|
||||
- [ ] Basic data administration works
|
||||
|
||||
---
|
||||
|
||||
## Phase E: Migration & Export (8-10 hours)
|
||||
|
||||
### E.1: Schema Export Wizard (4 hours)
|
||||
|
||||
- [ ] Create `ExportWizard.tsx` component
|
||||
- [ ] Implement format selection (Postgres, Supabase, PocketBase, JSON)
|
||||
- [ ] Implement include data option
|
||||
- [ ] Generate export via IPC
|
||||
- [ ] Display result in code viewer
|
||||
- [ ] Implement copy to clipboard
|
||||
- [ ] Implement download as file
|
||||
- [ ] Test: Each export format generates valid output
|
||||
|
||||
### E.2: Parse Migration Wizard (4 hours)
|
||||
|
||||
- [ ] Create `ParseMigrationWizard.tsx` component
|
||||
- [ ] Implement Parse schema fetching
|
||||
- [ ] Display schema review with record counts
|
||||
- [ ] Implement migration progress UI
|
||||
- [ ] Create local backend during migration
|
||||
- [ ] Import schema structure
|
||||
- [ ] Import data in batches
|
||||
- [ ] Handle migration errors gracefully
|
||||
- [ ] Test: Full migration from Parse to local
|
||||
|
||||
### E.3: External Migration Guide (2 hours)
|
||||
|
||||
- [ ] Create migration documentation
|
||||
- [ ] Document Supabase migration steps
|
||||
- [ ] Document PocketBase migration steps
|
||||
- [ ] Add links from export wizard
|
||||
- [ ] Test: Following guide successfully migrates
|
||||
|
||||
### Phase E Verification
|
||||
|
||||
- [ ] Schema exports to all target formats
|
||||
- [ ] Parse migration preserves all data
|
||||
- [ ] Migration can be cancelled/resumed
|
||||
- [ ] Export documentation is complete
|
||||
|
||||
---
|
||||
|
||||
## Phase F: Standalone Deployment (8-10 hours)
|
||||
|
||||
### F.1: Backend Bundler (4 hours)
|
||||
|
||||
- [ ] Create `backend-bundler.ts` utility
|
||||
- [ ] Pre-compile server bundle for Node.js
|
||||
- [ ] Pre-compile server bundle for Electron
|
||||
- [ ] Implement schema copying
|
||||
- [ ] Implement workflows copying
|
||||
- [ ] Implement optional data copying
|
||||
- [ ] Generate package.json for standalone
|
||||
- [ ] Generate startup script
|
||||
- [ ] Test: Bundled backend runs standalone
|
||||
|
||||
### F.2: Electron Deployment Integration (4 hours)
|
||||
|
||||
- [ ] Modify Electron deployer to support backend
|
||||
- [ ] Add "Include Backend" option to deploy wizard
|
||||
- [ ] Add "Include Data" sub-option
|
||||
- [ ] Integrate backend bundle into Electron package
|
||||
- [ ] Modify Electron main.js to start backend
|
||||
- [ ] Handle backend cleanup on app quit
|
||||
- [ ] Test: Electron app with embedded backend works
|
||||
- [ ] Test: App starts backend on launch
|
||||
|
||||
### F.3: Documentation (2 hours)
|
||||
|
||||
- [ ] Document standalone deployment process
|
||||
- [ ] Document backend configuration options
|
||||
- [ ] Document production considerations
|
||||
- [ ] Add troubleshooting guide
|
||||
- [ ] Test: Follow docs to deploy successfully
|
||||
|
||||
### Phase F Verification
|
||||
|
||||
- [ ] Backend can be bundled for standalone use
|
||||
- [ ] Electron apps include working backend
|
||||
- [ ] Deployed apps work offline
|
||||
- [ ] Documentation covers all scenarios
|
||||
|
||||
---
|
||||
|
||||
## Final Verification
|
||||
|
||||
### Functional Testing
|
||||
|
||||
- [ ] New user can create backend with zero configuration
|
||||
- [ ] Visual workflows execute correctly in local backend
|
||||
- [ ] Realtime updates work via WebSocket
|
||||
- [ ] All database operations work correctly
|
||||
- [ ] All trigger types fire correctly
|
||||
- [ ] Schema export produces valid output
|
||||
- [ ] Parse migration preserves data
|
||||
- [ ] Electron deployment works with backend
|
||||
|
||||
### Backward Compatibility
|
||||
|
||||
- [ ] Existing Parse projects load without errors
|
||||
- [ ] Parse adapter functions correctly
|
||||
- [ ] Can switch from Parse to local backend
|
||||
- [ ] No regressions in existing functionality
|
||||
|
||||
### Performance
|
||||
|
||||
- [ ] Query performance acceptable with 100K records
|
||||
- [ ] WebSocket can handle 100 simultaneous clients
|
||||
- [ ] Hot reload completes within 1 second
|
||||
- [ ] Memory usage stable over time
|
||||
|
||||
### Documentation
|
||||
|
||||
- [ ] README covers all features
|
||||
- [ ] API documentation complete
|
||||
- [ ] Migration guides complete
|
||||
- [ ] Troubleshooting guide complete
|
||||
|
||||
---
|
||||
|
||||
## Post-Implementation
|
||||
|
||||
- [ ] Update main phase documentation with completion status
|
||||
- [ ] Create PR with comprehensive description
|
||||
- [ ] Request code review
|
||||
- [ ] Address review feedback
|
||||
- [ ] Merge to development branch
|
||||
- [ ] Update CHANGELOG.md
|
||||
- [ ] Plan Phase 5 follow-up tasks (external adapters)
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference: Key Files
|
||||
|
||||
| Purpose | File Path |
|
||||
|---------|-----------|
|
||||
| Adapter Interface | `packages/noodl-runtime/src/api/adapters/cloudstore-adapter.ts` |
|
||||
| LocalSQL Adapter | `packages/noodl-runtime/src/api/adapters/local-sql/LocalSQLAdapter.ts` |
|
||||
| Backend Server | `packages/noodl-editor/src/main/src/local-backend/LocalBackendServer.ts` |
|
||||
| Backend Manager | `packages/noodl-editor/src/main/src/local-backend/BackendManager.ts` |
|
||||
| Backend Model | `packages/noodl-editor/src/editor/src/models/BackendModel.ts` |
|
||||
| Launcher UI | `packages/noodl-editor/src/editor/src/views/Launcher/BackendManager/` |
|
||||
| Database Nodes | `packages/noodl-viewer-cloud/src/nodes/database/` |
|
||||
| Trigger Nodes | `packages/noodl-viewer-cloud/src/nodes/triggers/` |
|
||||
| Workflow Compiler | `packages/noodl-editor/src/editor/src/utils/workflow-compiler.ts` |
|
||||
| Backend Bundler | `packages/noodl-editor/src/editor/src/utils/deployment/backend-bundler.ts` |
|
||||
|
||||
---
|
||||
|
||||
## Estimated Time by Phase
|
||||
|
||||
| Phase | Description | Estimated Hours |
|
||||
|-------|-------------|-----------------|
|
||||
| A | LocalSQL Adapter | 16-20 |
|
||||
| B | Backend Server | 12-16 |
|
||||
| C | Workflow Runtime | 12-16 |
|
||||
| D | Launcher Integration | 8-10 |
|
||||
| E | Migration & Export | 8-10 |
|
||||
| F | Standalone Deployment | 8-10 |
|
||||
| **Total** | | **64-82 hours** |
|
||||
|
||||
**Recommended Session Structure:** 4-6 hour Cline sessions, one sub-phase per session
|
||||
@@ -0,0 +1,554 @@
|
||||
# TASK-007: Integrated Local Backend - Working Notes
|
||||
|
||||
## Quick Links
|
||||
|
||||
- [README.md](./README.md) - Full specification
|
||||
- [CHECKLIST.md](./CHECKLIST.md) - Implementation checklist
|
||||
- [CHANGELOG.md](./CHANGELOG.md) - Progress tracking
|
||||
|
||||
---
|
||||
|
||||
## Key Code Locations
|
||||
|
||||
### Existing Code to Study
|
||||
|
||||
```
|
||||
# CloudStore (current implementation)
|
||||
packages/noodl-runtime/src/api/cloudstore.js
|
||||
|
||||
# Cloud Runtime
|
||||
packages/noodl-viewer-cloud/src/index.ts
|
||||
packages/noodl-viewer-cloud/src/sandbox.isolate.js
|
||||
packages/noodl-viewer-cloud/src/sandbox.viewer.js
|
||||
|
||||
# Cloud Function Server
|
||||
packages/noodl-editor/src/main/src/cloud-function-server.js
|
||||
|
||||
# Parse Dashboard
|
||||
packages/noodl-parse-dashboard/
|
||||
|
||||
# Existing Data Nodes
|
||||
packages/noodl-runtime/src/nodes/std-library/data/dbcollectionnode2.js
|
||||
packages/noodl-runtime/src/api/records.js
|
||||
|
||||
# Cloud Nodes
|
||||
packages/noodl-viewer-cloud/src/nodes/cloud/request.ts
|
||||
packages/noodl-viewer-cloud/src/nodes/cloud/response.ts
|
||||
packages/noodl-viewer-cloud/src/nodes/data/aggregatenode.js
|
||||
```
|
||||
|
||||
### Key Interfaces to Implement
|
||||
|
||||
```typescript
|
||||
// From cloudstore.js - methods to implement in LocalSQLAdapter
|
||||
interface CloudStoreMethods {
|
||||
query(options): void; // Query records
|
||||
fetch(options): void; // Fetch single record
|
||||
count(options): void; // Count records
|
||||
aggregate(options): void; // Aggregate query
|
||||
create(options): void; // Create record
|
||||
save(options): void; // Update record
|
||||
delete(options): void; // Delete record
|
||||
addRelation(options): void; // Add relation
|
||||
removeRelation(options): void; // Remove relation
|
||||
increment(options): void; // Increment field
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Parse Query Syntax Reference
|
||||
|
||||
The LocalSQL adapter needs to translate these Parse query operators:
|
||||
|
||||
```javascript
|
||||
// Comparison
|
||||
{ field: { equalTo: value } }
|
||||
{ field: { notEqualTo: value } }
|
||||
{ field: { greaterThan: value } }
|
||||
{ field: { lessThan: value } }
|
||||
{ field: { greaterThanOrEqualTo: value } }
|
||||
{ field: { lessThanOrEqualTo: value } }
|
||||
|
||||
// Array
|
||||
{ field: { containedIn: [values] } }
|
||||
{ field: { notContainedIn: [values] } }
|
||||
{ field: { containsAll: [values] } }
|
||||
|
||||
// String
|
||||
{ field: { contains: "substring" } }
|
||||
{ field: { startsWith: "prefix" } }
|
||||
{ field: { endsWith: "suffix" } }
|
||||
{ field: { regex: "pattern" } }
|
||||
|
||||
// Existence
|
||||
{ field: { exists: true/false } }
|
||||
|
||||
// Logical
|
||||
{ and: [conditions] }
|
||||
{ or: [conditions] }
|
||||
|
||||
// Relations
|
||||
{ field: { pointsTo: objectId } }
|
||||
{ field: { relatedTo: { object, key } } }
|
||||
|
||||
// Sorting
|
||||
{ sort: ['field'] } // Ascending
|
||||
{ sort: ['-field'] } // Descending
|
||||
|
||||
// Pagination
|
||||
{ limit: 100, skip: 0 }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SQLite Type Mapping
|
||||
|
||||
```
|
||||
Noodl Type → SQLite Type → Notes
|
||||
─────────────────────────────────────────────
|
||||
String → TEXT → UTF-8 strings
|
||||
Number → REAL → 64-bit float
|
||||
Boolean → INTEGER → 0 or 1
|
||||
Date → TEXT → ISO8601 format
|
||||
Object → TEXT → JSON stringified
|
||||
Array → TEXT → JSON stringified
|
||||
Pointer → TEXT → objectId reference
|
||||
Relation → (junction table) → Separate table
|
||||
File → TEXT → URL or base64
|
||||
GeoPoint → TEXT → JSON {lat, lng}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backend Storage Structure
|
||||
|
||||
```
|
||||
~/.noodl/
|
||||
├── backends/
|
||||
│ ├── backend-abc123/
|
||||
│ │ ├── config.json # Backend configuration
|
||||
│ │ │ {
|
||||
│ │ │ "id": "backend-abc123",
|
||||
│ │ │ "name": "My Backend",
|
||||
│ │ │ "createdAt": "2024-01-15T...",
|
||||
│ │ │ "port": 8577,
|
||||
│ │ │ "projectIds": ["proj-1", "proj-2"]
|
||||
│ │ │ }
|
||||
│ │ │
|
||||
│ │ ├── schema.json # Schema definition (git-trackable)
|
||||
│ │ │ {
|
||||
│ │ │ "version": 1,
|
||||
│ │ │ "tables": [
|
||||
│ │ │ {
|
||||
│ │ │ "name": "todos",
|
||||
│ │ │ "columns": [
|
||||
│ │ │ { "name": "title", "type": "String" },
|
||||
│ │ │ { "name": "completed", "type": "Boolean" }
|
||||
│ │ │ ]
|
||||
│ │ │ }
|
||||
│ │ │ ]
|
||||
│ │ │ }
|
||||
│ │ │
|
||||
│ │ ├── workflows/ # Compiled visual workflows
|
||||
│ │ │ ├── GetTodos.workflow.json
|
||||
│ │ │ └── CreateTodo.workflow.json
|
||||
│ │ │
|
||||
│ │ └── data/
|
||||
│ │ └── local.db # SQLite database
|
||||
│ │
|
||||
│ └── backend-xyz789/
|
||||
│ └── ...
|
||||
│
|
||||
├── projects/
|
||||
│ └── my-project/
|
||||
│ └── noodl.project.json
|
||||
│ {
|
||||
│ "name": "My Project",
|
||||
│ "backend": {
|
||||
│ "type": "local",
|
||||
│ "id": "backend-abc123",
|
||||
│ "settings": {
|
||||
│ "autoStart": true
|
||||
│ }
|
||||
│ }
|
||||
│ }
|
||||
│
|
||||
└── launcher-config.json # Global launcher settings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## IPC API Design
|
||||
|
||||
```typescript
|
||||
// Main Process Handlers (BackendManager)
|
||||
|
||||
// List all backends
|
||||
ipcMain.handle('backend:list', async () => {
|
||||
return BackendMetadata[];
|
||||
});
|
||||
|
||||
// Create new backend
|
||||
ipcMain.handle('backend:create', async (_, name: string) => {
|
||||
return BackendMetadata;
|
||||
});
|
||||
|
||||
// Delete backend
|
||||
ipcMain.handle('backend:delete', async (_, id: string) => {
|
||||
return void;
|
||||
});
|
||||
|
||||
// Start backend server
|
||||
ipcMain.handle('backend:start', async (_, id: string) => {
|
||||
return void;
|
||||
});
|
||||
|
||||
// Stop backend server
|
||||
ipcMain.handle('backend:stop', async (_, id: string) => {
|
||||
return void;
|
||||
});
|
||||
|
||||
// Get backend status
|
||||
ipcMain.handle('backend:status', async (_, id: string) => {
|
||||
return { running: boolean; port?: number };
|
||||
});
|
||||
|
||||
// Export schema
|
||||
ipcMain.handle('backend:export-schema', async (_, id: string, format: string) => {
|
||||
return string; // SQL or JSON
|
||||
});
|
||||
|
||||
// Export data
|
||||
ipcMain.handle('backend:export-data', async (_, id: string, format: string) => {
|
||||
return string; // SQL or JSON
|
||||
});
|
||||
|
||||
// Update workflow
|
||||
ipcMain.handle('backend:update-workflow', async (_, params: {
|
||||
backendId: string;
|
||||
name: string;
|
||||
workflow: object;
|
||||
}) => {
|
||||
return void;
|
||||
});
|
||||
|
||||
// Reload all workflows
|
||||
ipcMain.handle('backend:reload-workflows', async (_, id: string) => {
|
||||
return void;
|
||||
});
|
||||
|
||||
// Import Parse schema
|
||||
ipcMain.handle('backend:import-parse-schema', async (_, params: {
|
||||
backendId: string;
|
||||
schema: object;
|
||||
}) => {
|
||||
return void;
|
||||
});
|
||||
|
||||
// Import records
|
||||
ipcMain.handle('backend:import-records', async (_, params: {
|
||||
backendId: string;
|
||||
collection: string;
|
||||
records: object[];
|
||||
}) => {
|
||||
return void;
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## REST API Design
|
||||
|
||||
```
|
||||
Base URL: http://localhost:{port}
|
||||
|
||||
# Health Check
|
||||
GET /health
|
||||
Response: { "status": "ok", "backend": "name" }
|
||||
|
||||
# Schema
|
||||
GET /api/_schema
|
||||
Response: { "tables": [...] }
|
||||
|
||||
POST /api/_schema
|
||||
Body: { "tables": [...] }
|
||||
Response: { "success": true }
|
||||
|
||||
# Export
|
||||
GET /api/_export?format=postgres|supabase|json&includeData=true
|
||||
Response: string (SQL or JSON)
|
||||
|
||||
# Query Records
|
||||
GET /api/{table}?where={json}&sort={json}&limit=100&skip=0
|
||||
Response: { "results": [...] }
|
||||
|
||||
# Fetch Single Record
|
||||
GET /api/{table}/{id}
|
||||
Response: { "objectId": "...", ... }
|
||||
|
||||
# Create Record
|
||||
POST /api/{table}
|
||||
Body: { "field": "value", ... }
|
||||
Response: { "objectId": "...", "createdAt": "..." }
|
||||
|
||||
# Update Record
|
||||
PUT /api/{table}/{id}
|
||||
Body: { "field": "newValue", ... }
|
||||
Response: { "updatedAt": "..." }
|
||||
|
||||
# Delete Record
|
||||
DELETE /api/{table}/{id}
|
||||
Response: { "success": true }
|
||||
|
||||
# Execute Workflow
|
||||
POST /functions/{name}
|
||||
Body: { ... }
|
||||
Response: { "result": ... }
|
||||
|
||||
# Batch Operations
|
||||
POST /api/_batch
|
||||
Body: {
|
||||
"requests": [
|
||||
{ "method": "POST", "path": "/api/todos", "body": {...} },
|
||||
{ "method": "PUT", "path": "/api/todos/abc", "body": {...} }
|
||||
]
|
||||
}
|
||||
Response: { "results": [...] }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## WebSocket Protocol
|
||||
|
||||
```typescript
|
||||
// Client → Server: Subscribe to collection
|
||||
{
|
||||
"type": "subscribe",
|
||||
"collection": "todos"
|
||||
}
|
||||
|
||||
// Client → Server: Unsubscribe
|
||||
{
|
||||
"type": "unsubscribe",
|
||||
"collection": "todos"
|
||||
}
|
||||
|
||||
// Server → Client: Record created
|
||||
{
|
||||
"event": "create",
|
||||
"data": {
|
||||
"collection": "todos",
|
||||
"object": { "objectId": "...", ... }
|
||||
},
|
||||
"timestamp": 1234567890
|
||||
}
|
||||
|
||||
// Server → Client: Record updated
|
||||
{
|
||||
"event": "save",
|
||||
"data": {
|
||||
"collection": "todos",
|
||||
"objectId": "...",
|
||||
"object": { ... }
|
||||
},
|
||||
"timestamp": 1234567890
|
||||
}
|
||||
|
||||
// Server → Client: Record deleted
|
||||
{
|
||||
"event": "delete",
|
||||
"data": {
|
||||
"collection": "todos",
|
||||
"objectId": "..."
|
||||
},
|
||||
"timestamp": 1234567890
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Node Definitions
|
||||
|
||||
### noodl.local.query
|
||||
|
||||
```typescript
|
||||
{
|
||||
name: 'noodl.local.query',
|
||||
displayNodeName: 'Query Records',
|
||||
category: 'Local Database',
|
||||
|
||||
inputs: {
|
||||
collection: { type: 'string' },
|
||||
where: { type: 'query-filter' },
|
||||
sort: { type: 'array' },
|
||||
limit: { type: 'number', default: 100 },
|
||||
skip: { type: 'number', default: 0 },
|
||||
fetch: { type: 'signal' }
|
||||
},
|
||||
|
||||
outputs: {
|
||||
results: { type: 'array' },
|
||||
count: { type: 'number' },
|
||||
success: { type: 'signal' },
|
||||
failure: { type: 'signal' },
|
||||
error: { type: 'string' }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### noodl.trigger.schedule
|
||||
|
||||
```typescript
|
||||
{
|
||||
name: 'noodl.trigger.schedule',
|
||||
displayNodeName: 'Schedule Trigger',
|
||||
category: 'Triggers',
|
||||
singleton: true,
|
||||
|
||||
inputs: {
|
||||
cron: { type: 'string', default: '0 * * * *' },
|
||||
enabled: { type: 'boolean', default: true }
|
||||
},
|
||||
|
||||
outputs: {
|
||||
triggered: { type: 'signal' },
|
||||
lastRun: { type: 'date' }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### noodl.trigger.dbChange
|
||||
|
||||
```typescript
|
||||
{
|
||||
name: 'noodl.trigger.dbChange',
|
||||
displayNodeName: 'Database Change Trigger',
|
||||
category: 'Triggers',
|
||||
singleton: true,
|
||||
|
||||
inputs: {
|
||||
collection: { type: 'string' },
|
||||
events: {
|
||||
type: 'enum',
|
||||
enums: ['all', 'create', 'save', 'delete'],
|
||||
default: 'all'
|
||||
}
|
||||
},
|
||||
|
||||
outputs: {
|
||||
triggered: { type: 'signal' },
|
||||
eventType: { type: 'string' },
|
||||
record: { type: 'object' },
|
||||
recordId: { type: 'string' }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Scenarios
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- [ ] QueryBuilder translates all operators correctly
|
||||
- [ ] SchemaManager creates valid SQLite tables
|
||||
- [ ] SchemaManager exports valid PostgreSQL
|
||||
- [ ] LocalSQLAdapter handles concurrent access
|
||||
- [ ] WebSocket broadcasts to correct subscribers
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- [ ] Full CRUD cycle via REST API
|
||||
- [ ] Workflow execution with database access
|
||||
- [ ] Realtime updates via WebSocket
|
||||
- [ ] Backend start/stop lifecycle
|
||||
- [ ] Multiple simultaneous backends
|
||||
|
||||
### End-to-End Tests
|
||||
|
||||
- [ ] New user creates backend in launcher
|
||||
- [ ] Project uses backend for data storage
|
||||
- [ ] Visual workflow saves data to database
|
||||
- [ ] Frontend receives realtime updates
|
||||
- [ ] Export schema and migrate to Supabase
|
||||
- [ ] Deploy Electron app with embedded backend
|
||||
|
||||
---
|
||||
|
||||
## Performance Targets
|
||||
|
||||
| Scenario | Target | Notes |
|
||||
|----------|--------|-------|
|
||||
| Query 1K records | < 10ms | With index |
|
||||
| Query 100K records | < 100ms | With index |
|
||||
| Insert single record | < 5ms | |
|
||||
| Batch insert 1K records | < 500ms | Within transaction |
|
||||
| WebSocket broadcast | < 10ms | To 100 clients |
|
||||
| Workflow hot reload | < 1s | Including compile |
|
||||
| Backend startup | < 2s | Cold start |
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Local only**: Backend only binds to localhost by default
|
||||
2. **No auth required**: Local development doesn't need authentication
|
||||
3. **Master key in memory**: Don't persist sensitive keys to disk
|
||||
4. **SQL injection**: Use parameterized queries exclusively
|
||||
5. **Path traversal**: Validate all file paths
|
||||
6. **Data export**: Warn about exposing sensitive data
|
||||
|
||||
---
|
||||
|
||||
## Session Notes
|
||||
|
||||
_Use this space for notes during implementation sessions_
|
||||
|
||||
### Session 1 Notes
|
||||
|
||||
```
|
||||
Date: TBD
|
||||
Focus: Phase A.1 - SQLite Integration
|
||||
|
||||
Notes:
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
Issues encountered:
|
||||
-
|
||||
-
|
||||
|
||||
Next session:
|
||||
-
|
||||
```
|
||||
|
||||
### Session 2 Notes
|
||||
|
||||
```
|
||||
Date: TBD
|
||||
Focus: Phase A.2 - Query Translation
|
||||
|
||||
Notes:
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
Issues encountered:
|
||||
-
|
||||
-
|
||||
|
||||
Next session:
|
||||
-
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Questions & Decisions to Make
|
||||
|
||||
- [ ] Should we support full-text search in SQLite? (FTS5)
|
||||
- [ ] How to handle file uploads in local backend?
|
||||
- [ ] Should triggers persist across backend restarts?
|
||||
- [ ] What's the backup/restore strategy for local databases?
|
||||
- [ ] Should we support multiple databases per backend?
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user