---
slug: "vibe-agent-pro"
source_type: "readme"
source_url: "https://cdn.jsdelivr.net/gh/Standed/vibe-agent-pro@main/README.md"
repo: "https://github.com/Standed/vibe-agent-pro"
source_file: "README.md"
branch: "main"
---
# Video Agent Pro
**AI-Powered Video Production Tool | AI 驱动的影视创作工具**
[](https://vercel.com/new/clone?repository-url=https://github.com/Standed/vibe-agent-pro)
[Features (中文)](#features) | [Quick Start](#quick-start) | [Deployment](#deployment) | [Usage](#usage) | [Tech Stack](#tech-stack)
---
## 🎬 Introduction
Video Agent Pro is an AI-powered video storyboard generation and editing tool built with Next.js 15 and multiple AI models (Gemini + Volcano Engine + Sora + Vidu). It provides both conversational AI (Agent Mode) and fine-grained control (Pro Mode) to help creators produce videos from script to final cut.
> ⚠️ **Note**: This project requires user authentication. All data is stored in the cloud (Supabase + Cloudflare R2).
---
## ✨ Core Features
### 🎭 AI Storyboard Generation
- Input script and AI automatically breaks down scenes and shots
- Based on professional 8-principle storyboard rules
- Extracts shot size, camera movement, and descriptions
### 👥 Character Turnaround Generation
- AI-generated character design sheets
- Layout: 1/3 face closeup + 2/3 front/side/back views
- Pure white background, official art style
- Powered by Volcano Engine SeeDream 4.0
### 🖼️ Grid Multi-View Generation
- Generate 2×2 (4 views) or 3×3 (9 views) storyboard grids
- Multiple aspect ratios: 16:9, 4:3, 21:9, 1:1, etc.
- Style presets: Cinematic, Anime, Realistic, Cyberpunk
- Reference image support for consistency
### 🎬 Grid Slice Preview & Manual Assignment
- **GridPreviewModal**: Visual preview of full grid and individual slices
- Click to assign slices to specific shots
- Smart auto-suggestion: first N slices → first N shots
- Confirmation before updating shot data
### 🎥 Video Generation
- Image-to-Video generation (4-6 seconds)
- Powered by Volcano Engine SeeDance 1.0 Pro
- Async task processing with progress tracking
### 🤖 Dual Work Modes
- **Agent Mode**: Conversational AI control with natural language
- **Pro Mode**: Manual parameter adjustment for fine control
- Seamless mode switching
### 📍 Infinite Canvas
- Drag-and-drop scene and shot management
- Zoom (50%-200%) and pan controls (Horizontal & Vertical)
- Smart Auto-Arrange based on aspect ratio
- Unified Sidebar with Floating Collapse Button
- Visual status indicators (draft/generating/done/failed)
### ✂️ Timeline Editor
- Collapsible Sidebar with Pro History Sync
- Video and audio tracks
- Time ruler with 5-second intervals
- Playhead indicator
- Preview and export buttons (UI ready)
### 🎨 UI/UX Enhancements (2026-02 Latest)
- **Tooltip System**: Jimeng model tooltips use React Portal rendering to prevent clipping
- **Dropdown Z-Index**: Style selector dropdown now on top layer (z-9999) with opaque background
- **Avatar Upload**: Stream lined avatar upload flow with step-by-step logging
- **Status Indicators**: Simplified green dot indicator for completed shots (minimalist design)
- **Login Flow**: Optimized authentication flow to prevent database update blocking
- **Grid Result Visual Unification**: Shot-level Gemini Grid cards now reuse Pro-style output (hide `Grid 2x2/3x3` corner badge and `选择切片` button) for cleaner Agent/Pro consistency.
### 🆕 Sora Video Generation
- **Sora Orchestrator** - Automated video generation pipeline
- **Character Registration** - @username-based character consistency
- **Dynamic Aspect Ratio** - Auto-detect image ratio for optimal output
- **Smart Scene Splitting** - >15s scenes auto-split into chunks
- **Quality Control** - Mandated prompts for high-quality output
- **R2 Persistence** - Automatic upload to Cloudflare R2
- **Sora 2 Pro** - 15s / 25s durations with 1024x1792 / 1792x1024
- **Batch Polling** - Single API call to check status for 60+ tasks simultaneously
### 🆕 Vidu Video Generation
- **Image-to-Video** - Convert storyboard images to video
- **Start-End Frames** - Use start/end frames for transition videos
- **Reference-based Generation** - Use reference images for style consistency
- **Flexible Duration** - 1-10 seconds video generation
- **Resolution Options** - 720p and 1080p support
- **Pro Mode Integration** - Mode-specific reference management (img2video / start-end / reference2video)
- **Agent Tool** - Shot-level Vidu generation via tool calling
- **R2 Persistence** - Vidu completion auto-transfer to Cloudflare R2
### 🎵 Audio Asset Management
- Upload audio files (all formats)
### 🎨 Cinematic UI/UX & Premium Glass (2026-02 Latest)
- **High-Contrast Aesthetics**: Minimalist Black & White (Zinc) theme to emphasize user content.
- **Premium Glassmorphism**: Discarded heavy, dirty drop-shadows in favor of clean `.premium-panel` with intensive `backdrop-blur` and ultra-thin inset borders (`border-white/10`).
- **Refined Interactions**: Unified, sleek hover effects replacing large glowing halos with crisp 1px border highlights, ensuring a consistent premium feel.
- **Responsive Workspace**: Optimised layout structures, preventing overlaps with dynamic padding (e.g., bottom toolbars vs User Widget), and maintaining a minimalist approach for the collapsible Right Control Panel with no superfluous borders.
- Category classification: Music / Voice / Sound Effects
- Auto-convert to Data URL for storage
- Display and delete functionality
### 🆕 User Authentication & Credits System
- **Supabase Auth Integration** - Secure user authentication
- **Three-tier Role System** - admin (free) / vip (80% off) / user (standard price)
- **Credits Management** - All AI operations consume credits
- **Agent Credit Estimation & Confirmation** - Advanced conversational interceptor predicting Server-Side execution costs, coupled with a robust double-confirmation UI and Text-Fallback defense against accidental duplicate charges.
- **Auto Profile Creation** - Profile auto-created on first login
- **Session Persistence** - Cookie-based session with auto-refresh (client + middleware)
- **Server-side Refresh** - Middleware refreshes expired access tokens and injects `Authorization` for API routes
### 🆕 Request Cancellation (AbortController)
- **Cancel AI Requests** - Stop ongoing AI operations anytime
- **Agent Mode Support** - Cancel long-running conversations
- **Clean Resource Cleanup** - Proper cleanup of network requests
### 🆕 Cloud Storage & Sync
- **Supabase Database** - PostgreSQL cloud storage for all data
- **Cloudflare R2** - Media file storage (images, videos, audio)
- **R2 Auto Persistence** - Any local `blob:` or `data:` URL from chat history, grids or drag-and-drop is seamlessly converted to R2 links before being persisted to the project sequence.
- **Project-based R2 Paths** - Assets stored under `projects/{projectId}/...` for traceability (legacy paths remain valid)
- **Chat History Sync** - Three-level scope (project/scene/shot)
- **Auto-sync** - Automatic data synchronization across devices
- **Asset Logs** - Failed transfers (Sora/Vidu) are recorded for admin tracking
### 🛠 Admin Repair Tools
- **Fix direct_generation history** - `/api/admin/fix-direct-generation-history`
- Supports `provider` filter (e.g., `vidu`)
- Re-transfers to R2 and backfills chat/history
- **Server-side R2 Persistence** - Gemini/SeeDream/Jimeng results are persisted to R2 before returning URLs
> ⚠️ Guest mode is not supported. Login is required to use all features.
---
## 📦 Quick Start
### 1. Install Dependencies
```bash
cd finalAgent/video-agent-pro
npm install
```
### 2. Configure API Keys
Create `.env.local` file:
```env
# Gemini API (for Grid generation)
NEXT_PUBLIC_GEMINI_API_KEY=your_gemini_api_key
# Volcano Engine API
NEXT_PUBLIC_VOLCANO_API_KEY=your_volcano_api_key
NEXT_PUBLIC_VOLCANO_BASE_URL=https://ark.cn-beijing.volces.com/api/v3
# Model Endpoints (create in Volcano Engine Console)
NEXT_PUBLIC_SEEDREAM_MODEL_ID=ep-xxxxxx-xxxxx # Image generation
NEXT_PUBLIC_SEEDANCE_MODEL_ID=ep-xxxxxx-xxxxx # Video generation
NEXT_PUBLIC_DOUBAO_MODEL_ID=ep-xxxxxx-xxxxx # AI conversation
# Supabase (for cloud storage and authentication)
NEXT_PUBLIC_SUPABASE_URL=your_supabase_url
NEXT_PUBLIC_SUPABASE_ANON_KEY=your_supabase_anon_key
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key # Server-side only
# Cloudflare R2 (required for production media persistence)
R2_ENDPOINT=https://your-account.r2.cloudflarestorage.com
R2_ACCESS_KEY_ID=your_access_key
R2_SECRET_ACCESS_KEY=your_secret_key
R2_BUCKET_NAME=your_bucket_name
NEXT_PUBLIC_R2_PUBLIC_URL=https://your-public-domain.r2.dev
NEXT_PUBLIC_R2_ENDPOINT=https://your-account.r2.cloudflarestorage.com
```
**Get API Keys:**
- **Gemini**: [Google AI Studio](https://makersuite.google.com/app/apikey)
- **Volcano Engine**: [Volcano Engine Console](https://console.volcengine.com/ark)
- **Supabase**: [Supabase Dashboard](https://app.supabase.com/) - Create a new project
### 3. Start Development Server
```bash
npm run dev
```
Visit [http://localhost:3000](http://localhost:3000)
---
## 🎯 Usage Guide
### Create a Project
1. Click "Create New Project" on homepage
2. Enter project name and description
3. Enter project editing page
### AI Storyboard Generation
1. Click "Script" tab in left sidebar
2. Input or paste script content
3. Click "AI Generate Storyboard"
4. AI automatically analyzes and generates scenes and shots
### Character Turnaround Generation
1. Click "Characters" tab in left sidebar
2. Click "+ Add", fill in character information
3. Enter character name, appearance, art style
4. Click "AI Generate Character Turnaround"
5. Generated image auto-added to reference library
### Grid Multi-View Generation
**Method 1: Pro Mode (Manual)**
1. Select a shot on canvas
2. Switch to "Pro" mode on right panel
3. Select "Grid Multi-View"
4. Set Grid size (2x2 or 3x3)
### 🔧 Admin Maintenance (Asset Recovery)
用于修复临时链接失效、Base64 残留、外链丢失等问题：
- **资产扫描/自愈**：`/api/admin/scan-assets`
- 扫描 shots/scenes/characters/projects/chat_messages
- 外链自动转存 R2
- 输出 `lost_items` 供批量重生成
- 支持 `mode=cron`（未设置 `CRON_SECRET` 时可被 Vercel Cron 调用）
- **一键重生成**：`/api/admin/regenerate-assets`
- 支持传入 `lost_items` 批量重生成（当前以分镜为主）
- 可指定 `mode`（gemini/seedream/jimeng/grid）
> Vercel Cron 可配置为每日 UTC 0 点调用：`/api/admin/scan-assets?mode=cron`
5. Set aspect ratio and style preset
6. Enter prompt, click "Generate Grid"
7. Manually assign slices to shots in preview modal
8. Click "Confirm Assignment"
**Method 2: Agent Mode (AI Conversation)**
1. Select shot, switch to "Agent" mode
2. Type: "Generate a grid for this shot"
3. AI automatically executes generation
### Video Generation
**Prerequisite: Shot must have Grid image**
1. Select shot with Grid image
2. Switch to "Pro" mode, select "Video Generation"
3. Enter video camera movement prompt
4. Click "Generate Video", wait 2-3 minutes
---
## 🛠️ Tech Stack
- **Framework**: Next.js 15.1 with App Router + Turbopack
- **Frontend**: React 19, TypeScript 5.8
- **Styling**: Tailwind CSS 3.4 (Cinema Dark theme)
- **State Management**: Zustand + Immer middleware
- **Database**: Supabase (PostgreSQL) - Cloud only, no local fallback
- **Authentication**: Supabase Auth (Email/Password + OAuth)
- **File Storage**: Cloudflare R2 (images, videos, audio)
- **AI Models**:
- Google Gemini 3.1 Pro & 3.1 Flash Image (Agent reasoning, Image generation)
- Volcano Engine SeeDream 4.0 (Image generation)
- Volcano Engine SeeDance 1.0 Pro (Video generation)
- **Sora 2** via Kaponai API (Professional video with character consistency)
- **Vidu** via Kaponai API (Fast image-to-video generation)
- **Jimeng** (Chinese-optimized image generation)
---
## 项目结构
```
src/
├── app/ # Next.js App Router
│ ├── api/ # API Routes (41 endpoints)
│ ├── admin/ # Admin dashboard
│ ├── auth/ # Authentication pages
│ └── project/[id]/ # Project editing page
├── components/ # React components (13 directories)
│ ├── agent/ # Agent components
│ ├── asset/ # Asset management (characters, locations)
│ ├── auth/ # Authentication (AuthProvider)
│ ├── canvas/ # Infinite canvas
│ ├── chat/ # Chat interface + Pro mode (17 files)
│ ├── director/ # Director/Planning mode
│ ├── grid/ # Grid generation UI
│ ├── jimeng/ # Jimeng integration
│ ├── layout/ # Layout (sidebars, panels, settings)
│ ├── project/ # Project dialogs
│ ├── providers/ # React Context providers
│ ├── shot/ # Shot components
│ └── ui/ # Shared UI components (StatusComponents)
├── hooks/ # Custom hooks (22 files)
│ ├── agent/ # Agent-related hooks
│ ├── chat/ # Chat panel hooks (12 files)
│ │ ├── useChatGeneration.ts # Core generation logic
│ │ ├── useChatHistory.ts # Message history & pagination
│ │ ├── useChatScroll.ts # Scroll behavior
│ │ ├── useChatActions.ts # Restore/reuse callbacks
│ │ ├── useChatModals.ts # Modal state
│ │ ├── useChatReferenceInteractions.ts # Drag/upload interactions
│ │ ├── useApplyVideoToShot.ts # Video-to-shot logic
│ │ ├── useAutoReference.ts # @mention detection
│ │ ├── useVideoModeReferences.ts # Video mode reference state
│ │ ├── useStartEndFrames.ts # Start-end frame management
│ │ ├── useVideoReferences.ts # Per-mode reference isolation
│ │ └── useReferenceCallbacks.ts # Reference callbacks
│ ├── generation/ # AI generation hooks (5 files)
│ │ ├── useViduGeneration.ts # Vidu video generation
│ │ ├── useJimengGeneration.ts # Jimeng image generation
│ │ ├── useAIStoryboard.ts # Script to storyboard
│ │ ├── useAssetGeneration.ts # Asset batch generation
│ │ └── useThreeViewGeneration.ts # Character turnaround
│ └── sora/ # Sora video hooks (5 files)
│ ├── useSoraConfig.ts # Sora parameters
│ ├── useSoraGeneration.ts # Video generation
│ ├── useSoraTaskManager.ts # Task polling & sync
│ ├── useSoraCharacter.ts # Character consistency
│ └── useSoraVideoMessages.ts # Video message handling
├── services/ # Business services (22+ files)
│ ├── agentService.ts # Agent core (Function Calling)
│ ├── agentToolDefinitions.ts # 28+ Agent tools
│ ├── geminiService.ts # Gemini Grid generation
│ ├── SoraOrchestrator.ts # Sora video orchestration
│ ├── KaponaiService.ts # Sora API wrapper
│ ├── ViduService.ts # Vidu video API wrapper
│ ├── ViduTaskManager.ts # Vidu task management
│ ├── jimengService.ts # Jimeng integration
│ ├── TaskQueueService.ts # Concurrency control
│ └── tools/ # Tool implementations (7 files)
├── lib/ # Core libraries (16 files)
│ ├── dataService.ts # Unified data service (97 methods)
│ ├── storageService.ts # R2 file upload
│ ├── auth-middleware.ts # Authentication middleware
│ ├── api-response.ts # Standardized API response
│ ├── video-transfer.ts # Video transfer utility
│ ├── assetLogService.ts # Asset logging service
│ ├── cloudflare-r2.ts # R2 client
│ └── supabase/transactions.ts # Supabase atomic operations
├── utils/ # Utility functions
│ ├── uploadQueue.ts # Concurrent upload control
│ └── fileValidation.ts # File validation
├── store/ # Zustand state management
│ └── useProjectStore.ts # Project state
├── types/ # TypeScript definitions
│ ├── project.ts # Project types
│ └── vidu.ts # Vidu API types
└── config/ # Configuration
└── credits.ts # Credits pricing config
scripts/
├── deploy/ # Deployment scripts
├── test/ # Test scripts
└── tools/ # Utility scripts
```
---
## 📋 Pending Features
### High Priority
- Grid generation history (per scene)
- Timeline playback with sync
- Drag shots to Timeline
- Video export with audio mixing
- TTS audio generation
### Medium Priority
- Scene drag & reorder on canvas
- Timeline clip adjustment (trim, reorder)
- Payment integration for credits
- OAuth login (GitHub, Google)
For detailed feature list, see [FEATURES.md](https://github.com/Standed/vibe-agent-pro/blob/HEAD/FEATURES.md)
---
## 📚 Documentation
- **Quick Reference for AI**: [AGENTS.md](https://github.com/Standed/vibe-agent-pro/blob/HEAD/AGENTS.md) - Commands and best practices
- **API Architecture**: [API_ARCHITECTURE.md](https://github.com/Standed/vibe-agent-pro/blob/HEAD/API_ARCHITECTURE.md) - API design and authentication
- **Authentication System**: [AUTHENTICATION.md](https://github.com/Standed/vibe-agent-pro/blob/HEAD/AUTHENTICATION.md) - User auth and roles
- **Credits System**: [CREDITS_SYSTEM.md](https://github.com/Standed/vibe-agent-pro/blob/HEAD/CREDITS_SYSTEM.md) - Credits pricing and management
- **Development Guide**: [CLAUDE.md](https://github.com/Standed/vibe-agent-pro/blob/HEAD/CLAUDE.md) - Detailed development philosophy
- **Chat Migration**: [CHAT_STORAGE_MIGRATION.md](https://github.com/Standed/vibe-agent-pro/blob/HEAD/CHAT_STORAGE_MIGRATION.md) - Cloud storage migration guide
---
## 🐛 Troubleshooting
### Grid Generation Failure
- Check `NEXT_PUBLIC_GEMINI_API_KEY` in `.env.local`
- Ensure network can access Google API
### Video Generation Failure
- Verify inference endpoints created in Volcano Engine Console
- Confirm endpoint_id format is correct (ep-xxxxxx-xxxxx)
- Ensure shot has Grid image
### Agent Not Responding
- Check `NEXT_PUBLIC_DOUBAO_MODEL_ID` configuration
---
## 🚀 Deployment
### Deploy to Vercel (Recommended)
Click the button below for one-click deployment:
[](https://vercel.com/new/clone?repository-url=https://github.com/Standed/vibe-agent-pro)
**Manual Deployment Steps**:
1. Visit [Vercel Import](https://vercel.com/new/import?s=https://github.com/Standed/vibe-agent-pro)
2. Connect your GitHub account
3. Configure environment variables (see `.env.example`)
4. Click "Deploy"
**Required Environment Variables**:
- Supabase: `NEXT_PUBLIC_SUPABASE_URL`, `NEXT_PUBLIC_SUPABASE_ANON_KEY`, `SUPABASE_SERVICE_ROLE_KEY`
- Auth Redirect: `NEXT_PUBLIC_APP_URL`, `NEXT_PUBLIC_AUTH_REDIRECT_ORIGIN` (生产建议都设为 `https://xysaiai.com`)
- R2 Storage: `R2_BUCKET_NAME`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_ENDPOINT`, `NEXT_PUBLIC_R2_PUBLIC_URL`
- Gemini API: `GEMINI_TEXT_API_KEY`, `GEMINI_IMAGE_API_KEY`, `GEMINI_AGENT_API_KEY`
- Volcano Engine: `NEXT_PUBLIC_VOLCANO_API_KEY`, model IDs for SeeDream/SeeDance/Doubao
**Post-Deployment**:
- Auto-deploy on every push to `main` branch
- Preview deployments for PRs
- Custom domain configuration available
- Supabase Auth URL Configuration:
- `Site URL = https://xysaiai.com`
- `Redirect URLs` includes `https://xysaiai.com/auth/reset-password` and `https://www.xysaiai.com/auth/reset-password`
- Reset Password template uses `{{ .ConfirmationURL }}`
For detailed instructions, see [DEPLOY.md](https://github.com/Standed/vibe-agent-pro/blob/HEAD/DEPLOY.md)
---
## 📝 Changelog
### v4.0.7 (2026-03-06) - 🛡️ Auth System & Documentation
- ✅ **Password Reset Origin**: Implemented robust URL resolution for the password reset redirect (`NEXT_PUBLIC_AUTH_REDIRECT_ORIGIN` override, with fallback to `window.location.origin` and `https://xysaiai.com`).
- ✅ **Supabase Auth Runbook**: Documented mandatory `Site URL / Redirect URLs / Reset Password template` configuration to avoid fallback to `localhost` on expired/invalid OTP links.
- ✅ **Internal Docs Sync**: Updated team internal usage guide (`docs/团队使用指南.md`) with new portal domains and procedural instructions.
### v4.0.6 (2026-02-27) - 🎯 Agent/Pro Grid UI Consistency & Docs Sync
- ✅ **Grid Card Consistency** - Unified Gemini Grid rendering in shot-level chat results: removed `选择切片` entry and top-right `Grid 2x2/3x3` badge to match Pro visual style.
- ✅ **Reusable Presentation Strategy** - Added `src/components/chat/generationResultPresets.ts` to centralize display behavior instead of hardcoding UI conditions in multiple components.
- ✅ **Agent-Pro Sync Metadata** - Agent-generated Pro chat sync messages now mark `metadata.source = 'agent_sync'` to support stable rendering rules.
- ✅ **Documentation Alignment** - Updated `AGENTS.md`, `README.md`, `CLAUDE.md`, and `docs/PRO_MODE_ARCHITECTURE.md` to reflect the latest implementation.
### v4.0.5 (2026-02-06) - ⚡ Performance & Documentation
- ✅ **Core Optimization** - Decoupled `ChatBubble` from `useAuth` context, eliminating unnecessary re-renders of hundreds of chat bubbles during score updates (P0 Performance Fix).
- ✅ **Stability Fix** - Fixed regression in `PlanningChat` where user avatars were missing; restored prop passing for consistent display.
- ✅ **Documentation** - Added deep architectural reviews (Risk Analysis, Gemini Performance) and updated AGENTS.md guide.
### v4.0.4 (2026-02-06) - 🎨 UI Polish & Core Stability (Codex Integration)
#### ✨ UI Polish & Experience (Antigravity)
- ✅ **Unified User Avatar** - Standardized user avatar display across Pro Mode (`ChatBubble`) and Agent Mode, using the same `Avatar` component for consistency.
- ✅ **Theme Adaptability** - Overhauled Login page to fully support both Light and Dark modes, ensuring optimal readability and aesthetic appeal in any environment.
- ✅ **Visual Refinement** - Reduced border thickness on image hover states (`GenerationResult`, `ChatBubble`) for a sleeker, less intrusive look.
- ✅ **Performance Tuning** - Optimized `loadData` with `useCallback` to prevent unnecessary re-fetches.
#### 🛡️ Core Stability & Auth (Codex)
- ✅ **Hybrid Auth Strategy** - Implemented a resilient "Proxy-First" auth check in `getUserProfile` bypasses client hydration delays, ensuring user data loads reliably even after browser refresh.
- ✅ **Race Condition Protection** - Introduced `activeUserIdRef` lock in `AuthProvider` to prevent profile state pollution during fast account switching or logout.
- ✅ **Dropdown Interaction Fix** - Reverted the complex Portal mask solution and switched to a lightweight global click listener, permanently fixing the issue where dropdown menus were unclickable.
- ✅ **Session Recovery** - Added a smart fallback mechanism that recovers session from Supabase persistence with an 800ms timeout if cookies are missing.
### v4.0.2 (2026-02-05) - 🛡️ Login & UI UX Improvements
- ✅ **Login Flow Overhaul** - Replaced aggressive auto-logout with a user-friendly "Pending Approval" UI. Users waiting for whitelist approval can now refresh their status without re-logging.
- ✅ **Robust Profile Fetching** - Fixed `getUserProfile` to explicitly include `Authorization` headers in proxy requests, resolving "User not found" issues caused by missing auth context.
- ✅ **IME Input Fix** - Solved an issue where pressing `Enter` during IME composition (e.g., Chinese Pinyin) would accidentally submit forms in Login/Register pages.
- ✅ **Image Upload Fix** - Fixed the Chat Input image upload button which was unresponsive due to a missing event handler.
- ✅ **New Skill Integration** - Integrated Anthropic's official `frontend-design` skill for higher quality UI generation.
### v4.0.1 (2026-02-05) - 🛡️ Critical Risk Fixes
- ✅ **Atomic Transactions** - Migrated batch operations to Supabase RPC (`delete_scene_atomic`), eliminating data inconsistency risks during network failures.
- ✅ **Async Cron Architecture** - Decoupled heavy video uploads from status checks. `check-sora-status` is now lightweight (status only), while `retry-transfers` handles R2 uploads asynchronously, solving Vercel timeout issues.
- ✅ **Image Upload Optimization** - Added "Fast Path" for uploads < 3MB (skips compression) and implemented progress feedback.
- ✅ **Reference Management Fix** - Fixed `useAutoReference` to strictly sync UI with deletion actions using `useRef` + versioning.
- ✅ **SSR Hydration Fix** - Resolved Next.js hydration mismatches in `useProjectStore` by deferring localStorage reads to client-side effects.
### v3.9.6 (2026-02-04)
- ✅ **Gemini Reference Image Optimization** - Implemented "URL-first + Auto-fallback" strategy for reference images.
- First attempt: Use R2 presigned URLs (`fileData.fileUri`) for instant transfer.
- Auto-retry: If Gemini returns "Cannot fetch" error, automatically switch to server-side download mode (`inlineData`).
- Benefits: Maximizes speed while ensuring reliability through transparent fallback.
- ✅ **R2 Presigned URL Support** - Added `generatePresignedUrl()` function in `r2-server-upload.ts` for generating signed URLs.
### v3.9.5 (2026-02-02)
- ✅ **Pro Mode Scroll Optimization** - Created `useChatScroll` hook to unify chat scrolling logic, fixing initial load scroll issues and media load compensation.
- ✅ **Code Refactoring** - Reduced 120+ lines of scattered scroll logic in `ChatPanel.tsx` to a clean ~20 lines using the new hook.
- ✅ **Scroll Position Preservation** - Properly maintains scroll position when loading more messages (pagination).
### v3.9.4 (2026-02-01)
- ✅ **Pro Mode History Loading** - Pro mode chat now loads last 30 messages by default, with scroll-up pagination for older history.
- ✅ **Sora/Vidu R2 Persistence** - Unified video transfer pipeline to Cloudflare R2 for both Sora and Vidu providers.
- ✅ **Sora 2 Pro Support** - Added 15s/25s duration options with 1024x1792 / 1792x1024 resolution.
### v3.9.3 (2026-02-01)
- ✅ **Reference Logic Decoupling** - Completely refactored `ChatPanel` to use isolated state hooks (`useVideoReferences`) for Vidu/Sora modes, preventing state pollution.
- ✅ **Interaction Encapsulation** - Moved drag-and-drop and file upload logic to `useChatReferenceInteractions`, significantly improving code maintainability.
- ✅ **Start-End Frame Optimization** - "Ghost" projection removed; start frame is now a real state initialized from shot reference, allowing clear deletion.
### v3.9.2 (2026-01-31)
- ✅ **Vidu Video Integration** - Added Vidu video generation support with img2video and reference2video modes.
- ✅ **Unified Video Task Table** - Extended `sora_tasks` table with `provider` field to support multiple video providers (Sora, Vidu, Jimeng, etc.).
- ✅ **Agent Tool Support** - Added `generateViduVideo` tool for AI-driven video generation.
- ✅ **Task Management** - Implemented `ViduTaskManager` for task creation, status tracking, and R2 persistence.
- ✅ **Ownership Validation** - All video task APIs now validate user ownership before operations.
- ✅ **Credits Integration** - Vidu video generation uses unified credits system (720p: 2 credits/sec, 1080p: 4 credits/sec).
### v3.9.1 (2026-01-30)
- ✅ **Jimeng Pro Mode Upgrade** - Consolidated multi-image generations into a single "Grid-like" message (2x2), removing the need for manual refresh.
- ✅ **Critical Login Fix** - Resolved infinite login loop on `/auth/login` by strictly enforcing server-side session validation.
- ✅ **User UX** - Added download button for user-uploaded reference images in Pro mode chat history.
- ✅ **Grid Layout Optimization** - Fixed layout issues for 3x3 Grids, ensuring full visibility and adaptive width for all grid sizes (Single/2x2/3x3).
- ✅ **Image Preview Enhancements** - Image preview overlay now supports keyboard navigation (Arrow Left/Right) with auto-focus support.
- ✅ **Download Positioning** - Relocated the download button in Grid images to the right-side action group for better usability.
### v3.9.0 (2026-01-30)
- ✅ **Grid Instant Apply** - Applying a Grid image or slice to a shot (via drag, click, or new canvas button) now **immediately** marks the shot as 'Done' and persists to the database, ensuring "what you see is what you get".
- ✅ **Proxy Download System** - Introduced `/api/proxy-download` to handle cross-origin R2 file downloads reliably, forcing file download instead of browser preview.
- ✅ **Canvas Interaction** - Fixed missing Download/Edit actions in Grid state.
- ✅ **Reference Image Fix** - Resolved issue where reusing history reference images failed in production by disabling fetch caching and adding proper User-Agent headers.
- ✅ **Frontend & UI Experience Optimizations**:
- **Compact Grid Layout**: Optimized 3x3 and 2x2 grid display for better screen utilization.
- **Smart Interaction**: Separated click logic for Slice Selection (Grid) vs Image Preview (Single/Jimeng).
- **Aspect Ratio Fidelity**: strict metadata persistence for vertical/mobile aspect ratios across all models.
- ✅ **Backend & AI Integration**:
- **Multi-Model Support**: Gemini, Jimeng, SeeDream integrated.
- **Robust Asset Pipeline**: R2 storage, proxy downloads, and automatic history tracking.
### v3.8.9 (2026-01-29)
- ✅ **Canvas Flicker Fix** - Completely resolved canvas flickering during zoom/pan by refactoring grid rendering to CSS background and implementing memoized scene components.
- ✅ **Planning Chat Expansion** - Added a fullscreen edit mode for the Planning Chat input, improving the experience for writing long prompts.
- ✅ **Performance Optimization** - Significantly reduced re-renders in the infinite canvas through component memoization and optimized state management.
### v3.8.8 (2026-01-28)
- ✅ **AI Command Restoration** - Restored AI Quick Commands (Grid/Sora/Jimeng) in the Agent Panel for immediate access in image/video generation workflows.
- ✅ **New Script Tab Integration** - Enabled "Script" tab in the unified sidebar, consolidating story brainstorming and AI script analysis into a dedicated view.
- ✅ **AI Quick Commands Relocation** - Added "AI Quick Commands" to the new Script Tab while retaining them in Agent Panel, allowing for flexible workflows.
- ✅ **Canvas Rendering Optimization** - Fixed canvas flickering during zoom/pan by removing conflicting CSS transitions and enabling hardware acceleration.
- ✅ **Agent Panel Cleanup** - Enhanced Agent interaction flow while preserving key shortcuts.
- ✅ **Standardized Image Interaction** - Added standard "Use as Reference" and "One-click Apply" hover actions to user-uploaded images in chat, matching the Grid interaction experience.
### v3.0.0 (2026-01-19)
- ✅ **Pure Cloud Architecture** - Removed guest mode, all data stored in cloud
- ✅ **28 Agent Tools** - Complete CRUD + generation + batch operations
- ✅ **Jimeng Integration** - Chinese-optimized image generation
- ✅ **Pro Mode WYSIWYG References** - "What You See Is What You Get" reference image logic
- ✅ **Latest-Video API Optimization** - Pure read-only, no more R2 upload delays
- ✅ **Auth Middleware Refresh Token** - Auto-refresh expired access tokens
- ✅ **SmartRecovery Fix** - Prevents infinite polling loop on page refresh
### v3.8.7 (2026-01-27)
- ✅ **Prompt Logic Unified** - Centralized prompt construction logic (`constructBaseShotPrompt`) for Agent, Pro Mode, and Canvas generation.
- ✅ **Punctuation Optimization** - Intelligent prompt concatenation that removes redundant punctuation and uses periods to separate scene descriptions for better clarity.
- ✅ **Canvas Generation Repair** - Fixed the "Regenerate" button on canvas to use the latest prompt logic, ensuring consistency with Pro Mode.
- ✅ **Canvas Layout Optimization** - Shot cards on canvas now dynamically adapt to the project's aspect ratio (e.g., 9:16 vertical cards).
- ✅ **Adaptive Grid Density** - Grid columns automatically adjust based on card width (6 columns for vertical, 4 for wide) for better screen utilization.
- ✅ **Responsive Card UI** - Redesigned bottom toolbar and info layout specifically for narrow vertical cards (`Tall` mode) to prevent overcrowding.
- ✅ **Pro Mode UX** - Auto-fills complete prompt in Pro Mode when viewing shots with no user history (only AI-generated content).
### v3.8.6 (2026-01-27)
- ✅ **Reference Drag & Drop** - Implemented draggable sorting for reference images in Pro Mode. New references now append to the end, preserving user-defined order.
- ✅ **Prompt Enhancement** - Injected `scene.description` into generation prompts for better context awareness.
- ✅ **Preview Fallback** - Image previews now fallback to project aspect ratio when metadata is missing, ensuring consistent UI visuals.
- ✅ **Code Refactoring** - Consolidated shot translation logic and improved state management for reference ordering.
### v3.8.5 (2026-01-27)
- ✅ **API Timeout Optimization** - Unified all API routes to 120s timeout (SeeDream/Jimeng/Gemini-Image), fixing production `FUNCTION_INVOCATION_TIMEOUT` errors.
- ✅ **Agent Concurrency Fix** - Separated Jimeng and SeeDream call logic; implemented Jimeng client-side polling (`check-status-once`) to avoid 60s server blocking.
- ✅ **Supabase Connection Resilience** - Auth middleware now uses exponential backoff retry (up to 3 times); client session refresh adds retry (up to 2 times).
- ✅ **Jimeng API Enhancement** - Added `checkTaskOnce()`, `check-status-once` action, and `pollTaskClient()` for non-blocking status checks.
### v3.8.4 (2026-01-27)
- ✅ **Concurrency Optimization** - Implemented parallel R2 uploads (Promise.all) for Grid generation, reducing upload time by ~4x.
- ✅ **Non-blocking UI** - Decoupled upload status from generation status, allowing immediate interaction after AI generation completes while uploads continue in background.
- ✅ **Refined UX** - Removed persistent "Generating..." spinner during upload phase to improve perceived performance.
### v3.8.3 (2026-01-27)
- ✅ **R2 Presigned Uploads** - Implemented direct-to-R2 upload using Presigned URLs to bypass Vercel 4.5MB payload limit, fixing 413 errors for large Grid images.
- ✅ **Chat Message Deletion** - Added ability to delete chat messages in Pro Mode (Soft delete, preserving referenced R2 files).
- ✅ **UI Refinement** - Enhanced Chat Bubble UI with hover-visible actions and refined iconography.
- ✅ **Linting Fixes** - Fixed React Hook dependencies in Chat History management.
### v3.8.2 (2026-01-27)
- ✅ **Server-side Image Compression** - All API routes (gemini-grid, gemini-image, jimeng) now use sharp to compress reference images to max 2048px JPEG
- ✅ **Payload Size Fix** - Prevents "请求载荷过大 (5MB+)" errors when using high-resolution images as references
- ✅ **R2 Upload Resilience** - Added exponential backoff retry (up to 5 times) for R2 uploads
- ✅ **Data Safety** - Added page unload warning during generation/upload to prevent data loss
- ✅ **Scene Grid Persistence** - Fixed bug where Grid generated in Scene mode wasn't saved to history
- ✅ **View State Sync** - URL now syncs with current view (Canvas/Planning/Timeline), fixing persistent view issues
- ✅ **Prompt Engineering** - Removed redundant prefixes (e.g., "镜头画面:", "景别:") for cleaner, more direct prompts
- ✅ **Favicon Support** - Added favicon.png and apple-touch-icon.png for browser tab icons
- ✅ **Project Structure Reorganization** - Merged components (input/pro → chat, navigation/settings → layout), organized hooks into functional groups
### v3.8.1 (2026-01-27)
- ✅ **Pro Mode Refactor** - New `useChatGeneration` hook for message/upload logic
- ✅ **Image Compression** - Auto-compress uploads to 2048px JPEG before R2 upload
- ✅ **Backend Optimization** - API routes fetch images from URL, 20MB limit, 15s timeout
- ✅ **Drag Fix** - Unified handling for shot drag and file drag
### v3.8.0 (2026-01-27)
- ✅ **Bidirectional Drag-Drop** - Drag generated images to Storyboard shots (with history sync) & Drag shots to Pro chat as references
- ✅ **ChatPanel Decoupling** - Refactored core logic into `useChatHistory` and `useAutoReference` hooks
- ✅ **Image Preview Drag** - Direct drag support from full-screen image preview
- ✅ **History Synchronization** - Optimistic updates + Backend persistence for all drag-drop actions
### v3.6.0 (2026-01-26)
- ✅ **Pro Mode Grid Simplified** - Simple prompt template for user creativity
- ✅ **Reference Image Optimization** - Max 10 images, 10MB each, 4MB budget after compression
- ✅ **Timeout Unified** - All API routes use 120s timeout
- ✅ **Code Cleanup** - Removed unused `useProGeneration.ts` hook
### v3.1.0 (2026-01-20)
- ✅ **Sora Registration Optimization** - Async registration (<1s response) + Smart Task Recovery
- ✅ **Smart Asset Generation** - Auto-detects missing assets for imported storyboards
- ✅ **Inspiration Auto-Trigger** - Auto-starts AI storyboard from homepage inspiration
- ✅ **Conflict Resolution** - Mutual exclusion between Auto-Gen and Asset-Gen flows
- ✅ **UI Refinement** - Unified color theme (Zinc/Neutral) for progress indicators
### v4.0.0 (2026-02-02)
- ✅ **ChatPanel Hook 架构重构** - 提取 6 个独立 Hook 减少组件复杂度
- `useSoraConfig` - Sora 参数管理
- `useApplyVideoToShot` - 视频应用逻辑
- `useChatActions` - 消息操作回调
- `useChatModals` - 模态框状态
- `useVideoModeReferences` 增强
- ✅ **统一 API 响应** - `api-response.ts` 标准化错误格式
- ✅ **任务队列服务** - `TaskQueueService.ts` 并发控制
- ✅ **Cron 重试机制** - 视频转存失败自动重试
- ✅ **Supabase 事务工具** - 批量操作原子性保证
- ✅ **UI 组件库** - `StatusComponents.tsx` 统一加载/空状态
### v0.6.0 (2025-12-24)
- ✅ **Sora Video Generation** - Full Sora 2 integration via Kaponai API
- ✅ **SoraOrchestrator** - Automated pipeline for character registration and video generation
- ✅ **Character Consistency** - @username-based character tracking across scenes
- ✅ **Dynamic Aspect Ratio** - Auto-detect image ratio for optimal video output
- ✅ **Smart Scene Splitting** - >15s scenes auto-split into chunks (Greedy Packing)
- ✅ **R2 Persistence** - Automatic upload to Cloudflare R2 for video storage
### v0.4.0 (2025-12-17)
- ✅ **User Authentication System** - Supabase Auth integration
- ✅ **Credits System** - Three-tier pricing (admin free, vip 80% off, user standard)
- ✅ **Request Cancellation** - AbortController support for AI requests
- ✅ **Cloud Storage** - Supabase PostgreSQL for projects and chat history
- ✅ **Chat History Sync** - Three-level scope (project/scene/shot) cloud storage
### v0.2.0 (2025-01-03)
- ✅ Character AI turnaround generation (1/3 face + 2/3 views)
- ✅ GridPreviewModal component for slice preview & manual assignment
- ✅ Pro mode Grid generation integrated with preview modal
- ✅ Audio upload functionality (music/voice/sfx)
### v0.1.0 (2025-01-03)
- ✅ Canvas zoom and pan
- ✅ Gemini API integration for Grid generation
- ✅ AI Agent conversation system (streaming output)
- ✅ AI storyboard generation (8-principle rules)
- ✅ Timeline editor
---
## 📄 License
MIT License
---
## 👨💻 Authors
Developed by **西羊石 Team**, assisted by Claude Code + Gemini Code.
---
**Star ⭐ this repo if you find it helpful!**