vibe-agent-pro

内容来源：README.md（说明文档） · 原始地址 · 查看安装指南

原始内容

Video Agent Pro

Version Next.js React TypeScript

AI-Powered Video Production Tool | AI 驱动的影视创作工具

Deploy with Vercel

Features (中文) | Quick Start | Deployment | Usage | Tech Stack

🎬 Introduction

Video Agent Pro is an AI-powered video storyboard generation and editing tool built with Next.js 15 and multiple AI models (Gemini + Volcano Engine + Sora + Vidu). It provides both conversational AI (Agent Mode) and fine-grained control (Pro Mode) to help creators produce videos from script to final cut.

⚠️ Note: This project requires user authentication. All data is stored in the cloud (Supabase + Cloudflare R2).

✨ Core Features

🎭 AI Storyboard Generation

  • Input script and AI automatically breaks down scenes and shots
  • Based on professional 8-principle storyboard rules
  • Extracts shot size, camera movement, and descriptions

👥 Character Turnaround Generation

  • AI-generated character design sheets
  • Layout: 1/3 face closeup + 2/3 front/side/back views
  • Pure white background, official art style
  • Powered by Volcano Engine SeeDream 4.0

🖼️ Grid Multi-View Generation

  • Generate 2×2 (4 views) or 3×3 (9 views) storyboard grids
  • Multiple aspect ratios: 16:9, 4:3, 21:9, 1:1, etc.
  • Style presets: Cinematic, Anime, Realistic, Cyberpunk
  • Reference image support for consistency

🎬 Grid Slice Preview & Manual Assignment

  • GridPreviewModal: Visual preview of full grid and individual slices
  • Click to assign slices to specific shots
  • Smart auto-suggestion: first N slices → first N shots
  • Confirmation before updating shot data

🎥 Video Generation

  • Image-to-Video generation (4-6 seconds)
  • Powered by Volcano Engine SeeDance 1.0 Pro
  • Async task processing with progress tracking

🤖 Dual Work Modes

  • Agent Mode: Conversational AI control with natural language
  • Pro Mode: Manual parameter adjustment for fine control
  • Seamless mode switching

📍 Infinite Canvas

  • Drag-and-drop scene and shot management
  • Zoom (50%-200%) and pan controls (Horizontal & Vertical)
  • Smart Auto-Arrange based on aspect ratio
  • Unified Sidebar with Floating Collapse Button
  • Visual status indicators (draft/generating/done/failed)

✂️ Timeline Editor

  • Collapsible Sidebar with Pro History Sync
  • Video and audio tracks
  • Time ruler with 5-second intervals
  • Playhead indicator
  • Preview and export buttons (UI ready)

🎨 UI/UX Enhancements (2026-02 Latest)

  • Tooltip System: Jimeng model tooltips use React Portal rendering to prevent clipping
  • Dropdown Z-Index: Style selector dropdown now on top layer (z-9999) with opaque background
  • Avatar Upload: Stream lined avatar upload flow with step-by-step logging
  • Status Indicators: Simplified green dot indicator for completed shots (minimalist design)
  • Login Flow: Optimized authentication flow to prevent database update blocking
  • Grid Result Visual Unification: Shot-level Gemini Grid cards now reuse Pro-style output (hide Grid 2x2/3x3 corner badge and 选择切片 button) for cleaner Agent/Pro consistency.

🆕 Sora Video Generation

  • Sora Orchestrator - Automated video generation pipeline
  • Character Registration - @username-based character consistency
  • Dynamic Aspect Ratio - Auto-detect image ratio for optimal output
  • Smart Scene Splitting - >15s scenes auto-split into chunks
  • Quality Control - Mandated prompts for high-quality output
  • R2 Persistence - Automatic upload to Cloudflare R2
  • Sora 2 Pro - 15s / 25s durations with 1024x1792 / 1792x1024
  • Batch Polling - Single API call to check status for 60+ tasks simultaneously

🆕 Vidu Video Generation

  • Image-to-Video - Convert storyboard images to video
  • Start-End Frames - Use start/end frames for transition videos
  • Reference-based Generation - Use reference images for style consistency
  • Flexible Duration - 1-10 seconds video generation
  • Resolution Options - 720p and 1080p support
  • Pro Mode Integration - Mode-specific reference management (img2video / start-end / reference2video)
  • Agent Tool - Shot-level Vidu generation via tool calling
  • R2 Persistence - Vidu completion auto-transfer to Cloudflare R2

🎵 Audio Asset Management

  • Upload audio files (all formats)

🎨 Cinematic UI/UX & Premium Glass (2026-02 Latest)

  • High-Contrast Aesthetics: Minimalist Black & White (Zinc) theme to emphasize user content.
  • Premium Glassmorphism: Discarded heavy, dirty drop-shadows in favor of clean .premium-panel with intensive backdrop-blur and ultra-thin inset borders (border-white/10).
  • Refined Interactions: Unified, sleek hover effects replacing large glowing halos with crisp 1px border highlights, ensuring a consistent premium feel.
  • Responsive Workspace: Optimised layout structures, preventing overlaps with dynamic padding (e.g., bottom toolbars vs User Widget), and maintaining a minimalist approach for the collapsible Right Control Panel with no superfluous borders.
  • Category classification: Music / Voice / Sound Effects
  • Auto-convert to Data URL for storage
  • Display and delete functionality

🆕 User Authentication & Credits System

  • Supabase Auth Integration - Secure user authentication
  • Three-tier Role System - admin (free) / vip (80% off) / user (standard price)
  • Credits Management - All AI operations consume credits
  • Agent Credit Estimation & Confirmation - Advanced conversational interceptor predicting Server-Side execution costs, coupled with a robust double-confirmation UI and Text-Fallback defense against accidental duplicate charges.
  • Auto Profile Creation - Profile auto-created on first login
  • Session Persistence - Cookie-based session with auto-refresh (client + middleware)
  • Server-side Refresh - Middleware refreshes expired access tokens and injects Authorization for API routes

🆕 Request Cancellation (AbortController)

  • Cancel AI Requests - Stop ongoing AI operations anytime
  • Agent Mode Support - Cancel long-running conversations
  • Clean Resource Cleanup - Proper cleanup of network requests

🆕 Cloud Storage & Sync

  • Supabase Database - PostgreSQL cloud storage for all data
  • Cloudflare R2 - Media file storage (images, videos, audio)
  • R2 Auto Persistence - Any local blob: or data: URL from chat history, grids or drag-and-drop is seamlessly converted to R2 links before being persisted to the project sequence.
  • Project-based R2 Paths - Assets stored under projects/{projectId}/... for traceability (legacy paths remain valid)
  • Chat History Sync - Three-level scope (project/scene/shot)
  • Auto-sync - Automatic data synchronization across devices
  • Asset Logs - Failed transfers (Sora/Vidu) are recorded for admin tracking

🛠 Admin Repair Tools

  • Fix direct_generation history - /api/admin/fix-direct-generation-history
    • Supports provider filter (e.g., vidu)
    • Re-transfers to R2 and backfills chat/history
  • Server-side R2 Persistence - Gemini/SeeDream/Jimeng results are persisted to R2 before returning URLs

⚠️ Guest mode is not supported. Login is required to use all features.

📦 Quick Start

1. Install Dependencies

cd finalAgent/video-agent-pro
npm install

2. Configure API Keys

Create .env.local file:

# Gemini API (for Grid generation)
NEXT_PUBLIC_GEMINI_API_KEY=your_gemini_api_key

# Volcano Engine API
NEXT_PUBLIC_VOLCANO_API_KEY=your_volcano_api_key
NEXT_PUBLIC_VOLCANO_BASE_URL=https://ark.cn-beijing.volces.com/api/v3

# Model Endpoints (create in Volcano Engine Console)
NEXT_PUBLIC_SEEDREAM_MODEL_ID=ep-xxxxxx-xxxxx  # Image generation
NEXT_PUBLIC_SEEDANCE_MODEL_ID=ep-xxxxxx-xxxxx  # Video generation
NEXT_PUBLIC_DOUBAO_MODEL_ID=ep-xxxxxx-xxxxx    # AI conversation

# Supabase (for cloud storage and authentication)
NEXT_PUBLIC_SUPABASE_URL=your_supabase_url
NEXT_PUBLIC_SUPABASE_ANON_KEY=your_supabase_anon_key
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key  # Server-side only

# Cloudflare R2 (required for production media persistence)
R2_ENDPOINT=https://your-account.r2.cloudflarestorage.com
R2_ACCESS_KEY_ID=your_access_key
R2_SECRET_ACCESS_KEY=your_secret_key
R2_BUCKET_NAME=your_bucket_name
NEXT_PUBLIC_R2_PUBLIC_URL=https://your-public-domain.r2.dev
NEXT_PUBLIC_R2_ENDPOINT=https://your-account.r2.cloudflarestorage.com

Get API Keys:

3. Start Development Server

npm run dev

Visit http://localhost:3000

🎯 Usage Guide

Create a Project

  1. Click "Create New Project" on homepage
  2. Enter project name and description
  3. Enter project editing page

AI Storyboard Generation

  1. Click "Script" tab in left sidebar
  2. Input or paste script content
  3. Click "AI Generate Storyboard"
  4. AI automatically analyzes and generates scenes and shots

Character Turnaround Generation

  1. Click "Characters" tab in left sidebar
  2. Click "+ Add", fill in character information
  3. Enter character name, appearance, art style
  4. Click "AI Generate Character Turnaround"
  5. Generated image auto-added to reference library

Grid Multi-View Generation

Method 1: Pro Mode (Manual)

  1. Select a shot on canvas
  2. Switch to "Pro" mode on right panel
  3. Select "Grid Multi-View"
  4. Set Grid size (2x2 or 3x3)

🔧 Admin Maintenance (Asset Recovery)

用于修复临时链接失效、Base64 残留、外链丢失等问题：

  • 资产扫描/自愈：/api/admin/scan-assets

    • 扫描 shots/scenes/characters/projects/chat_messages
    • 外链自动转存 R2
    • 输出 lost_items 供批量重生成
    • 支持 mode=cron（未设置 CRON_SECRET 时可被 Vercel Cron 调用）

  • 一键重生成：/api/admin/regenerate-assets

    • 支持传入 lost_items 批量重生成（当前以分镜为主）
    • 可指定 mode（gemini/seedream/jimeng/grid）

Vercel Cron 可配置为每日 UTC 0 点调用：/api/admin/scan-assets?mode=cron

  1. Set aspect ratio and style preset
  2. Enter prompt, click "Generate Grid"
  3. Manually assign slices to shots in preview modal
  4. Click "Confirm Assignment"

Method 2: Agent Mode (AI Conversation)

  1. Select shot, switch to "Agent" mode
  2. Type: "Generate a grid for this shot"
  3. AI automatically executes generation

Video Generation

Prerequisite: Shot must have Grid image

  1. Select shot with Grid image
  2. Switch to "Pro" mode, select "Video Generation"
  3. Enter video camera movement prompt
  4. Click "Generate Video", wait 2-3 minutes

🛠️ Tech Stack

  • Framework: Next.js 15.1 with App Router + Turbopack
  • Frontend: React 19, TypeScript 5.8
  • Styling: Tailwind CSS 3.4 (Cinema Dark theme)
  • State Management: Zustand + Immer middleware
  • Database: Supabase (PostgreSQL) - Cloud only, no local fallback
  • Authentication: Supabase Auth (Email/Password + OAuth)
  • File Storage: Cloudflare R2 (images, videos, audio)
  • AI Models:
    • Google Gemini 3.1 Pro & 3.1 Flash Image (Agent reasoning, Image generation)
    • Volcano Engine SeeDream 4.0 (Image generation)
    • Volcano Engine SeeDance 1.0 Pro (Video generation)
    • Sora 2 via Kaponai API (Professional video with character consistency)
    • Vidu via Kaponai API (Fast image-to-video generation)
    • Jimeng (Chinese-optimized image generation)

项目结构

src/
├── app/                              # Next.js App Router
│   ├── api/                          # API Routes (41 endpoints)
│   ├── admin/                        # Admin dashboard
│   ├── auth/                         # Authentication pages
│   └── project/[id]/                 # Project editing page
├── components/                       # React components (13 directories)
│   ├── agent/                        # Agent components
│   ├── asset/                        # Asset management (characters, locations)
│   ├── auth/                         # Authentication (AuthProvider)
│   ├── canvas/                       # Infinite canvas
│   ├── chat/                         # Chat interface + Pro mode (17 files)
│   ├── director/                     # Director/Planning mode
│   ├── grid/                         # Grid generation UI
│   ├── jimeng/                       # Jimeng integration
│   ├── layout/                       # Layout (sidebars, panels, settings)
│   ├── project/                      # Project dialogs
│   ├── providers/                    # React Context providers
│   ├── shot/                         # Shot components
│   └── ui/                           # Shared UI components (StatusComponents)
├── hooks/                            # Custom hooks (22 files)
│   ├── agent/                        # Agent-related hooks
│   ├── chat/                         # Chat panel hooks (12 files)
│   │   ├── useChatGeneration.ts      # Core generation logic
│   │   ├── useChatHistory.ts         # Message history & pagination
│   │   ├── useChatScroll.ts          # Scroll behavior
│   │   ├── useChatActions.ts         # Restore/reuse callbacks
│   │   ├── useChatModals.ts          # Modal state
│   │   ├── useChatReferenceInteractions.ts  # Drag/upload interactions
│   │   ├── useApplyVideoToShot.ts    # Video-to-shot logic
│   │   ├── useAutoReference.ts       # @mention detection
│   │   ├── useVideoModeReferences.ts # Video mode reference state
│   │   ├── useStartEndFrames.ts      # Start-end frame management
│   │   ├── useVideoReferences.ts     # Per-mode reference isolation
│   │   └── useReferenceCallbacks.ts  # Reference callbacks
│   ├── generation/                   # AI generation hooks (5 files)
│   │   ├── useViduGeneration.ts      # Vidu video generation
│   │   ├── useJimengGeneration.ts    # Jimeng image generation
│   │   ├── useAIStoryboard.ts        # Script to storyboard
│   │   ├── useAssetGeneration.ts     # Asset batch generation
│   │   └── useThreeViewGeneration.ts # Character turnaround
│   └── sora/                         # Sora video hooks (5 files)
│       ├── useSoraConfig.ts          # Sora parameters
│       ├── useSoraGeneration.ts      # Video generation
│       ├── useSoraTaskManager.ts     # Task polling & sync
│       ├── useSoraCharacter.ts       # Character consistency
│       └── useSoraVideoMessages.ts   # Video message handling
├── services/                         # Business services (22+ files)
│   ├── agentService.ts               # Agent core (Function Calling)
│   ├── agentToolDefinitions.ts       # 28+ Agent tools
│   ├── geminiService.ts              # Gemini Grid generation
│   ├── SoraOrchestrator.ts           # Sora video orchestration
│   ├── KaponaiService.ts             # Sora API wrapper
│   ├── ViduService.ts                # Vidu video API wrapper
│   ├── ViduTaskManager.ts            # Vidu task management
│   ├── jimengService.ts              # Jimeng integration
│   ├── TaskQueueService.ts           # Concurrency control
│   └── tools/                        # Tool implementations (7 files)
├── lib/                              # Core libraries (16 files)
│   ├── dataService.ts                # Unified data service (97 methods)
│   ├── storageService.ts             # R2 file upload
│   ├── auth-middleware.ts            # Authentication middleware
│   ├── api-response.ts               # Standardized API response
│   ├── video-transfer.ts             # Video transfer utility
│   ├── assetLogService.ts            # Asset logging service
│   ├── cloudflare-r2.ts              # R2 client
│   └── supabase/transactions.ts      # Supabase atomic operations
├── utils/                            # Utility functions
│   ├── uploadQueue.ts                # Concurrent upload control
│   └── fileValidation.ts             # File validation
├── store/                            # Zustand state management
│   └── useProjectStore.ts            # Project state
├── types/                            # TypeScript definitions
│   ├── project.ts                    # Project types
│   └── vidu.ts                       # Vidu API types
└── config/                           # Configuration
    └── credits.ts                    # Credits pricing config

scripts/
├── deploy/                           # Deployment scripts
├── test/                             # Test scripts
└── tools/                            # Utility scripts

📋 Pending Features

High Priority

  • Grid generation history (per scene)
  • Timeline playback with sync
  • Drag shots to Timeline
  • Video export with audio mixing
  • TTS audio generation

Medium Priority

  • Scene drag & reorder on canvas
  • Timeline clip adjustment (trim, reorder)
  • Payment integration for credits
  • OAuth login (GitHub, Google)

For detailed feature list, see FEATURES.md

📚 Documentation

🐛 Troubleshooting

Grid Generation Failure

  • Check NEXT_PUBLIC_GEMINI_API_KEY in .env.local
  • Ensure network can access Google API

Video Generation Failure

  • Verify inference endpoints created in Volcano Engine Console
  • Confirm endpoint_id format is correct (ep-xxxxxx-xxxxx)
  • Ensure shot has Grid image

Agent Not Responding

  • Check NEXT_PUBLIC_DOUBAO_MODEL_ID configuration

🚀 Deployment

Deploy to Vercel (Recommended)

Click the button below for one-click deployment:

Deploy with Vercel

Manual Deployment Steps:

  1. Visit Vercel Import
  2. Connect your GitHub account
  3. Configure environment variables (see .env.example)
  4. Click "Deploy"

Required Environment Variables:

  • Supabase: NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY
  • Auth Redirect: NEXT_PUBLIC_APP_URL, NEXT_PUBLIC_AUTH_REDIRECT_ORIGIN (生产建议都设为 https://xysaiai.com)
  • R2 Storage: R2_BUCKET_NAME, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_ENDPOINT, NEXT_PUBLIC_R2_PUBLIC_URL
  • Gemini API: GEMINI_TEXT_API_KEY, GEMINI_IMAGE_API_KEY, GEMINI_AGENT_API_KEY
  • Volcano Engine: NEXT_PUBLIC_VOLCANO_API_KEY, model IDs for SeeDream/SeeDance/Doubao

Post-Deployment:

  • Auto-deploy on every push to main branch
  • Preview deployments for PRs
  • Custom domain configuration available
  • Supabase Auth URL Configuration:
    • Site URL = https://xysaiai.com
    • Redirect URLs includes https://xysaiai.com/auth/reset-password and https://www.xysaiai.com/auth/reset-password
    • Reset Password template uses {{ .ConfirmationURL }}

For detailed instructions, see DEPLOY.md

📝 Changelog

v4.0.7 (2026-03-06) - 🛡️ Auth System & Documentation

  • ✅ Password Reset Origin: Implemented robust URL resolution for the password reset redirect (NEXT_PUBLIC_AUTH_REDIRECT_ORIGIN override, with fallback to window.location.origin and https://xysaiai.com).
  • ✅ Supabase Auth Runbook: Documented mandatory Site URL / Redirect URLs / Reset Password template configuration to avoid fallback to localhost on expired/invalid OTP links.
  • ✅ Internal Docs Sync: Updated team internal usage guide (docs/团队使用指南.md) with new portal domains and procedural instructions.

v4.0.6 (2026-02-27) - 🎯 Agent/Pro Grid UI Consistency & Docs Sync

  • ✅ Grid Card Consistency - Unified Gemini Grid rendering in shot-level chat results: removed 选择切片 entry and top-right Grid 2x2/3x3 badge to match Pro visual style.
  • ✅ Reusable Presentation Strategy - Added src/components/chat/generationResultPresets.ts to centralize display behavior instead of hardcoding UI conditions in multiple components.
  • ✅ Agent-Pro Sync Metadata - Agent-generated Pro chat sync messages now mark metadata.source = 'agent_sync' to support stable rendering rules.
  • ✅ Documentation Alignment - Updated AGENTS.md, README.md, CLAUDE.md, and docs/PRO_MODE_ARCHITECTURE.md to reflect the latest implementation.

v4.0.5 (2026-02-06) - ⚡ Performance & Documentation

  • ✅ Core Optimization - Decoupled ChatBubble from useAuth context, eliminating unnecessary re-renders of hundreds of chat bubbles during score updates (P0 Performance Fix).
  • ✅ Stability Fix - Fixed regression in PlanningChat where user avatars were missing; restored prop passing for consistent display.
  • ✅ Documentation - Added deep architectural reviews (Risk Analysis, Gemini Performance) and updated AGENTS.md guide.

v4.0.4 (2026-02-06) - 🎨 UI Polish & Core Stability (Codex Integration)

✨ UI Polish & Experience (Antigravity)

  • ✅ Unified User Avatar - Standardized user avatar display across Pro Mode (ChatBubble) and Agent Mode, using the same Avatar component for consistency.
  • ✅ Theme Adaptability - Overhauled Login page to fully support both Light and Dark modes, ensuring optimal readability and aesthetic appeal in any environment.
  • ✅ Visual Refinement - Reduced border thickness on image hover states (GenerationResult, ChatBubble) for a sleeker, less intrusive look.
  • ✅ Performance Tuning - Optimized loadData with useCallback to prevent unnecessary re-fetches.

🛡️ Core Stability & Auth (Codex)

  • ✅ Hybrid Auth Strategy - Implemented a resilient "Proxy-First" auth check in getUserProfile bypasses client hydration delays, ensuring user data loads reliably even after browser refresh.
  • ✅ Race Condition Protection - Introduced activeUserIdRef lock in AuthProvider to prevent profile state pollution during fast account switching or logout.
  • ✅ Dropdown Interaction Fix - Reverted the complex Portal mask solution and switched to a lightweight global click listener, permanently fixing the issue where dropdown menus were unclickable.
  • ✅ Session Recovery - Added a smart fallback mechanism that recovers session from Supabase persistence with an 800ms timeout if cookies are missing.

v4.0.2 (2026-02-05) - 🛡️ Login & UI UX Improvements

  • ✅ Login Flow Overhaul - Replaced aggressive auto-logout with a user-friendly "Pending Approval" UI. Users waiting for whitelist approval can now refresh their status without re-logging.
  • ✅ Robust Profile Fetching - Fixed getUserProfile to explicitly include Authorization headers in proxy requests, resolving "User not found" issues caused by missing auth context.
  • ✅ IME Input Fix - Solved an issue where pressing Enter during IME composition (e.g., Chinese Pinyin) would accidentally submit forms in Login/Register pages.
  • ✅ Image Upload Fix - Fixed the Chat Input image upload button which was unresponsive due to a missing event handler.
  • ✅ New Skill Integration - Integrated Anthropic's official frontend-design skill for higher quality UI generation.

v4.0.1 (2026-02-05) - 🛡️ Critical Risk Fixes

  • ✅ Atomic Transactions - Migrated batch operations to Supabase RPC (delete_scene_atomic), eliminating data inconsistency risks during network failures.
  • ✅ Async Cron Architecture - Decoupled heavy video uploads from status checks. check-sora-status is now lightweight (status only), while retry-transfers handles R2 uploads asynchronously, solving Vercel timeout issues.
  • ✅ Image Upload Optimization - Added "Fast Path" for uploads < 3MB (skips compression) and implemented progress feedback.
  • ✅ Reference Management Fix - Fixed useAutoReference to strictly sync UI with deletion actions using useRef + versioning.
  • ✅ SSR Hydration Fix - Resolved Next.js hydration mismatches in useProjectStore by deferring localStorage reads to client-side effects.

v3.9.6 (2026-02-04)

  • ✅ Gemini Reference Image Optimization - Implemented "URL-first + Auto-fallback" strategy for reference images.
    • First attempt: Use R2 presigned URLs (fileData.fileUri) for instant transfer.
    • Auto-retry: If Gemini returns "Cannot fetch" error, automatically switch to server-side download mode (inlineData).
    • Benefits: Maximizes speed while ensuring reliability through transparent fallback.
  • ✅ R2 Presigned URL Support - Added generatePresignedUrl() function in r2-server-upload.ts for generating signed URLs.

v3.9.5 (2026-02-02)

  • ✅ Pro Mode Scroll Optimization - Created useChatScroll hook to unify chat scrolling logic, fixing initial load scroll issues and media load compensation.
  • ✅ Code Refactoring - Reduced 120+ lines of scattered scroll logic in ChatPanel.tsx to a clean ~20 lines using the new hook.
  • ✅ Scroll Position Preservation - Properly maintains scroll position when loading more messages (pagination).

v3.9.4 (2026-02-01)

  • ✅ Pro Mode History Loading - Pro mode chat now loads last 30 messages by default, with scroll-up pagination for older history.
  • ✅ Sora/Vidu R2 Persistence - Unified video transfer pipeline to Cloudflare R2 for both Sora and Vidu providers.
  • ✅ Sora 2 Pro Support - Added 15s/25s duration options with 1024x1792 / 1792x1024 resolution.

v3.9.3 (2026-02-01)

  • ✅ Reference Logic Decoupling - Completely refactored ChatPanel to use isolated state hooks (useVideoReferences) for Vidu/Sora modes, preventing state pollution.
  • ✅ Interaction Encapsulation - Moved drag-and-drop and file upload logic to useChatReferenceInteractions, significantly improving code maintainability.
  • ✅ Start-End Frame Optimization - "Ghost" projection removed; start frame is now a real state initialized from shot reference, allowing clear deletion.

v3.9.2 (2026-01-31)

  • ✅ Vidu Video Integration - Added Vidu video generation support with img2video and reference2video modes.
  • ✅ Unified Video Task Table - Extended sora_tasks table with provider field to support multiple video providers (Sora, Vidu, Jimeng, etc.).
  • ✅ Agent Tool Support - Added generateViduVideo tool for AI-driven video generation.
  • ✅ Task Management - Implemented ViduTaskManager for task creation, status tracking, and R2 persistence.
  • ✅ Ownership Validation - All video task APIs now validate user ownership before operations.
  • ✅ Credits Integration - Vidu video generation uses unified credits system (720p: 2 credits/sec, 1080p: 4 credits/sec).

v3.9.1 (2026-01-30)

  • ✅ Jimeng Pro Mode Upgrade - Consolidated multi-image generations into a single "Grid-like" message (2x2), removing the need for manual refresh.
  • ✅ Critical Login Fix - Resolved infinite login loop on /auth/login by strictly enforcing server-side session validation.
  • ✅ User UX - Added download button for user-uploaded reference images in Pro mode chat history.
  • ✅ Grid Layout Optimization - Fixed layout issues for 3x3 Grids, ensuring full visibility and adaptive width for all grid sizes (Single/2x2/3x3).
  • ✅ Image Preview Enhancements - Image preview overlay now supports keyboard navigation (Arrow Left/Right) with auto-focus support.
  • ✅ Download Positioning - Relocated the download button in Grid images to the right-side action group for better usability.

v3.9.0 (2026-01-30)

  • ✅ Grid Instant Apply - Applying a Grid image or slice to a shot (via drag, click, or new canvas button) now immediately marks the shot as 'Done' and persists to the database, ensuring "what you see is what you get".
  • ✅ Proxy Download System - Introduced /api/proxy-download to handle cross-origin R2 file downloads reliably, forcing file download instead of browser preview.
  • ✅ Canvas Interaction - Fixed missing Download/Edit actions in Grid state.
  • ✅ Reference Image Fix - Resolved issue where reusing history reference images failed in production by disabling fetch caching and adding proper User-Agent headers.
  • ✅ Frontend & UI Experience Optimizations:
    • Compact Grid Layout: Optimized 3x3 and 2x2 grid display for better screen utilization.
    • Smart Interaction: Separated click logic for Slice Selection (Grid) vs Image Preview (Single/Jimeng).
    • Aspect Ratio Fidelity: strict metadata persistence for vertical/mobile aspect ratios across all models.
  • ✅ Backend & AI Integration:
    • Multi-Model Support: Gemini, Jimeng, SeeDream integrated.
    • Robust Asset Pipeline: R2 storage, proxy downloads, and automatic history tracking.

v3.8.9 (2026-01-29)

  • ✅ Canvas Flicker Fix - Completely resolved canvas flickering during zoom/pan by refactoring grid rendering to CSS background and implementing memoized scene components.
  • ✅ Planning Chat Expansion - Added a fullscreen edit mode for the Planning Chat input, improving the experience for writing long prompts.
  • ✅ Performance Optimization - Significantly reduced re-renders in the infinite canvas through component memoization and optimized state management.

v3.8.8 (2026-01-28)

  • ✅ AI Command Restoration - Restored AI Quick Commands (Grid/Sora/Jimeng) in the Agent Panel for immediate access in image/video generation workflows.
  • ✅ New Script Tab Integration - Enabled "Script" tab in the unified sidebar, consolidating story brainstorming and AI script analysis into a dedicated view.
  • ✅ AI Quick Commands Relocation - Added "AI Quick Commands" to the new Script Tab while retaining them in Agent Panel, allowing for flexible workflows.
  • ✅ Canvas Rendering Optimization - Fixed canvas flickering during zoom/pan by removing conflicting CSS transitions and enabling hardware acceleration.
  • ✅ Agent Panel Cleanup - Enhanced Agent interaction flow while preserving key shortcuts.
  • ✅ Standardized Image Interaction - Added standard "Use as Reference" and "One-click Apply" hover actions to user-uploaded images in chat, matching the Grid interaction experience.

v3.0.0 (2026-01-19)

  • ✅ Pure Cloud Architecture - Removed guest mode, all data stored in cloud
  • ✅ 28 Agent Tools - Complete CRUD + generation + batch operations
  • ✅ Jimeng Integration - Chinese-optimized image generation
  • ✅ Pro Mode WYSIWYG References - "What You See Is What You Get" reference image logic
  • ✅ Latest-Video API Optimization - Pure read-only, no more R2 upload delays
  • ✅ Auth Middleware Refresh Token - Auto-refresh expired access tokens
  • ✅ SmartRecovery Fix - Prevents infinite polling loop on page refresh

v3.8.7 (2026-01-27)

  • ✅ Prompt Logic Unified - Centralized prompt construction logic (constructBaseShotPrompt) for Agent, Pro Mode, and Canvas generation.
  • ✅ Punctuation Optimization - Intelligent prompt concatenation that removes redundant punctuation and uses periods to separate scene descriptions for better clarity.
  • ✅ Canvas Generation Repair - Fixed the "Regenerate" button on canvas to use the latest prompt logic, ensuring consistency with Pro Mode.
  • ✅ Canvas Layout Optimization - Shot cards on canvas now dynamically adapt to the project's aspect ratio (e.g., 9:16 vertical cards).
  • ✅ Adaptive Grid Density - Grid columns automatically adjust based on card width (6 columns for vertical, 4 for wide) for better screen utilization.
  • ✅ Responsive Card UI - Redesigned bottom toolbar and info layout specifically for narrow vertical cards (Tall mode) to prevent overcrowding.
  • ✅ Pro Mode UX - Auto-fills complete prompt in Pro Mode when viewing shots with no user history (only AI-generated content).

v3.8.6 (2026-01-27)

  • ✅ Reference Drag & Drop - Implemented draggable sorting for reference images in Pro Mode. New references now append to the end, preserving user-defined order.
  • ✅ Prompt Enhancement - Injected scene.description into generation prompts for better context awareness.
  • ✅ Preview Fallback - Image previews now fallback to project aspect ratio when metadata is missing, ensuring consistent UI visuals.
  • ✅ Code Refactoring - Consolidated shot translation logic and improved state management for reference ordering.

v3.8.5 (2026-01-27)

  • ✅ API Timeout Optimization - Unified all API routes to 120s timeout (SeeDream/Jimeng/Gemini-Image), fixing production FUNCTION_INVOCATION_TIMEOUT errors.
  • ✅ Agent Concurrency Fix - Separated Jimeng and SeeDream call logic; implemented Jimeng client-side polling (check-status-once) to avoid 60s server blocking.
  • ✅ Supabase Connection Resilience - Auth middleware now uses exponential backoff retry (up to 3 times); client session refresh adds retry (up to 2 times).
  • ✅ Jimeng API Enhancement - Added checkTaskOnce(), check-status-once action, and pollTaskClient() for non-blocking status checks.

v3.8.4 (2026-01-27)

  • ✅ Concurrency Optimization - Implemented parallel R2 uploads (Promise.all) for Grid generation, reducing upload time by ~4x.
  • ✅ Non-blocking UI - Decoupled upload status from generation status, allowing immediate interaction after AI generation completes while uploads continue in background.
  • ✅ Refined UX - Removed persistent "Generating..." spinner during upload phase to improve perceived performance.

v3.8.3 (2026-01-27)

  • ✅ R2 Presigned Uploads - Implemented direct-to-R2 upload using Presigned URLs to bypass Vercel 4.5MB payload limit, fixing 413 errors for large Grid images.
  • ✅ Chat Message Deletion - Added ability to delete chat messages in Pro Mode (Soft delete, preserving referenced R2 files).
  • ✅ UI Refinement - Enhanced Chat Bubble UI with hover-visible actions and refined iconography.
  • ✅ Linting Fixes - Fixed React Hook dependencies in Chat History management.

v3.8.2 (2026-01-27)

  • ✅ Server-side Image Compression - All API routes (gemini-grid, gemini-image, jimeng) now use sharp to compress reference images to max 2048px JPEG
  • ✅ Payload Size Fix - Prevents "请求载荷过大 (5MB+)" errors when using high-resolution images as references
  • ✅ R2 Upload Resilience - Added exponential backoff retry (up to 5 times) for R2 uploads
  • ✅ Data Safety - Added page unload warning during generation/upload to prevent data loss
  • ✅ Scene Grid Persistence - Fixed bug where Grid generated in Scene mode wasn't saved to history
  • ✅ View State Sync - URL now syncs with current view (Canvas/Planning/Timeline), fixing persistent view issues
  • ✅ Prompt Engineering - Removed redundant prefixes (e.g., "镜头画面:", "景别:") for cleaner, more direct prompts
  • ✅ Favicon Support - Added favicon.png and apple-touch-icon.png for browser tab icons
  • ✅ Project Structure Reorganization - Merged components (input/pro → chat, navigation/settings → layout), organized hooks into functional groups

v3.8.1 (2026-01-27)

  • ✅ Pro Mode Refactor - New useChatGeneration hook for message/upload logic
  • ✅ Image Compression - Auto-compress uploads to 2048px JPEG before R2 upload
  • ✅ Backend Optimization - API routes fetch images from URL, 20MB limit, 15s timeout
  • ✅ Drag Fix - Unified handling for shot drag and file drag

v3.8.0 (2026-01-27)

  • ✅ Bidirectional Drag-Drop - Drag generated images to Storyboard shots (with history sync) & Drag shots to Pro chat as references
  • ✅ ChatPanel Decoupling - Refactored core logic into useChatHistory and useAutoReference hooks
  • ✅ Image Preview Drag - Direct drag support from full-screen image preview
  • ✅ History Synchronization - Optimistic updates + Backend persistence for all drag-drop actions

v3.6.0 (2026-01-26)

  • ✅ Pro Mode Grid Simplified - Simple prompt template for user creativity
  • ✅ Reference Image Optimization - Max 10 images, 10MB each, 4MB budget after compression
  • ✅ Timeout Unified - All API routes use 120s timeout
  • ✅ Code Cleanup - Removed unused useProGeneration.ts hook

v3.1.0 (2026-01-20)

  • ✅ Sora Registration Optimization - Async registration (<1s response) + Smart Task Recovery
  • ✅ Smart Asset Generation - Auto-detects missing assets for imported storyboards
  • ✅ Inspiration Auto-Trigger - Auto-starts AI storyboard from homepage inspiration
  • ✅ Conflict Resolution - Mutual exclusion between Auto-Gen and Asset-Gen flows
  • ✅ UI Refinement - Unified color theme (Zinc/Neutral) for progress indicators

v4.0.0 (2026-02-02)

  • ✅ ChatPanel Hook 架构重构 - 提取 6 个独立 Hook 减少组件复杂度
    • useSoraConfig - Sora 参数管理
    • useApplyVideoToShot - 视频应用逻辑
    • useChatActions - 消息操作回调
    • useChatModals - 模态框状态
    • useVideoModeReferences 增强
  • ✅ 统一 API 响应 - api-response.ts 标准化错误格式
  • ✅ 任务队列服务 - TaskQueueService.ts 并发控制
  • ✅ Cron 重试机制 - 视频转存失败自动重试
  • ✅ Supabase 事务工具 - 批量操作原子性保证
  • ✅ UI 组件库 - StatusComponents.tsx 统一加载/空状态

v0.6.0 (2025-12-24)

  • ✅ Sora Video Generation - Full Sora 2 integration via Kaponai API
  • ✅ SoraOrchestrator - Automated pipeline for character registration and video generation
  • ✅ Character Consistency - @username-based character tracking across scenes
  • ✅ Dynamic Aspect Ratio - Auto-detect image ratio for optimal video output
  • ✅ Smart Scene Splitting - >15s scenes auto-split into chunks (Greedy Packing)
  • ✅ R2 Persistence - Automatic upload to Cloudflare R2 for video storage

v0.4.0 (2025-12-17)

  • ✅ User Authentication System - Supabase Auth integration
  • ✅ Credits System - Three-tier pricing (admin free, vip 80% off, user standard)
  • ✅ Request Cancellation - AbortController support for AI requests
  • ✅ Cloud Storage - Supabase PostgreSQL for projects and chat history
  • ✅ Chat History Sync - Three-level scope (project/scene/shot) cloud storage

v0.2.0 (2025-01-03)

  • ✅ Character AI turnaround generation (1/3 face + 2/3 views)
  • ✅ GridPreviewModal component for slice preview & manual assignment
  • ✅ Pro mode Grid generation integrated with preview modal
  • ✅ Audio upload functionality (music/voice/sfx)

v0.1.0 (2025-01-03)

  • ✅ Canvas zoom and pan
  • ✅ Gemini API integration for Grid generation
  • ✅ AI Agent conversation system (streaming output)
  • ✅ AI storyboard generation (8-principle rules)
  • ✅ Timeline editor

📄 License

MIT License

👨‍💻 Authors

Developed by 西羊石 Team, assisted by Claude Code + Gemini Code.

Star ⭐ this repo if you find it helpful!

本页内容由上游仓库自动镜像，如有疑问请访问原始仓库。