PixelMind
Video editor that accepts natural language commands, first place at the Trilha hackathon
Backend architect
Private repository
About the project
PixelMind lets you edit video by talking to the system. Instead of hunting for the clip on the timeline, the user types what they want (for example, removing the part where a certain topic is discussed) and the system locates and executes the cut. The project took first place among six teams at the Trilha hackathon, an initiative of TRIL Lab at UFPB, and was built in about a month. It started as a single TypeScript backend and grew into a multi service architecture as new capabilities were added: transcription, semantic analysis, and finally audio processing. The idea behind it is to make editing accessible to people who don't master traditional tools, turning the video's own transcript into the index by which it's manipulated.
How it works
The uploaded video has its audio extracted with FFmpeg and transcribed by Whisper, which returns timestamps for each segment. Gemini analyzes that text and builds a semantic map of the content. When a natural language command comes in, the model interprets the intent, locates the matching segments in the transcript, and FFmpeg executes the cuts using the timestamps. Audio quality is handled by a dedicated Python microservice, which applies spectral noise reduction with librosa and noisereduce and returns the processed track for reassembling the final video.
Architecture
- 01
Next.js frontend with App Router, TypeScript, and Tailwind
- 02
Node.js and TypeScript backend: upload, orchestration, and controllers for cutting, transcription, and analysis
- 03
Dedicated Python FastAPI microservice for spectral noise reduction
- 04
FFmpeg: audio extraction and cut execution
- 05
Whisper for timestamped transcription and Gemini for semantic analysis
Features
- Video editing through natural language commands
- Automatic transcription with timestamps
- Semantic analysis of spoken content
- Precise cuts guided by the transcript
- Spectral noise reduction in a dedicated microservice
- Processing step tracking in the interface
Tech stack
- Frontend
- Next.js · TypeScript · Tailwind CSS
- Backend
- Node.js · Python · FastAPI
- AI/ML
- Whisper · Gemini · librosa · noisereduce
- Tools
- FFmpeg