Vero
A clinical platform that follows the consultation in real time and runs the patient's pre-visit intake on WhatsApp
Lead author, clinical LLM pipeline, pre-visit intake agent, and document import
Private repository
About the project
Vero gives doctors cognitive support while the visit is happening. Audio is captured in the browser and transcribed via streaming; on top of that transcript, a clinical AI service raises diagnostic hypotheses, flags red flags, suggests courses of action, and drafts a prescription. The doctor follows all of this in a clinical reasoning panel with chat, marking what's useful and dismissing the rest, then closes the visit with everything consolidated. Before that, the patient has already gone through the other side of the platform: a WhatsApp agent runs the pre-visit intake, collecting the chief complaint, history, allergies, current medications, and exams, and hands the doctor an already organized case instead of a blank form. The system is a pnpm monorepo where frontend and backend share the same TypeScript and Zod contracts, including the WebSocket message format. Two Python microservices round out the stack: one handles the clinical LLM with RAG over embeddings, the other handles importing medical records and reports via OCR.
How it works
Each visit opens a session with two WebSocket channels: one for audio, one for state. Audio streams to Deepgram and comes back as a transcript; the connection recovers on its own with exponential backoff if the network drops. The clinical LLM service consumes the transcript as it comes in, retrieves context by similarity over the vector store, and returns hypotheses, red flags, and suggested courses of action, which appear to the doctor as the conversation progresses. The reasoning panel can be maximized, and the text streams in word by word instead of appearing all at once. PDF or image documents go through the import service, which extracts content with PyMuPDF or OCR and organizes the findings in the patient record. Scheduling, patients, visits, organization members, and the audit trail are all modules of the Fastify API, with JWT authentication via Supabase.
Architecture
- 01
React frontend with Vite: doctor's panel, schedule, active visit, and clinical reasoning chat
- 02
Fastify API in TypeScript: REST and WebSocket for audio and visit state
- 03
API modules: visits, appointments, patients, doctors, organization, and audit
- 04
Python clinical LLM service: embeddings pipeline, RAG, and hypothesis generation
- 05
Python import service: PDF extraction and OCR for medical records and reports
- 06
WhatsApp agent: pre-visit intake conducted with the patient before the appointment
- 07
Shared contracts package in TypeScript and Zod, including WebSocket messages
- 08
PostgreSQL with Supabase for authentication, data, and storage
Features
- Real time visit transcription with automatic reconnection
- Diagnostic hypotheses and red flags during the visit
- Suggested course of action and draft prescription
- Clinical reasoning panel with chat, marking insights as useful or dismissed
- Visit wrap up screen with all consolidated material
- Pre-visit intake conducted with the patient on WhatsApp
- Import of medical records and reports via OCR
- Scheduling and patient records with an audit trail
- Shared contracts between frontend and backend, including over WebSocket
Tech stack
- Frontend/Backend
- TypeScript
- Frontend
- React 19 · Vite · Tailwind CSS 4 · TanStack Query
- Backend
- Fastify · Node.js · WebSocket · Python · FastAPI
- AI/ML
- Deepgram · OpenAI · RAG
- Tools
- PyMuPDF · pytesseract · Zod
- Database
- PostgreSQL · Supabase
- Infrastructure
- Docker