Three Years Building a Data Platform: Lessons in Speed, Scale, and Sustainability

August 2022 to October 2025. Three years building a data intelligence platform as an internal tool for a venture capital fund, from spreadsheet to AI-powered system that helps identify opportunities early and make better investment decisions.

This is a reflection on practical lessons learned while building an internal VC tool that became genuinely valuable. Not about what went wrong, but about what early-stage projects naturally teach you about speed, sustainability, and evolution.

If you're building a tech team or early-stage project, these lessons might help.

The Beginning: Speed as a Feature

We started with a web app, basic but functional. Crunchbase data in spreadsheets, simple filtering, basic styling. The vision: an AI-powered intelligence platform for deal sourcing and analysis.

A team of four, moving fast. The first six months were incredible. Features that typically take months shipped in days. Each demo validated the approach. The energy was contagious, every sprint brought new capabilities, and we could see the platform's value growing with each iteration.

This early momentum was crucial. We needed to prove the concept quickly, show that automated deal sourcing could work, and demonstrate value to internal stakeholders. Speed wasn't just nice to have, it was essential for validation.

Early lesson: Speed validates ideas. Sustainability requires different tools.

What We Built

By late 2024, the platform included automated web crawling across dozens of sources, entity resolution across 10+ databases, ML-powered scoring and agentic workflows, web application, Chrome extension, and CRM integrations, processing millions of records with full-text and semantic search capabilities.

The scale: we crawled over 5 million public domains to track startup domains and their evolution. For companies of interest, we actively crawl their landing pages every 3 months to track changes in positioning, messaging, and product offerings. This systematic monitoring captures pivots, product launches, and strategic shifts before they're announced elsewhere.

Beyond website tracking, the platform monitors signals like news mentions, headcount changes, web traffic patterns, GitHub stars, and other growth indicators to identify momentum early. This continuous monitoring means the fund doesn't miss opportunities, every signal gets captured and analyzed.

Working with this volume of data came with interesting constraints. The team got creative about staying within boundaries while still getting the quickest alerts possible. Every new data source meant thinking through privacy implications, finding clever approaches to collect signals without overstepping. It slowed us down at times, but it pushed us to be more thoughtful about what data actually mattered and how to get value from public information responsibly.

The architecture: Python pipelines, PostgreSQL, custom crawling infrastructure, multiple API integrations. Built to handle scale while remaining flexible enough to add new data sources quickly.

What makes it special: the AI system can identify potential stealth founders before they publicly launch, catching emerging opportunities extremely early and providing early alerts on promising founders. Combined with continuous tracking of key growth signals, this early identification capability is the key competitive advantage, invest early, win. I'd say we're ahead of any VC in Europe on this capability.

Key insight: Building features is straightforward. Building maintainable systems is the real craft.

Team Evolution: Natural Growth Patterns

Building the platform was one thing. Building it with an evolving team was another challenge entirely.

Over three years, the team evolved naturally, typical of early-stage projects:

Started with 4, grew to 6 engineers at peak
Mix of experience levels, each bringing unique strengths
High velocity building phase (2022-2023)
Natural transitions as people pursued new opportunities (2024-2025)
Platform continues serving users with evolving team composition

Lessons on team dynamics in early-stage projects:

Early-stage projects naturally see team evolution. People join for growth and learning, then move to new chapters. This is healthy and expected.

The challenge: Knowledge transfer. Custom systems work brilliantly with original builders present. New team members need conventional patterns, clear documentation, and explicit context.

Key insight: Sustainable systems outlast any individual contributor. That's the goal.

A note to the new team: I hope for the best for you as you carry this platform forward. I have deep respect for the previous team members who built this with incredible dedication. I trust you to take it to new heights. I regret we didn't establish proper conventions or a cleaner codebase for you, but if we hadn't moved that fast, we wouldn't have validated the concept and delivered value this quickly. So I apologize for the mess, but know that the mess is a reminder of how hard we pushed to make this real. You're inheriting something that works and delivers value, make it yours.

Practical Lessons from Early-Stage Development

These team dynamics and technical challenges taught me specific lessons about building early-stage projects. Not theoretical insights, but practical patterns that emerged from real experience.

1. Speed vs. Sustainability

Speed validates ideas early. Sustainability makes you faster later. They're not opposites, they require different tools at different stages.

In our first year, shipping fast was the right choice. We needed to prove the concept, demonstrate value, and iterate based on feedback. But as the platform grew and the team evolved, the lack of documentation and conventional patterns started slowing us down. What once took hours to implement now took days because new team members needed to understand custom systems.

The good news: AI coding assistants now make documentation nearly free. Established frameworks come with community support and familiar patterns. Both speed and sustainability are achievable, you just need different approaches at different stages.

2. Architecture and Knowledge Transfer

Custom solutions work brilliantly with original builders present. Conventional patterns ("boring technology") are gifts to future team members, they provide immediate understanding and community resources. Onboarding time is the best signal of architectural quality.

3. Technical Debt is Product Decisions

Technical debt compounds exponentially. The first shortcut costs 2 hours to pay back. The hundredth costs 2 weeks. "Later" rarely comes because there's always something more urgent.

We took many shortcuts to ship fast, and each made sense at the time. But debt accumulates. Systems become fragile. Changes that should be simple become risky. New features take longer because you're working around old shortcuts.

The lesson: treat shortcuts as explicit product tradeoffs. Document why you're taking them, acknowledge the cost, schedule the payback, or consciously accept living with it. Just don't pretend the debt doesn't exist.

4. Documentation as Architecture

Implicit knowledge walks out the door with people. Document decisions (why), not just code (what). If new engineers can't get productive quickly, the system needs better documentation or simpler architecture.

5. Light Governance Enables Speed

Autonomy without coordination creates fragmentation. Good governance is clarity about decisions and coherence, not bureaucracy. Architecture Decision Records (ADRs) for major choices. Clear ownership of technical direction.

6. Sustainable Pace

Early-stage projects often glorify hustle. Late nights, weekends, "whatever it takes" to ship. The energy is real, and sometimes that intensity is necessary for short bursts.

But burnout is predictable math: excessive work + insufficient support + inadequate time = inevitable breakdown. Sustainable pace isn't weakness, it's wisdom. When firefighting becomes routine instead of rare, that's a signal to stop and fix the root cause, not a badge of honor.

The goal: build systems that don't require heroes. Make the platform robust enough that it doesn't need constant intervention. Protect team energy as a core value, not an afterthought.

7. Compliance as Insurance

Data governance and legal constraints slow you down early, but they prevent existential risks later. One regulatory issue can destroy years of work, so we kept eyes open from the start.

Constraints also force better thinking, what data actually matters? What signals can you extract responsibly? Treat compliance as insurance, not overhead. Build systems that won't come back to haunt you.

The Impact and Lessons

Looking back at three years of building, what did we actually accomplish?

A full-stack platform with 50+ major features, LLM agents autonomously sourcing and screening startups, and millions of records processed. The platform continues powering investment decisions today, with the ability to identify opportunities before they become obvious to the market.

We solved genuinely hard problems—entity resolution, real-time processing, complex crawling, ML-powered scoring. Working with talented engineers who brought diverse approaches taught invaluable lessons.

Core insights from early-stage development:

Speed and sustainability require different tools at different stages - Both are necessary, not opposites. Speed validates ideas early. Documentation, conventional architectures, and light governance become valuable as the platform matures.
Knowledge transfer is architecture - Custom systems work brilliantly with original builders. New team members need conventional patterns and clear documentation. Onboarding time is the best signal of architectural quality.
Teams evolve naturally in early-stage projects - People join for growth, then move to new chapters. Build systems that outlast individuals.
Technical debt is product decisions - Treat shortcuts as explicit tradeoffs. Document why, acknowledge the cost, schedule payback, or consciously accept living with it.

The lessons aren't about what went wrong, they're about what different stages need.

What's Next

I'm now moving to a new journey with Jadkan.ai, an intelligence layer for business operations, applying these lessons from day one: conventional architecture, AI-powered documentation, light governance, sustainable pace, and onboarding as a metric.

Also running Intelek.ai part-time, helping teams build sustainable systems.

Hoping to bring these lessons to build the best engineering team. Please keep in touch if you'd like to witness the journey, ping me anytime at supawat@tamsri.com