Data warehousing & activation for a Google-native BigQuery estate
Dubai
Re: Data Warehousing & Activation
muhammadasim@codeaza.org · codeaza.com
We build and scale Google-native data estates. Raw ingestion, then cleaned and certified view layers, then consolidated master bases, then activation. That is your exact shape.
We have not spent years in Snowflake or Redshift hunting for the next thing to port. We stay inside the stack you picked, sharpen it, write down what we change, and hand the logic back. You end up with an estate that is better documented than the one we walked into.
Why there is no Loom. Most of our BigQuery work sits under client NDAs, so I can't record their environments. Your brief allows a written note instead, so that is what this is. Same ground, same stack, minus the client names. Happy to walk through any of it on a call.
Warehouse organisation & retrieval
We fix estates where the bill grows faster than the data. Consistent naming, partitioning and clustering on the keys you actually join on, and turning heavy recomputed views into materialised or incremental tables. The right row comes back fast and cheap, even as sources pile up.
At scale: search indexes on the master bases so a single-couple lookup stops scanning the partition, physical (compressed) storage billing where the wide sparse rows earn it, and slot reservations split by workload so batch enrichment never starves interactive queries. We wire INFORMATION_SCHEMA.JOBS into a cost-attribution table, so every dataset has a bytes-billed number next to it and the top ten offenders are named, not guessed.
Governance & documentation at scale
Object and column docs plus a live registry that is the first stop before any query. Freshness signals flag when something has gone stale. Nothing ends up living only in one person's head. Commission and management data stays walled off under least-privilege access.
At scale the registry has to maintain itself. We drive lineage from Dataplex and the audit logs, so the dependency map reflects what actually runs, not what someone remembered to write down. Freshness is a computed MAX(ingest_timestamp) per object, not a green tick on a scheduled query that quietly succeeded on stale upstream data. The "team views never re-derive from raw" rule gets enforced by IAM on the raw datasets, not by convention that erodes the day a new hire gets read access.
In-warehouse AI agents
Routine pulls and enrichment that run natively against BigQuery, so the warehouse can answer on its own instead of routing everything through one tool. Every agent query runs under a cost cap.
The pattern we trust: Vertex AI function-calling with a small set of BigQuery tools that resolve against the registry first, so the agent reads documented truth before it writes a line of SQL. Each generated query is dry-run and capped with maximum_bytes_billed before it executes, so a bad prompt can't trigger a full-estate scan. Routine work moves to scheduled SQL plus Gemini, and the agent handles the ad-hoc questions a person would otherwise queue behind one analyst.
Gemini enrichment expansion
Scoring and classification run inside BigQuery on documented models. We extend that across calls, leads, and content. Records get re-scored only when their inputs change, so the Gemini bill stays flat as volume climbs.
We prefer AI.GENERATE_TABLE against a remote model, so scoring returns typed columns instead of free-text JSON someone has to parse in SQL downstream. An input-hash watermark on every row means a batch re-run only touches what actually changed. Batch scoring runs on its own reservation inside Vertex QPS limits, so a hundred-thousand-row backfill doesn't throttle the live call-scoring path.
Team activation & self-serve
Certified data reaches the team through Looker Studio, Sheets wired to BigQuery, and documented views. Answers stop waiting on one person.
BI Engine sits under the dashboards so the sales and ops boards return in under a second and don't re-bill on every refresh. Row access policies and authorised views mean a team member sees their segment and never the commission columns, without us forking the data into a second table to keep it safe.
Output automation
Certified data feeds automated content and reporting through Workspace and Apps Script. Where it matters, a person still signs off before anything goes out.
Event-driven is where this gets real. A certified row lands, a Cloud Function or Workflow fires, Gemini drafts the post or the report, and it parks in an approval queue before it publishes. No cron job scraping stale tables at midnight, and no content going live that a human hasn't cleared.