I've worked with BigQuery and Snowflake for years, and one thing has always bugged me: you can't really run anything locally.

Testing a small dbt change, debugging a join, or running CI all mean hitting a real warehouse. That means compute costs, credentials, and shared dev datasets, even for tiny changes.

Today's GCP problems in London were a good reminder too. If your warehouse region has a bad day, you can't even do development, never mind run production.

So I started Crucible, a local BigQuery API emulator backed by DuckDB. Point dbt-bigquery or the Python client at localhost and run without a real GCP project, cloud credentials, or warehouse bill.

It's pre-alpha, but here's what works so far:

• dbt-core seed, run, test and unit tests against an example project
• Python BigQuery client, CSV loads, MERGE, partitioned tables, JSON, arrays and window functions

The part I care most about is correctness. Supported behaviour is verified against recorded results from real BigQuery. If Crucible can't match something, it explicitly returns "Not supported by Crucible" rather than guessing. An emulator that silently gives wrong answers is worse than none.

Still missing: scripting, INFORMATION_SCHEMA, wildcard tables and the Storage Read API (so dbt Fusion can seed/run but not test).

The goal is to cover typical BigQuery/dbt projects, expand SQL compatibility from there, and eventually support Snowflake too.

I'd really appreciate feedback from anyone using BigQuery/dbt. If you try it on your project, the most useful thing you can send me is the first query it refuses. That tells me what to build next.

Criticism and "this is a bad idea because..." are equally welcome.

Repo: https://github.com/CallumWalterWhite/crucible

Quick start:

docker run --rm -p 127.0.0.1:9050:9050 ghcr.io/callumwalterwhite/crucible-emulator:latest serve --dialect bigquery

table in company's data source >>> bigquery table >>> view >>> error after added field

Errror: Invalid view definition for project.business_vault.table_view with 17 columns vs. 18 columns from view query

Does anyhow have an idea how to avoid that unwanted behaviour when the created view fields to not fit anymore to the fields in the underlaying table, while a view field description is created?

context:

We have database in Bigquery that loads data from a business source into tables. Once in a while a new column is added in the source. The pipelines here work fine and fetch easily the new fields.

The tables feed into views to get the latest instances of the loaded data. Those views are what we actual work with.

Until recently, those views run smoothly, and a new field in the source and in Bigquery just did not appear in the view until it was added, but it did not cause errors and broke the view. That was until one manager decided to have field description inside the view.

I have now a spreadsheet that collects the field descriptions, and Pyhton code creates my Bigquey view creation statement. If this executions find a new field, it even catches that and does not break. This works fine.

However, as soon as the source adds a columns and before I execute my python script, a Bigquery query runs into an error and cannot execute the view anymore:

The view query is actually very small itself and does not show much, of course the CREATE statement, which is complex.

# simple SELECT in the view
SELECT
      *
FROM
      `project.dataset.table`
    QUALIFY
      ROW_NUMBER()
        OVER (PARTITION BY id ORDER BY SAFE_CAST(_dlt_load_id AS FLOAT64) DESC)
      = 1

# complex part of CREATE that does not show in the view query later
CREATE OR REPLACE VIEW
 `project.business_vault.table_view`
   (id OPTIONS (DESCRIPTION = 'primary key'),
   updated_at OPTIONS (DESCRIPTION = 'timestamp for data updated in original source database'),
...
 OPTIONS (DESCRIPTION = 'What does the view do?',
 FRIENDLY_NAME = 'My friendly name for the view',
...

Ran a cost analysis over a 30-day window on a production project — 27,076 jobs, 45.1 TB billed, ~$282 in query cost. Anonymized.

Methodological caveat first, because it matters: BigQuery doesn't report bytes per table, so a job's entire scan gets counted against every table it references. Rank with that column, never sum it. Same for the findings below — they overlap, one query can trigger three rules. Don't add them up.

The interesting part: the project was spending ~$2.30/day. On 25 August it jumped to ~$25/day and stayed there.

Cause: one query pattern, executed 2,638 times in 30 days, billing 30.8 TB. On its own that's $192 — 68% of the entire month's query cost.

Not a badly written query. A good one running far too often.

What else showed up:

• 633 queries used SELECT * on tables over 1 GB → 4.0 TB. BigQuery bills the columns you read, not the rows you return.
• 2,214 queries scanned 10 GB+ with no LIMIT → 34.8 TB.
• 126 queries read a partitioned table without its partition filter → every partition scanned instead of the few needed.
• 42 queries ran 10+ times and re-billed 40.9 TB — the result cache only helps while the underlying data is unchanged.

The part that surprised me: 98% of query spend came from a single service account — the dbt runner. Nobody was being careless. It was a pipeline doing exactly what it was told, at a scale nobody was watching.

Which is the real problem with warehouse bills. The invoice tells you the total. It never tells you that one model, executed 2,638 times, is two-thirds of it.

Everything above comes from INFORMATION_SCHEMA.JOBS_BY_PROJECT — bytes billed, who ran it, what it referenced. No table data is read.

I built a small read-only tool around this — self-hosted, metadata-only, nothing leaves your machine.

That project's report is confidential, so I can't share it. What I can share is a sample the tool generates from built-in synthetic data — nothing in it is real, but the format and the findings are exactly what you get: [sample_report]

For a while we’ve been working on (among other things) text-to-SQL, and the main thing we learned is that the agent usually gets the SQL right and the meaning wrong. It picks the wrong table, or writes a join that runs fine but counts the wrong thing.

What fixed it for us was writing the meaning down once, upstream, as a canonical model: entities, relationships, join keys. Then both the tables and the agent work from that. (Longer writeup on that)

When BigQuery shipped CREATE PROPERTY GRAPH and Conversational Analytics, we thought: that’s basically the same idea, native in the warehouse. Let’s try it.

So our colleague loaded Chinook data into BigQuery, declared the graph, and pointed Conversational Analytics at it. It wrote correct GQL for a 4-hop question (customer → invoice → track → album → artist) with the Instructions field left empty.

blog link

Is anyone using property graphs beyond a demo yet? I’m curious how you find it

Transparency: I work at dltHub. We make dlt, which did the ingestion here. We recently launched maintenance background agents

A few years ago I got thrown into BigQuery. I never did any data engineering before, coming from frontend/fullstack application engineering. It took me a while to understand all the concepts. I felt like nobody is really explaining them well. BigQuery docs do a good job of explaining the What, but not the Why or the How To Decide. In this article, I try to give you both of that. By the way, the advice has changed considerably this year with Fluidscaling, so a lot of the old guides are also out of date now. Let me know what you think! Does BigQuery's pricing model (with the improved autoscaler billing) make sense?

Hey everyone, here's the 09/2026 edition! (Apologies for the late post!)

🔤 GoogleSQL Language Features & Functions

• WHERE Clause in Aggregate Calls (Preview) - Filter aggregate function inputs directly with a boolean expression inside function calls.
• OBJ.LIST Function for Unstructured Data (GA) - Discover and analyze Cloud Storage files spontaneously without creating persistent object tables.

🧠 AI, Machine Learning & Foundation Models

• Gemini 3.8 Flash Support - Execute generative AI SQL functions using the Gemini 3.8 Flash model.
• Gemini 3.5, 3.6, and 3.7 Flash Support - Run generative AI SQL functions using Gemini 3.5 Flash-Lite, 3.6 Flash, and 3.7 Flash models.
• AI.KEY_DRIVERS Function (GA) - Identify data segments causing statistically significant changes to a summable metric.
• AI.CAUSAL_EFFECT Function (Preview) - Quantify the impact of specific interventions on time series data using native SQL.
• ML.METRICS Function (Preview) - Compute classification and regression evaluation metrics on any query without a stored model.
• ML.CORRELATION in Conversational Analytics (Preview) - Calculate statistical correlations between target and metric columns via conversational questions.
• Predictive Modeling in Conversational Analytics (Preview) - Ask predictive modeling questions powered by the AI.PREDICT function in conversational analytics.
• Market Basket Analysis in Conversational Analytics (GA) - Query and analyze item association rules and market baskets through conversational analytics.

💻 Developer Experience (DX) & BigQuery Tooling

• Publish Data Agents to Gemini Enterprise (Preview) - Publish BigQuery data agents directly to Gemini Enterprise using Google-managed credentials via Agent Gateway.
• Data Engineering Agent Graph Integration (GA) - Enhance schema mapping accuracy across pipelines with automatic BigQuery Graph context.
• DTS MCP Server (GA) - Manage BigQuery Data Transfer Service workflows using Model Context Protocol tools for AI agents.
• Rust SDK for BigQuery (Preview) - Build BigQuery applications and data pipelines natively using the official Rust client library.
• Query Results Recent Runs History (GA) - Inspect previous execution history for queries and multi-statement queries directly within the results pane.
• Persistent Row Numbers on Horizontal Scroll (GA) - Keep row numbers visible while scrolling horizontally across wide query result tables.
• Jobs Explorer UI Improvements (GA) - Group resources, view timeline metric charts, and filter jobs with status counts in the console.

🗄️ Lakehouse Architecture, Apache Iceberg & Open Data Formats

• Continuous Queries to Apache Iceberg (GA) - Stream continuous query output directly into Apache Iceberg managed tables using INSERT DML statements.

⚡ Core Engine Performance, Indexing & Optimization

• DML and Export Steps in Query Plans - View execution steps for UPDATE, DELETE, MERGE, and EXPORT statements directly in query plans.

🔌 Data Integration, Pipelines & Ingestion (ELT)

• Dataform Pipeline Unit Tests (GA) - Validate SQL pipeline transformation logic against mock datasets before deploying to production.
• Migration Lineage Service (Preview) - Visualize source database connections and data flows to plan BigQuery warehouse migrations.
• Simba ODBC Driver Update - Connect external tools and BI platforms using the latest Simba ODBC driver release.
• Simba JDBC Driver Update - Integrate Java applications and enterprise tools using the updated Simba JDBC driver.

🔒 Security, Governance & Workload Management

• BigQuery Security Center (GA) - Analyze security profiles and manage column-level policies and governance tags in the Cloud console.
• BigQuery Graph Knowledge Catalog Ingestion (Preview) - Search and discover BigQuery Graph metadata automatically indexed within Knowledge Catalog.
• Disaster Recovery Failover History View (Preview) - Query INFORMATION_SCHEMA.FAILOVER_HISTORY to monitor reservation failover events in near real time.

As usual, let us know if there's anything: comments, questions, concerns!

Hi all,

Has anyone found a decent open source tool for asking questions in plain English on top of BigQuery? Something self hosted.

I tried Gemini in BigQuery and had a look at the conversational analytics API. The API gives you back SQL and a chart spec, so you still end up building the front end yourself, which is the part I was hoping to avoid.

My main worry is the numbers. If two people ask for revenue in slightly different words I'd like them to get the same answer, and I don't see how that works unless the metrics are defined somewhere.

The other thing is cost. I have no feel for how many queries a single question turns into once the model starts looking around the schema.

Has anyone here managed to use Bigquery Data Agents on external applications through the A2A protocol?

Once published, the agent lets you register the agent into the Agent Registry or copy the A2A json file into your clipboard. I’ve successfully published the agent on Gemini Enterprise, for example, but I am unable to use this JSON in any other part. My main goal would be to connect it to a Copilot Studio agent so I can easily use it through Teams, without having to do any code.

Interesting framework on how to optimize BigQuery costs across four layers: query optimization, pricing models, commitments, and discounts. https://www.alvin.ai/blog/bigquery-cost-optimization-the-complete-finops-framework

A short deep dive into a BigQuery join problem that often goes unnoticed until data gets big shuffle, skew, cardinality and query performance.

Partner sends us a monthly export. Different customer ID scheme to ours, no mapping table, no docs, and whoever built it on their side left last year.

Right now I fuzzy match on lower(trim(email)) plus surname and eyeball a sample before it goes anywhere. It works well enough that nobody's complained, which is not the same as it being right. I have no idea what my false match rate is.

The part that bothers me is that a bad join doesn't announce itself. A wrong customer count looks identical to a right one.

So what do people actually do here. Do you block on something cheap first to cut the comparison space, compute a score and hold back anything under a threshold, or just accept a fuzzy match and monitor downstream? And if you threshold, where did the number come from?

I recently came across Databricks AI/BI and was curious to know how people are finding it.

It looks like Databricks is trying to bring BI and analytics more directly into the Databricks platform, with dashboards and Genie for asking questions in natural language.

Has anyone actually tried AI/BI in a real project?

How is it compared to Power BI or Tableau from your experience? Is it good enough for regular BI use cases, or is it still better to use a separate BI tool?

Would like to know your experience, especially if you have used both.

Curious how most people picked up tools like BigQuery. Did you learn and practice entirely on your own through side projects and certifications before applying, or did you just happen to be at a company when they migrated over and learned on the job?
Would love to hear what your path looked like!

As a data analyst what kind of conceptual or theoretical questions get thrown at you?

What kind of SQL queries am I actually expected to write or explain on the spot? (Window functions, multiple CTEs, optimization, etc.?)

It looks like every integer dtype stored is using 64 bits, there's no real tinyint and smallint and this is just a syntatic sugar. And I didn't found the reason for that when I searched about it. Is it to make storage artificially more expensive?

Almost every open tabular engine and storage format that I knew so far allows you to really set a column as smaller types of int, independently if it is columnar or row-stored, and if it's a database or just an engine. For example, any modern apache arrow based engine (like Datafusion, DuckDB and polars), apache parquet files, almost every columnar database (like Clickhouse), almost any row-oriented database (like PostgreSQL, Oracle, MySQL and et cetera). Why BigQuery wants to be different?

If a query suddenly takes 5–10x longer without any SQL or data changes, the bottleneck may be workload contention, capacity, or concurrency.

A practical guide on how to diagnose it using JOBS, JOBS_TIMELINE, total_slot_ms and period_estimated_runnable_units.

Get "Error loading permissions for selected resources." error when I click selected table -> share -> Manage permissions.
Anyone know what has updated ?

Olá, pessoal!!!

Alguém que já realizou a integração do Meta Ads ao BigQuery via DataTransfer saberia me responder que campo e tabela consideramos a visão de custo por anúncio? Eu estou olhando para a coluna spend da tabela AdInsights e em um mês aparece mais de 77k de custo, é um número totalmente absurdo. Estou olhando para o lugar errado e/ou da maneira incorreta?

Hey everyone! Back for the monthly summary.

🔤 GoogleSQL Language Features & Functions

• Table Parameters in TVFs (GA) - Pass dynamic tables into table-valued functions using the generic ANY TABLE parameter type.

🧠 AI, Machine Learning & Foundation Models

• Gemini 3.1 Flash-Lite & 3.5 Flash Support (GA) - Run generative AI SQL functions using Gemini 3.1 Flash-Lite and 3.5 Flash across multi-regional endpoints.
• XGBoost 2.1 Model Training (GA) - Train and serve BigQuery ML models using XGBoost version 2.1 via the XGBOOST_VERSION option.
• Time Series Analysis TVFs (Preview) - Analyze trend, seasonality, and change points in time series data using native BigQuery ML TVFs.

💻 Developer Experience (DX) & BigQuery Tooling

• Cloud CLI Remote MCP Server (Preview) - Enable AI agents to execute BigQuery CLI commands and reservation management via managed MCP tools.
• Data Agent Observability & Monitoring (Preview) - Track performance, adoption, latency, and operational costs for data agents using Google Cloud Observability.
• Python UDF Real-Time Logging (GA) - Stream and inspect real-time execution logs for Python user-defined functions directly within Cloud Logging.
• Reference Panel Migration - Access table details directly through BigQuery Studio's unified Reference panel following Table Explorer deprecation.

⚡ Core Engine Performance, Indexing & Optimization

• Restored Hybrid Search - Restored hybrid search functionality combining vector embeddings and keyword search in VECTOR_SEARCH and AI.SEARCH.

🔌 Data Integration, Pipelines & Ingestion (ELT)

• Cross-Cloud Connections (Preview) - Query data residing in AWS, Azure, and Salesforce Data 360 from any BigQuery region.
• OpenTelemetry for BigQuery JDBC Driver (GA) - Trace JDBC driver queries and automatically export database telemetry to Google Cloud Observability.
• Pipeline Automated Metadata Enrichment (Preview) - Define semantic pipeline metadata in SQLX configs for automated synchronization with Knowledge Catalog.

🔒 Security, Governance & Workload Management

• Data Clean Room Query Templates (GA) - Share predefined queries in data clean rooms securely without exposing underlying datasets or views.
• Data Engineering Agent HIPAA Compliance - Build pipelines in regulated healthcare environments with official HIPAA compliance for Data Engineering Agent.
• Terraform Support for Governance Tags (Preview) - Provision and manage BigQuery data governance tags declaratively using Terraform infrastructure-as-code.
• Expanded Reservation Assignment Limits - Increased the default per-project reservation assignment limit from 10 to 100 for user-specific workloads.

⚠️ Breaking Changes, Deprecations & Pricing Updates

• BigQuery Graph Edition Enforcement - Core graph processing will require Enterprise or Enterprise Plus reservations starting April 26, 2027.
• DTS JDBC Driver Security Patch - Fixed an improper input validation vulnerability in legacy BigQuery Data Transfer Service JDBC drivers.

As usual, let us know what you think!

I use BigQuery a lot and wanted a slightly easier way to inspect query results, so I made a small Chrome extension.

It opens copied query results in a separate viewer with things like nested JSON/ARRAY expansion, column pinning, search, and better handling of long cells.

If anyone finds it useful, or has feedback, I'd be happy to hear it.

https://chromewebstore.google.com/detail/query-result-viewer-for-b/fdndhccllolpcfiohahlpbncogehmdhh

Been digging into this lately and figured I'd share, since it cleared up something I was fuzzy on: how BigQuery actually discovers Iceberg tables, and what's genuinely new here vs. what's just been repackaged.

The old ways of doing this:

1. Pointing BigQuery straight at a metadata file. Something like gs://bucket/orders/metadata/00003.metadata.json. It works, but it's fragile. Iceberg writes new metadata files instead of editing the old ones, so if any other engine commits a new snapshot, your pointer is instantly stale. You're stuck manually bumping it to 00004.metadata.json every time.
2. Catalog based access, like using AWS Glue as an external catalog. This actually fixes the staleness problem since the catalog tracks current state for you. But every engine still needed its own custom integration to talk to that catalog.

What's actually new: Google now has a managed Lakehouse Runtime Catalog (BigLake) that exposes a real Apache Iceberg REST Catalog endpoint, using the same open REST Catalog spec other engines already speak. So instead of building a one off integration per engine, anything that already understands Iceberg REST can just talk to BigLake directly. The data itself never moves, it stays in Cloud Storage. BigLake is really just a metadata and governance layer with IAM sitting on top of it.

There's also a version detail worth knowing: Iceberg 1.10 shipped a native BigQueryMetastoreCatalog and BigQueryMetastoreClient, plus a GoogleAuthManager that handles Google credential auth inside Iceberg's REST auth framework. It's easy to conflate these two things, but they're not the same. One's a catalog implementation built for BigQuery Metastore specifically, the other is the standardized REST interface. Google's steering people toward the REST endpoint for anything new where cross engine interoperability matters.

The post I read also gets into the practical setup side: creating a BigLake catalog (single bucket vs multi bucket), the IAM roles Google recommends (BigLake Admin, Editor, Viewer, plus Storage Object User), and shows an open source tool called OLake Go (full disclosure, I work on this) writing Iceberg tables directly into that catalog so BigQuery can see them right away, queryable through the usual four part PROJECT.CATALOG.NAMESPACE.TABLE identifier with no manual metadata pointer updates. There's a decent troubleshooting section too. Apparently if your connection test passes but the actual sync fails, it's almost always a missing storage level IAM role rather than bad credentials.

Full post here if you want the details: https://olake.io/blog/biglake-iceberg-rest-catalog-olake-go-setup/

Curious if anyone here has already moved existing Iceberg on BigQuery setups over to the REST catalog, or if you're all still doing the manual metadata file dance like I was.