Query a Folder of Parquet Files with SQL

See everything Query Streams can do beyond this guide Get Started Free

Supports

+more

Live Data

Replace exports with on-demand SQL

Excel Sheets

BI & AI Analytics

Ask Nova, get SQL + charts

Meet Nova

+more

Automation

Scheduled sync to 6+ platforms

Explore

+more

API to SQL

Query APIs with SQL, no code

Explore

Query Streams is a secure, real-time database integration platform, and its Parquet Folder connector turns a folder of .parquet files into one live SQL table you can query from Microsoft Excel, Google Sheets, Airtable and your AI assistant. Learn more at QueryStreams.com and sign up for free to query your first Parquet folder in minutes.

Somebody gave you a folder of Parquet files

It arrives by one of a few well-worn routes. A data-lake export lands on a share as a few hundred part-00000.parquet files. A dbt or Spark job writes its output somewhere and the somewhere turns out to be a network folder. An analytics team hands over an extract. A warehouse table gets archived to disk when the licence renewal looks expensive, and eighteen months later somebody wants to know what was in it.

The awkward part is that Parquet is not human-readable. Double-clicking a .parquet file does nothing useful. Opening it in a text editor gives you a screenful of binary. The advice you get when you ask is either “spin up Spark” or “write a bit of Python”, and neither is a serious answer for someone whose actual job is to put a number in front of the finance director on Thursday.

The Parquet Folder connector is the short route. You point the Query Streams Agent at the folder, and every matching file becomes part of a single SQL table. Nothing is converted, nothing is merged into a new file, and nothing is uploaded — the files stay exactly where they are and you query them where they sit.

The schema is already inside the file

This is the one fact that makes Parquet different from every other folder type, and it is worth understanding because it explains why this connector is so uneventful to run.

Every Parquet file ends with a footer: a block of metadata that states the column names, their exact data types, whether each column allows nulls, and how many rows the file holds. That footer is written by whatever produced the file, and it is authoritative. It is not a hint or a convention — it is the file’s own declaration of what it contains.

So when the connector samples the folder, it is not inspecting data values and drawing conclusions from them. It asks each sampled file to describe itself, reads the answer, and pins that as the table definition. The types are declared rather than inferred, which means there is no sample size to worry about and no possibility that the ten-thousandth row contradicts what the first hundred implied.

Types are also carried through untouched. A DECIMAL(18,2) column in the file is a decimal column in the table, not a floating point approximation of one, because the value never passes through a string on its way in. Everything else in this family — the folder scan, the file ledger, the encrypted local cache, the housekeeping tables — works exactly as described in how File Set connectors work. What follows is only what is different about Parquet.

What you never have to argue about

The clearest way to see the advantage is to line it up against CSV, which is the same job without the footer. Everything a CSV Folder connector has to detect, agree on and defend is simply stated up front in Parquet.

There are correspondingly few options to set. A CSV Folder connector carries a delimiter, a quote character, a header switch, an encoding, null strings, a date format and a layout mode, because all of those are genuinely uncertain about a text file. A Parquet Folder connector has none of that surface, because the file has already answered — the settings it does have are listed further down.

Columns that only some files have

Folders of Parquet files are rarely written all at once. A pipeline gets a new field in March, so files from March onwards have a column that January’s do not. This is the one genuine judgement the connector has to make, and it makes it in a specific way.

The pinned table is the union of the columns found across the sampled files, not their intersection. A column that appears in only some of them still joins the table — it is simply marked nullable, and reads as NULL for the files that predate it. You therefore keep every column in the folder rather than losing whichever ones were added late.

A column being missing is tolerated. A column having a different type is not. If order_id is a 64-bit integer in one file and text in another, the connector refuses to pin at all and names the disagreement precisely: which column, which type, in which two files. It will not pick a winner, and it will not quietly cast one to the other to make the numbers line up.

Sniffing is cheap, because nothing is read

Query Streams samples up to 64 files when it first connects a folder. For CSV that means opening and parsing 64 files. For Parquet it means seeking to the end of 64 files and reading a metadata block, which is a trivial amount of work even when the files are hundreds of megabytes each.

The same trick pays off during the sync. Each file’s row count comes out of the footer without touching the data, so the connector knows in advance whether a file will breach the per-file row cap, and whether the sync as a whole is heading past its row budget. It can refuse and name the limit before the offending file is written into the table, rather than discovering the problem halfway through and leaving you with a partial load.

File size is checked first of all, before a single byte moves. A file over the size cap is refused by name and the rest of the folder syncs around it.

Every row knows its position in its file

As with every File Set connector, each row carries a _source_file column naming the file it came from and a _row_id that is unique across the table. In a Parquet folder, _row_id is better than that: Parquet readers expose a genuine file row number, so the identifier is built from the file name plus that row’s real ordinal position within the file, zero-padded to twelve digits.

The padding is the point. Sorting by _row_id puts rows in file order rather than in the order 1, 10, 100, 2, and twelve digits is enough to keep that true for a trillion rows in a single file — comfortably past any cap you could raise. CSV has no equivalent concept to draw on, so this is one place where the columnar format gives you something extra rather than merely saving you trouble.

SELECT _source_file,
       COUNT(*)        AS row_count,
       SUM(net_amount) AS net_amount
FROM   orders
GROUP  BY _source_file
ORDER  BY net_amount DESC;

And because the row identifier is ordered, you can look at the beginning of a specific file without reading the whole folder — useful when a partition looks suspicious and you want to see what is actually in it.

SELECT _row_id, order_id, order_date, net_amount
FROM   orders
WHERE  _source_file = 'part-00042.parquet'
ORDER  BY _row_id
LIMIT  20;

When a file stops matching the pin

Once the shape is pinned, every file is checked against it on the way in — and again, that check is a footer read, not a data read, so it is cheap enough to do on every file every time. A file that no longer agrees is parked: its rows stay out of the table, the reason is recorded, and the rest of the folder syncs normally.

Four things count as a disagreement, and they are worth knowing apart because the third one surprises people.

• A type changed. The column exists but the footer now declares it as something other than what was pinned.
• A required column vanished. A column the pin marked as not-nullable is absent from the file.
• A file dropped a column it used to have. Even when the pinned column is nullable, a file that did carry that column when the folder was sampled and has since stopped carrying it is parked rather than read as nulls. Reading it as null would quietly deflate every sum built on that column, and a wrong total is worse than a missing file.
• A brand new column appeared. A column the pin has never seen, and that you have not excluded, parks the file rather than being silently discarded.

Re-reading a file replaces its rows rather than adding to them. Whatever a file previously contributed is removed and its current contents are inserted in one transaction, so a re-written partition cannot leave you with duplicates or with a half-updated table if something fails mid-way.

Every Parquet setting, and when to change it

The most useful thing to say about this card is what is not on it. A Parquet Folder connector has no format-specific settings at all. A CSV folder has a delimiter, a quote character, an encoding, null strings and a date format. An Excel folder has a sheet list. A JSONL folder has a flatten depth. Parquet has none of them, and the reason is the footer: the schema is declared inside every file, so there is nothing to tell the connector and nothing for it to guess. The whole category of “which option did I get wrong?” does not exist here.

What remains is a short list. Everything else you see on the Connection tab — root folders, scan subfolders and max depth, exclude patterns, following symlinks, the filename parse pattern, folder token names, scan interval, content hashing, the delete policy, event log retention and the scan guard rails — behaves identically on every folder type and is documented once, in full, in the File Set connector guide. This section covers only what is specific to a folder of Parquet files.

Table name

Required. Parquet is a single-table driver: every file the include patterns match unions into one table, and this is what that table is called in SQL — orders, events, trips, whatever the folder actually holds. Letters, digits and underscores only, and it may not start with a digit; anything else is refused when you save rather than at sync time.

Choose it deliberately, because it is the name every saved query, every Nova question and every spreadsheet refresh will use. Renaming it later is not destructive, but it does mean going back through whatever you have already built on top of it. A folder of part files called part-00000.parquet tells you nothing about what the rows are, so the table name is usually the only human-readable label the data ever gets.

Include patterns

Default: **/*.parquet — seeded for you when you pick the Parquet Folder card. Glob patterns, one per line, deciding which files under the root folders are in scope. Left alone it takes every Parquet file in the tree, which is what you want the first time you point it at an export.

Narrowing this is the main lever you have, and it is nearly always the better move than raising a cap. Data-lake exports are partitioned, so the folder structure already carries the filter you want. If a scan refuses because the share holds more files than the cap allows, scope the pattern to the partitions you actually query rather than reaching for the file limit — a connector that reads four thousand files is faster to scan, faster to sync and far easier to reason about than one that reads four hundred thousand and filters afterwards.

**/*.parquet              every Parquet file in the tree (the default)
year=2026/**/*.parquet    one year of a Hive-partitioned lake
**/part-*.parquet         Spark part files, skipping hand-dropped extras
orders/**/*.parquet       one dataset out of a share holding several

One rule catches people out, and the wizard states it on the field itself: a mixed folder is two connectors on the same root, one per driver. If \\lake\exports holds CSVs alongside Parquet, widening this pattern to **/*.* does not give you both — the two formats cannot share a pinned shape. Create a Parquet Folder connector and a CSV Folder connector pointed at the same path, each with its own include pattern and its own table name.

Manifest overrides (JSON)

Hidden behind Show advanced options, and more useful on Parquet than anywhere else. Everywhere else in this family, overrides are a way to correct something the connector guessed. Here nothing is guessed — so an override is a way to correct something the pipeline did, which is exactly the case where you cannot simply fix the files.

Three scenarios cover almost every real use. A Spark job named a column col_14 or amt_net_x and nobody wants to write that in a report. A column holds something confidential and should not exist in the table at all. Or a numeric column needs more precision in the table than the files declare. All three are one-line edits here, applied when the shape is pinned, so they become part of the table definition rather than something you re-apply in every query.

{
  "renames": { "qty": "quantity" },
  "retypes": { "price": "DECIMAL(18,4)" },
  "exclude": [ "internal_notes" ]
}

• renames — give a column a better name. You cannot rename onto _source_file or _row_id, and two columns cannot be renamed to the same thing.
• retypes — force a column’s type. A type you set here is a deliberate instruction, so it is applied without the strict footer check that would otherwise park a file. Plain type names with optional precision only: DECIMAL(18,4), VARCHAR, BIGINT, DATE.
• exclude — leave columns out of the table entirely. Worth knowing that an excluded column is also a column the drift check stops caring about, so this is the clean way to ignore a field a pipeline keeps changing.

The box must contain a JSON object or nothing at all — invalid JSON is rejected when you save, not silently ignored. Emptying the box clears your overrides and returns the table to the shape the footers declare, which is the intended way to undo an override rather than deleting and rebuilding the connector.

Why there is no Table layout option

CSV, Excel and JSONL folders offer a Table layout choice — one pinned layout, or one table per layout — because those are the formats humans assemble by hand, and a folder of them genuinely does accumulate different column sets over the years. Parquet does not have that setting, on purpose. Parquet is machine-written by definition: something produced those files programmatically, and if that something is emitting several different shapes into one folder, splitting them into separate tables would paper over a pipeline problem rather than surface it.

So a Parquet folder gets one table, and disagreement is reported rather than accommodated. Missing columns are absorbed into the union and marked nullable; conflicting types are a named refusal. If you really are holding several unrelated datasets on one share, the answer is one connector per dataset, scoped with Include patterns — which is clearer anyway, because each one then gets its own table name.

Ingest caps and scan limits

The four ingest caps (rows and bytes, per file and per sync) and the scan limits (files per scan, depth, time budget, parallelism per root) are shared across every folder type, live behind Show advanced options, and follow one rule worth repeating: leave a box blank to accept the default, because zero never means unlimited and is refused outright. Their values and the reasons to raise each one are in the File Set connector guide; the ones that most often matter to a Parquet folder are listed in the guard rail table above.

Parquet does get one advantage here, described earlier: because the row count is in the footer, the per-file row cap is checked before any of the file’s data is read, so a breach is a named refusal rather than a load that fails part-way through. Raising a cap is still the second-best answer. Narrow the include pattern first.

Getting Parquet data into Microsoft Excel

Converting Parquet to Excel is the request behind most of this, and the honest answer is that you should not convert it at all. Once the folder is a connector, the Query Streams add-in for Microsoft Excel runs a saved query and drops the rows into the sheet, refreshed whenever you ask for it. You get Parquet in Excel without a conversion step, without a Spark cluster, and without a copy of the data ageing quietly in someone’s Downloads folder. The same query works unchanged in Google Sheets.

You need a Query Streams account and the Network Agent running on a machine that can see the folder — a server, a workstation with the share mapped, or your own laptop. Parquet Folder connectors require Network Agent 2.6 or newer.

1. Install the agent from the download page if it is not already running.
2. In the portal, add a data connector and choose Parquet Folder.
3. Enter the folder path as the agent’s machine sees it, and give the table a name — that field is required, because everything in the folder unions into one table.
4. Leave the include pattern on **/*.parquet unless the folder holds more than you want in the table, in which case narrow it — **/part-*.parquet, or a single partition such as year=2026/**/*.parquet.
5. Save. The agent reads the footers of up to 64 files, pins the schema and reports the columns and types it found.
6. Open the query builder, pick the connector and write SQL — or ask Nova AI to write it for you.
7. Save the query, then run it from the Microsoft Excel or Google Sheets add-in whenever you need the numbers.

From there the folder behaves like any other data source. It joins to Microsoft SQL Server, PostgreSQL, MySQL, BigQuery and the rest, and to API connectors such as Stripe or Shopify, so an archived warehouse table on a file share can be joined to live production data in a single read-only statement without loading anything into a warehouse first.

Frequently Asked Questions

Get Started

Related guides: How to query a folder of files with SQL | Combine multiple CSV files into one table | Download the Query Streams Agent | All File Set Connector guides

Category: File Set Connectors

Tags: parquet to excel, query parquet files, read parquet without spark, parquet folder, parquet in excel, parquet folder connector, sql on parquet, data lake export

Meta Description: Query a folder of Parquet files as one SQL table and pull it into Microsoft Excel. No Spark, no uploads.