Index a File System and Query It with SQL

See everything Query Streams can do beyond this guide Get Started Free

Supports

+more

Live Data

Replace exports with on-demand SQL

Excel Sheets

BI & AI Analytics

Ask Nova, get SQL + charts

Meet Nova

+more

Automation

Scheduled sync to 6+ platforms

Explore

+more

API to SQL

Query APIs with SQL, no code

Explore

Query Streams is a secure, real-time database integration platform, and its File System connector turns a folder tree into a queryable file system inventory you can read from Microsoft Excel, Google Sheets, Airtable and your AI assistant. Learn more at QueryStreams.com and sign up for free to index your first share in minutes.

Every other mode of the File Set connector exists to read what is inside your files. This one exists to read everything around them. It walks a folder tree and records what it finds — names, sizes, timestamps, owners, attributes, folder structure — and it never opens a single file to do it.

That sounds modest until you try to answer an ordinary question about a file server. Storage questions are the ones nobody has good tools for. They get answered with a right-click and a Properties dialog, a PowerShell one-liner somebody pasted from a forum, or a licence for a storage-reporting product that does one thing. A File System connector answers them with SQL instead, from the same query builder and the same spreadsheet add-in you already use for your databases.

The questions a file system inventory answers

These are the six that come up most, and all six are a single query once the folder is indexed.

Four tables, and nothing inside your files is read

A File System connector produces no data tables at all. What it gives you is the four metadata tables that every File Set connector carries, populated properly and treated as the point rather than as housekeeping. If you have read how File Set connectors work, this is the mode where those four tables are the whole product.

Every sweep is a full walk of the tree. There is no cursor and no partial pass: the connector enumerates the roots, builds one consistent snapshot, compares it against what it recorded last time, and serves all four tables from that same snapshot. That is why the numbers in files, directories and volumes always agree with each other — they are three views of one moment, not three separate reads.

Because this mode never parses a file, most of the machinery described in the hub guide simply does not apply here. There is no sniffing, no pinned column shape, and no drift to worry about, because there is no file format to disagree about. The shape of these four tables is fixed.

The four tables in detail

Every column below is real and queryable. Timestamps are UTC and stored to the second, which matters when you compare them: the connector deliberately compares modification times at second precision so a clock that reports sub-second differences does not manufacture a change event on every sweep.

files — the live inventory

A file is identified by the pair root_id and relative_path, because two roots can legitimately contain the same relative path. Paths always use forward slashes, whatever the host operating system.

Two additions are possible on top of that fixed set. If you configure a filename pattern, each token in it becomes a column of its own plus a name_parse_ok flag. If you name folder tokens, each named path segment below the root becomes a column too. Both are covered further down.

One caveat on accessed_utc: last-access tracking is disabled or heavily lazy on many Windows and NAS configurations for performance reasons, so treat it as a hint rather than an audit trail. modified_utc is the reliable one.

files_events — the change log

Every sweep compares what it found against what it recorded last time and writes a row for each difference. The log is append-only, so it answers questions about the past that the current inventory cannot.

A file that is deleted and later restored is treated as the same file rather than a new one. Its first_seen_utc is preserved and its change_count keeps climbing, so the history stays intact instead of resetting every time somebody moves a folder out and back again.

The event log is retention-eligible on event_at, and an event retention setting bounds how long history is kept — 30 days by default, up to ten years if you want a long audit trail. That number is worth deciding early rather than late, because raising it later does not recover events that have already been trimmed.

SELECT event_at,
       event_type,
       relative_path,
       prev_size_bytes,
       size_bytes
FROM   files_events
WHERE  event_at >= CURRENT_TIMESTAMP - INTERVAL 7 DAY
ORDER  BY event_at DESC, relative_path;

directories — the disk usage report

The awkward thing about folder sizes is that “how big is this folder” has two answers, and most tools quietly pick one. Query Streams names both. direct_* counts only the files sitting immediately in that folder; subtree_* counts everything below it as well. Both are real columns, rolled up during the sweep, so a disk usage report never requires a recursive query.

A folder that was not walked still gets a row, and its four count and byte columns are left null rather than filled with zeroes. That distinction is deliberate: zero would claim the folder is empty, and null correctly says the connector does not know. A storage report that quietly reads nought bytes for the one subtree the agent cannot see is worse than one that admits the gap.

SELECT path,
       subtree_file_count                            AS files,
       ROUND(subtree_bytes / 1073741824.0, 2)        AS gb,
       newest_file_utc                               AS last_touched
FROM   directories
WHERE  depth <= 2
  AND  skip_reason IS NULL
ORDER  BY subtree_bytes DESC
LIMIT  25;

The same table, joined to nothing at all, also answers the stale-data question. Sorting files by size within an old modification window gives you a ranked list of archive candidates — the largest things nobody has opened in years, which is exactly the list a storage clean-up needs and exactly the list a Properties dialog cannot produce.

SELECT parent_path,
       name,
       modified_utc,
       owner,
       ROUND(size_bytes / 1048576.0, 1) AS mb
FROM   files
WHERE  modified_utc < CURRENT_TIMESTAMP - INTERVAL 3 YEAR
ORDER  BY size_bytes DESC
LIMIT  200;

volumes — capacity and free space

Each sweep also records the state of the drive behind each root, keyed by root and snapshot time so the rows accumulate into a series rather than overwriting each other.

UNC shares and unmounted roots often do not expose volume statistics. When that happens the row is still written with the capacity and free-space columns null, because “this root reported no volume statistics” is a fact worth having in the data rather than a row silently missing from a report.

SELECT label,
       ROUND(capacity_bytes / 1073741824.0, 1)              AS capacity_gb,
       ROUND(free_bytes     / 1073741824.0, 1)              AS free_gb,
       ROUND(100.0 * free_bytes / capacity_bytes, 1)        AS pct_free
FROM   volumes
WHERE  capacity_bytes IS NOT NULL
  AND  snapshot_at = (SELECT MAX(snapshot_at) FROM volumes)
ORDER  BY pct_free;

Daily growth history

Alongside the four tables there is a fifth, directory_snapshots, which exists purely to answer “is this getting worse?”. It records each folder’s size once a day, on a fixed daily cadence regardless of how often the connector syncs, so a share that syncs hourly does not produce twenty-four times as much history.

Only folders that were actually walked are snapshotted, and folders rather than files are the grain on purpose — there are orders of magnitude fewer of them, which keeps a multi-year series small enough to be free.

SELECT path,
       FIRST(subtree_bytes ORDER BY snapshot_at) AS bytes_then,
       LAST (subtree_bytes ORDER BY snapshot_at) AS bytes_now,
       ROUND((LAST (subtree_bytes ORDER BY snapshot_at)
            - FIRST(subtree_bytes ORDER BY snapshot_at))
             / 1073741824.0, 2)                  AS grew_gb
FROM   directory_snapshots
WHERE  snapshot_at >= CURRENT_TIMESTAMP - INTERVAL 30 DAY
GROUP  BY path
ORDER  BY grew_gb DESC
LIMIT  20;

When the filename is the data

This is the part most people do not expect, and it is often the reason a File System connector earns its place.

Filenames in a business folder are almost never arbitrary. Machines, exports and scanners write names to a convention, and that convention encodes real fields — a date, a region, a machine identifier, a part number, a customer code. Normally that information is trapped in a string, and getting it out means writing string functions in every single query.

Instead you can pin a filename parse pattern on the connector, and each token in it becomes a genuine column on the files table.

The pattern behind that is {report}_{year}_{month}_{region}. Literal text between the placeholders is matched literally, and the pattern is applied to the filename with its extension removed. Each token matches up to the next literal delimiter, except the last one, which is greedy — so a trailing part name or description may itself contain underscores without breaking the parse.

For anything the template form cannot express, prefix the pattern with regex: and write a regular expression with named groups, such as regex:^(?<machine>[^_]+)_(?<stamp>\d{8})_(?<part>.+)$. Named groups become the columns. A raw regex receives the full filename including its extension, so you decide how to handle it. Matching is bounded by a two-second timeout, and a pattern that hits that timeout counts as a failed match rather than hanging the sweep.

Alongside the token columns you get a name_parse_ok boolean. A filename that does not match the pinned pattern is not skipped and does not fail the sync: the file still gets its row with all the normal metadata, the token columns are set to null, and name_parse_ok is false. The violation is also reported in the agent’s log naming the file and the pattern. Finding the files that break your naming convention is then a WHERE NOT name_parse_ok away — which, for a lot of teams, is a naming-convention audit they have never previously been able to run.

Folder structure can be turned into columns the same way. Where a tree is laid out as 2026/08/EMEA/…, naming three folder tokens gives you a column per level: the first names the first path segment below the root, the second names the second, and so on. Segments that turn out to be the filename rather than a folder are left null, so a file sitting higher up the tree than the tokens expect does not produce a nonsense value.

Between them, the two features mean a folder tree with a convention behind it becomes a properly dimensioned table. Grouping a scan archive by machine and month stops being a string-parsing exercise and becomes an ordinary GROUP BY.

Hashing, and finding duplicate files

Duplicate detection by name is close to useless, because the copies are never called the same thing. Contract_final.pdf, Contract_final_v2.pdf and Contract FINAL (Amy's copy).pdf may well be byte-for-byte identical, and no amount of clever name matching will prove it.

Switching hashing on gives each file a SHA-256 digest in the hash column, and duplicates then group by content. Files with the same hash are the same file, whatever they are called and wherever they live.

SELECT COUNT(*)                                  AS copies,
       ROUND(MAX(size_bytes) / 1048576.0, 1)     AS each_mb,
       ROUND((SUM(size_bytes) - MAX(size_bytes))
             / 1048576.0, 1)                     AS wasted_mb,
       STRING_AGG(relative_path, '  |  ')        AS paths
FROM   files
WHERE  hash IS NOT NULL
GROUP  BY hash
HAVING COUNT(*) > 1
ORDER  BY wasted_mb DESC
LIMIT  50;

Hashing has a second use that is less obvious. The ordinary sweep spots a change by comparing size and modification time, which misses the cases where content changed but neither did — a file restored from backup, a copy tool that preserves timestamps, or a document altered by something trying not to be noticed. When a hash changes while size and date do not, the connector records a modified event and marks it detected_by = 'hash', so you can tell which tier caught it.

A sensible pattern on a big share is to narrow the connector with an include pattern first — hash the document folders where duplication actually costs you money, and leave the media archive on metadata alone.

What happens when a file disappears

A file that was in the inventory and is no longer on disk always produces a vanished row in files_events. What happens to its row in the inventory depends on the delete policy.

Tombstones are the right choice when you care about what a deleted file used to be — how big the invoice archive was before somebody tidied it, or what a file’s hash was before it went. Because the files view filters on state = 'present', turning tombstones on cannot skew a report that queries files; you have to ask for files_current to see them.

What gets skipped, and what happens to symlinks

Some folders are noise in every inventory ever taken, so four exclusions are always applied before your own:

• **/node_modules/** — the single largest source of meaningless file count on any developer machine.
• **/.git/** — repository internals, which tell you nothing about your data.
• **/$RECYCLE.BIN/** — deleted files that would otherwise be counted as live.
• **/System Volume Information/** — Windows system state, which the agent usually cannot read anyway.

Your own exclude patterns are added to those rather than replacing them, and include patterns narrow the scan to the files you actually want indexed — set those to **/* for a full inventory, because the wizard seeds them to **/*.csv. Notably, hidden and system files are not skipped: an inventory tool that cannot see them is not an inventory, so exclude patterns are the filter for junk rather than the attribute flags.

Every skipped folder becomes a row in directories with a skip_reason. Excluded folders say excluded, folders beyond the depth limit say max_depth, and a folder the agent’s account is refused becomes an is_accessible = false row with the operating system’s own message. Nothing is silently omitted, so the boundary of what you actually indexed is visible in the data rather than something you have to reconstruct from the configuration.

Symlinks, junctions and other reparse points are not followed by default. That is deliberate cycle protection: a junction that points at its own parent turns a recursive walk into an infinite one, and a link to another share silently doubles the size of the scan. If your tree genuinely relies on links to reach the data, there is a followSymlinks option, but it is off unless you turn it on and worth turning on knowingly.

Scan limits, and what a refusal looks like

A connector pointed at the wrong place could try to index an entire disk array, so the walk has guard rails. The defaults suit a normal share; each one can be raised.

When a scan would exceed one of these it refuses and names the cap it hit, along with the remedies: narrow the roots or the include patterns, lower the depth, or raise the limit. It does not store what it managed to collect. A file system inventory that is quietly missing a third of the share is worse than no inventory, because every report built on it looks plausible and is wrong — a folder that appears to hold 40 GB when it holds 600 GB does not read as an error, it reads as an answer.

The same principle applies to the settings themselves. Leaving a limit blank or setting it to zero means “use the restrictive default”, never “unlimited”; explicit values are clamped to the ceiling. There is no configuration in which a File Set connector will walk without a boundary.

Queries are read-only, and that is enforced rather than promised. A query must be a single statement, and only SELECT, WITH, PRAGMA, DESCRIBE and EXPLAIN are accepted. Since this connector holds nothing but metadata, the strongest thing a query could possibly do is tell you a file’s size.

Every setting on the File System card, and when to change it

The File System card is the shortest configuration form in the File Set family, and the reason is worth pausing on. There is no format section. No table name, no delimiter, no header switch, no encoding, no sheet picker, no flatten depth, no per-file or per-sync row and byte caps, and no manifest overrides. Those fields are not tucked behind a toggle and they are not defaults you could hunt down and change — they are not on the card at all, and the connector strips them from what it saves if anything tries to send them.

That absence is the structural proof of the claim this guide opened with. There is no setting that could make a File System connector open one of your files, because the fields that would configure such a thing do not exist here. The only option that touches file bytes at all is optional hashing, which computes a digest and keeps nothing else.

What follows is every field on the card, in the order the wizard presents it, with its real default and the situation that would make you change it. One of those defaults is actively wrong for a file system inventory, so read the include patterns entry even if you skip the rest.

Source. The six fields that decide which folders are walked and which files are recorded. For an inventory these are the ones that matter most, because between them they define what “everything” means.

Root folders

Required. One folder per line. These are paths as the agent machine sees them — a local path such as D:\finance, or a UNC path such as \\fileserver\finance. A drive letter mapped only on your own desktop will not resolve on the server the agent runs on.

Every file is identified by the pair of its root and its path relative to that root, and that has one consequence worth knowing before you save: change a root later and everything beneath it is re-identified as new. The first_seen_utc column resets, and the change log fills with appeared rows for files that have sat there for years. How you write the root counts. It is normalised to an absolute forward-slash path with no trailing slash, so a stray trailing backslash makes no difference — but a UNC path and a local path to the same folder are two different roots as far as the inventory is concerned. Decide the form of the root once.

Add a second root when related trees live in two places and you want one inventory spanning both — \\nas\finance and \\nas\legal, for instance. The root_id column keeps them apart in every table, and volumes reports capacity and free space per root, so two roots on different drives give you two capacity series rather than one blurred figure.

Scan subfolders

Default: on. Leave it on for an inventory, because the tree is the thing you are trying to inventory. Turn it off only when the roots’ immediate children genuinely are all you care about — a drop folder you watch for arrivals, where the archive subfolder beneath it would multiply the scan and tell you nothing new.

With it off, Max depth vanishes from the form, because there is no recursion left for it to cap. Turn recursion back on and the depth field reappears with its value intact.

Max depth

Default: 8. Required whenever Scan subfolders is on, and the wizard refuses to save without a whole number between 1 and 64. It is a mandatory cap rather than an optional one, which is deliberate: there is no configuration in which this connector walks to an unknown depth.

When to raise it. Eight levels covers an ordinary share comfortably. Deep archive trees are where it bites: a year/month/day/hour layout is four levels before a file appears, and per-customer and per-project folders beneath that will run past eight without anyone noticing. Scanned document archives and engineering project stores are the usual offenders.

What happens if you get it wrong is visible rather than silent, which is the point of the skip_reason column. Every folder the walk stopped at gets its row with skip_reason = 'max_depth', so one query tells you exactly where your inventory ends.

SELECT root_id,
       path,
       depth
FROM   directories
WHERE  skip_reason = 'max_depth'
ORDER  BY root_id, path;

Run that after your first sweep. Rows you care about mean raise the depth; no rows means eight was enough. Lowering it is a legitimate tactic as well — depth 2 or 3 against a sprawling share gives you a fast top-level report — but be honest with yourself about what that report says. The subtree_bytes total on a folder you did walk counts only the files that were indexed, so a shallow scan under-reports folder sizes rather than approximating them.

Include patterns

Glob patterns, one per line, deciding which files get a row. This is the one field on the card whose seeded default is wrong for what you are almost certainly trying to do, and it is worth fixing before you save rather than after.

With that corrected, narrowing the pattern on purpose becomes a useful tool rather than a trap. **/*.pdf gives you a documents-only inventory of a share whose media folders nobody manages, and it is also the cheapest way to make hashing affordable — hash the document folders where duplicate copies cost real money, and leave the video archive on metadata alone. A branch-scoped pattern such as exports/**/* keeps a sibling archive folder out of the file inventory entirely.

One clarification that saves confusion: this field filters files. Folders are still walked and still get their rows in directories. Keeping a folder out of the walk altogether is the job of exclude patterns and Max depth.

Exclude patterns

Four defaults arrive pre-filled, and anything you add is additive rather than a replacement. node_modules, .git, $RECYCLE.BIN and System Volume Information are excluded on every File Set connector whether or not they appear in the box, so clearing the textarea does not bring them back into scope. There is no way to switch them off, and no reason to want to.

What to add, for an inventory specifically. The patterns worth your time are the ones that would otherwise inflate a storage report with things nobody manages as data: a local backup target, a virtual machine disk directory, an application cache folder that regenerates itself. **/~$* is a good one on any share people open Microsoft Excel files from, because those are lock files rather than documents and they come and go constantly — excluding them keeps your change log about real files.

Excluding a folder does not hide it. It still gets a row in directories with skip_reason = 'excluded' and null counts, so the shape of what you deliberately left out stays visible next to what you measured.

Follow symlinks and junctions

Default: off, and that is the cycle protection. Skipping reparse points is what stops a junction pointing at its own parent turning a recursive walk into an endless one.

The scenario for turning it on is a root where the real data genuinely lives behind links — a parent folder that mounts each department’s storage as a junction is the common case, and with links skipped that root indexes as a handful of empty folders. If that is your layout, switch it on knowingly.

The risk is double counting, not just looping. A link to another share brings that share’s files into the inventory under a second path, so the same bytes are counted once under each and your storage report overstates the disk. If you turn it on, check the totals against what the volume itself reports in volumes before you circulate the numbers.

Schema. Two optional fields, both covered in detail earlier in this guide.

Filename parse pattern and Folder token names

Both optional, both blank by default. The first is a single-line field taking either the token template form or a regex: prefixed expression; the second is a textarea taking one token name per line, in path order. The grammars, the collision rules and what happens to a filename that does not match are all covered in When the filename is the data above, and there is no point restating them here.

What is worth adding is why they earn their place on an inventory rather than on a data connector. An inventory of a scan archive that has machine, date and part as real columns can be grouped and trended like any other table, which is the difference between a file list and a report. And name_parse_ok gives you a naming-convention audit for free: the files breaking the convention are a WHERE NOT name_parse_ok away, on a share where nobody has ever been able to produce that list.

Change detection. How often the agent looks, and what counts as a change.

Scan interval

Default: every hour. The choices are every 5 minutes, every 15 minutes, every 30 minutes, every hour, every 6 hours and daily. Match it to how the folder is actually fed, and remember that every sweep is a full walk of the tree rather than an incremental one, so the interval is what you are paying.

Fast end. A drop folder you are watching for arrivals wants 5 minutes, because the question being asked is “has last night’s export landed yet?” and an hourly answer is not an answer. Those folders are small, so the cost is trivial.

Slow end. A 400,000-file archive wants daily. Nothing in it changes hourly, and walking it twenty-four times a day buys you twenty-three identical snapshots and a great deal of disk activity on the file server. If the archive shares a spindle with something people are using, this is the single most considerate setting on the card.

Two consequences of a slower interval, so you can weigh it honestly. The interval is the resolution of your change log, so a file that appeared and vanished between two sweeps is never recorded at all — if catching short-lived files matters, that is an argument for 5 minutes rather than 6 hours. The growth history, on the other hand, does not suffer: directory_snapshots is written on a fixed daily cadence whatever the sync interval, so a daily sweep gives you exactly the same growth trend an hourly one would.

Advanced. The remaining seven fields appear only when Show advanced options is ticked. The defaults are safe, so leaving the toggle alone is a reasonable first move — but four of them are the ones you will come back for.

Hash file contents

Default: off, because it is the one thing on this card that reads file bytes, and that is the expensive tier. It is also the only tier that catches an edit which preserved the file’s timestamp. Both sides of that trade are covered in Hashing, and finding duplicate files above.

How to use it well: leave it off for the first sweep, look at what the inventory tells you, then switch it on with the include pattern narrowed to the folders where duplicate copies actually cost you money. Turning it on across a multi-terabyte media share to find duplicates in the contracts folder is the mistake to avoid, and it is entirely avoidable.

Max file size to hash (bytes)

Default: 33,554,432 bytes — 32 MB. The field appears only once hashing is on, and it is expressed in bytes rather than megabytes, so type the number: 33554432 for 32 MB, 268435456 for 256 MB, up to a ceiling of 1 GB.

Files larger than this are never hashed. They keep their full metadata row with a null hash and fall back to size-and-date change detection, so nothing breaks — but they are also invisible to your duplicate query, which is the failure mode to watch for. If the duplication you are chasing is in design files, video or virtual machine images, 32 MB means hashing is switched on and doing nothing for the files you care about. One query after a hashing sweep tells you how many files the bound left out:

SELECT COUNT(*)                                   AS unhashed,
       ROUND(SUM(size_bytes) / 1073741824.0, 1)   AS unhashed_gb,
       ROUND(MAX(size_bytes) / 1048576.0, 1)      AS largest_mb
FROM   files
WHERE  hash IS NULL;

When a file vanishes

Default: Drop its rows. The alternative is Keep rows, mark deleted (tombstone). What each does to the inventory is set out in What happens when a file disappears above.

The inventory-specific reason to choose tombstones is that they answer “what disappeared, and what was it?” from the inventory itself rather than from the change log — with the file’s last known size, dates and hash still attached. That matters more than it first looks, because the change log is trimmed by the retention setting below and tombstoned rows are not. If your reason for indexing the share is that things go missing from it, tombstones are the durable record and the event log is the recent one.

Event log retention (days)

Default: 30 days, up to a ceiling of 3,650 — ten years. This is the field most worth thinking about before your first sweep rather than after, and the reason is blunt: retention bounds how far back your history reaches. Thirty days of retention means thirty days of history, and no query can recover what has already been trimmed.

That applies directly to the growth history described earlier. If you want a year of trend for the “which folders grew most” report — the report that turns a storage budget conversation from an argument into a chart — set this to 365 now. Setting it in eleven months’ time gives you a year of retention and one month of data. History only accumulates from the point at which the window is wide enough to keep it.

The cost of a wide window is small. Event rows are narrow, and the growth series is recorded per folder per day rather than per file, so a multi-year retention on a share with a few thousand folders is measured in megabytes. Lower it below 30 only if you genuinely never look at what changed.

Scan parallelism per root

Default: 2. Range 1 to 16. The wizard’s own guidance is the right summary: parallel scanners help on SSD and NAS, and hurt on spinning disk. Solid-state storage and a NAS with a real controller behind it can serve several directory enumerations at once; a single mechanical drive cannot, and concurrent walkers turn a mostly sequential read into seek thrash that makes the sweep slower rather than faster.

How to set it. An SSD-backed share with hundreds of thousands of small files in many folders is the case that benefits — try 4, measure the sweep, try 8. An archive on a spinning disk or a slow remote share should go to 1. The setting is per root, so four roots at parallelism 4 means sixteen concurrent walkers hitting your storage; if the roots share a device, count them together rather than per root.

It interacts with the field below. Raising parallelism is the usual way to bring a large tree inside the scan time budget without narrowing the scope; dropping it to 1 on the wrong hardware can be what pushed the sweep over the budget in the first place.

Scan time budget (seconds)

Default: 300 seconds, up to 3,600 — one hour. A sweep that exceeds the budget stops and reports rather than running unbounded. That is the guarantee this field exists to provide: no configuration of this connector produces a walk that runs for as long as it takes on a share that turned out to be far bigger than anyone thought.

When to raise it. The first full walk of a large tree is the expensive one, and a cold NAS or a share reached over a WAN link is slower per folder than local storage by an order of magnitude. If your first sweep reports the time budget, you have three honest choices: raise the budget, raise parallelism if the storage will take it, or narrow the roots and the include patterns. Raising the budget is the correct answer when the share really is that large; the others are the correct answer when the scope was wider than you meant.

Blank means the default. Zero is refused, as it is on every cap here.

Max files per scan

Default: 100,000, raisable to an agent ceiling of 2,000,000. Blank means the default, and 0 is refused outright — it never means “unlimited”. No value you can type here removes the boundary.

Exceeding it is a refusal, not a truncation, and the reasoning is set out in Scan limits, and what a refusal looks like above. In practice this is the cap most people meet first, because 100,000 files is a normal departmental share and a large one goes well past it. A 400,000-file archive needs this raised before the first sweep will complete, and the connector will tell you so by name rather than quietly indexing the first hundred thousand files it happened to encounter.

It has a second, quieter use: it is a tripwire against a mistyped root. A connector aimed at D:\ instead of D:\finance hits the cap and refuses, which is a considerably better outcome than an overnight walk of an entire disk array.

What is not on this card, and why

Reading the other guides in this family, you will see settings that have no equivalent here. Their absence is the design rather than a gap:

• No Table name and no Table layout — there is no data table to name, and no column shape that could vary between files. The four metadata tables have fixed names and a fixed shape.
• No delimiter, quote, header, encoding, null strings or date format — every one of those describes how to interpret the text inside a file.
• No sheet selection, member table selection, record tag column or flatten depth — all of them pick which part of a file’s contents becomes a table.
• No rows or bytes caps, per file or per sync — those bound how much file content is ingested, and no content is ingested here. The caps on this card bound the walk instead: files, depth and time.
• No manifest overrides — renaming, retyping and excluding columns applies to a sampled shape that had to be guessed. Nothing here is guessed, so there is nothing to correct.

If you want both an inventory of a folder and the data inside its files, that is two connectors pointed at the same root — a File System connector for the metadata and a format connector for the contents. They coexist happily, they do not interfere with each other’s sweeps, and the two sets of tables join on the file path. How File Set connectors work covers the format side.

Setting it up

You need a Query Streams account and the Network Agent installed on a machine that can see the folder — a file server, a workstation with the share mounted, or your own laptop. File Set connectors require Network Agent 2.6 or newer.

1. Install the agent from the download page if it is not already running.
2. In the portal, add a data connector and choose File System.
3. Enter one or more root paths as the agent’s machine sees them.
4. Change the include pattern from **/*.csv to **/*. The seeded default indexes CSV files only, which is not an inventory. Add extra exclude patterns for anything you know is noise.
5. Check Max depth against your tree. Eight levels is the default; a deep dated or per-project archive needs more.
6. Leave hashing off for the first run. Index the tree, look at what you have, then switch hashing on for the folders where duplicates matter.
7. Add a filename parse pattern if the folder follows a naming convention you want as columns.
8. Save. The agent walks the tree and reports how many files and folders it found, along with anything it could not read.
9. Open the query builder, pick the connector and query files, directories or volumes — or ask Nova AI to write the query for you.

Once it exists, the inventory is a data source like any other. It can feed a Microsoft Excel or Google Sheets add-in, a scheduled sync into Airtable or Smartsheet, Nova AI, an AI assistant through the Query Streams MCP server, or a REST endpoint — and it joins to your other connectors. A file audit becomes considerably more interesting when the scan archive on the share can be joined to the job records in SQL Server that were supposed to produce it, and the gap between the two is a single query.

Frequently Asked Questions

Get Started

Related guides: How to query a folder of files with SQL | Combine multiple CSV files into one table | Download the Query Streams Agent | All File Set Connector guides

Category: File Set Connectors

Tags: file system inventory, query file metadata with sql, disk usage report, find duplicate files, file audit, storage report, file set connector

Meta Description: Index a file system with SQL: disk usage, duplicates, stale files and growth. Metadata only.