Any text log

Turn Custom Logs into queryable tables.

Some logs are written by software only your company runs. Say where a record starts, which grammar it uses, where the timestamp is and what the columns are called — once — and it is a table like any other.

Use the results in Excel, Google Sheets, AI tools and your apps through the Database REST API.

Every connector type is on Free. Start with 2 data sources and 1 agent.

Connect Custom Logs freeConnection details
Custom LogsFiles read where they live
Inside your networkQuery Streams Network Agent

Read, organize and query on your machine.

Read-only queries · outbound connection
ExcelGoogle SheetsYour AIYour apps

Source files stay in place. Your tools receive query results.

Source → agent → allowed results

Query Streams reads any text log, including the ones written by software only your company runs, and turns the folder into a live SQL table readable from Microsoft Excel, Google Sheets, AI assistants over MCP, the REST API and Power BI. You describe the format once - record boundary, grammar, timestamp, columns and rotation - and the agent holds that description steady from then on. Six grammars are available: delimited, key = value, grok, JSON lines, a #Fields header directive, and fixed width. Lines the description does not match are kept as raw rows rather than dropped, so an unanticipated format costs you nothing.

  • Six grammars, described once
  • Unmatched lines kept, never dropped
  • Nothing sent away to be interpreted

Connect Custom Logs with the right settings.

Keep the connection on the Network Agent. These are the details specific to Custom Logs.

Folder
Any folder of text logs - local disk or a network share typed as \\server\share (never a mapped drive letter; on Linux or a Mac, mounted first); subfolders included, include and exclude patterns supported
Grammars
Delimited, key = value, grok, JSON lines, a #Fields header directive, and fixed width
Record boundary
Yours to set - one line per record, or a pattern, a fixed count, a blank line or a sentinel, so multi-line records stay whole
Timestamps
Position and format are described; where the line omits the year it can be taken from the file's modified time or its filename
Rotation
Ten named rotation patterns plus plain unrotated files; compressed archives are read in place
Unmatched lines
Kept verbatim in events_raw_lines, never dropped
Credentials
None - there is no server; the agent reads the files in place

This is the only log profile you can edit, and that is the point: every other one is pinned by us so it cannot drift. The trade is that a custom description cannot be proven correct in advance the way a shipped profile can - what stands in for that is the preview, which runs your description against your own files and shows you the rows before anything is committed.

Built for how Custom Logs works.

The five parts of a format

A locked profile pins these for you. Custom exposes all five, and they are the whole description - there is nothing else to configure and nothing hidden.

  • Record boundary - where one record ends and the next begins, which is what keeps a multi-line stack trace in a single row.
  • Grammar - delimited, key = value, grok, JSON lines, a #Fields header directive, or fixed width.
  • Timestamp - where it is and how it is written, including where the year comes from when the line omits it.
  • Columns - what each field is called and what type it is, so numbers sort as numbers and dates filter as dates.
  • Growth model - how the file grows and rotates, so yesterday's archive and today's live file are one table.
-- Whatever you called the columns, this is just SQL
SELECT   file_name, severity, COUNT(*) AS lines
FROM     fileset.events
WHERE    severity IN ('WARN','ERROR')
GROUP BY file_name, severity
ORDER BY lines DESC

Try the built-ins first

Test connection matches what it finds against the profiles that ship, and offers the closest one. That is fingerprint matching, not a model reading your logs - nothing about your files is sent anywhere to be interpreted. If a built-in fits, use it: a pinned profile is less to maintain than a description of your own. Describe it yourself when nothing fits, and ask us for a built-in if you would rather have one.

The formats and behavior, in detail.

Open the reference you need for Custom Logs.

If it is a text file, it can be a table.
The big idea

If it is a text file, it can be a table.

Three files in three unrelated shapes — one written by a line-of-business app, one by a batch job, one by a machine nobody has supported since 2011. Describe each one once and the columns you named are the columns you query. No collector to run, no index to maintain, no second copy of the data.

Log folderyour files
plant-audit.logkey=value
batch-2026-09-10.txtfixed width
robot.tracedelimited

Rotations, compressed archives and re-read files — handled

what-went-wrong.sqlSQL
SELECT file_name,
       severity,
       count(*) AS lines
FROM   fileset.events
WHERE  severity IN ('WARN','ERROR')
GROUP BY file_name, severity
ORDER BY lines DESC
Resultyour columns
file_nameseveritylines
plant-audit.logERROR268
batch-2026-09-10WARN154
robot.traceERROR61
plant-audit.logWARN9

There is a severity column because you said there was one

Illustrative figures — the shape of the answer, not a benchmark

Six ways to say what a line means
Grammars

Six ways to say what a line means

Nearly every text log ever written is one of these six shapes. Find yours below, and the sample line will usually settle it faster than reading the description.

Grammars you can choose

6 grammars
DelimitedCSV, TSV, pipes, anything
04:11:07,web-01,ERROR,disk full
  • Name the separator — comma, tab, semicolon, pipe, whatever the file uses
  • Quoted fields are respected, so a comma inside a message stays inside it
  • A header row can name the columns for you instead of typing them
  • Decimal separator is yours to set, for the files where it is a comma
Key = valueself-describing pairs
ts=04:11:07 host=web-01 level=error
  • The keys in the line become the columns — you do not restate them
  • Both separators are yours: what sits between key and value, and between pairs
  • A prefix in front of the pairs — a timestamp, a tag — can be skipped
  • Common in application and machine logs that were written to be parsed
Grok patternsfor the awkward ones
%{TIME:ts} %{LOGLEVEL:lvl} %{DATA:msg}
  • Named captures become columns — the escape hatch when nothing else fits
  • The pattern libraries behind the built-in profiles are there to borrow from
  • Several patterns can be tried in order, for a file with more than one line shape
  • The starting default — one message column, then you narrow it
JSON linesone object per line
{"ts":"04:11:07","level":"error"}
  • Fields become columns, and you choose which ones you actually want
  • A nested object is kept whole as JSON rather than flattened or dropped
  • A container runtime’s wrapper can be unwrapped so you query the inner record
  • What most modern logging libraries write when told to be structured
Header directivethe file names its own columns
#Fields: date time s-ip cs-method
  • A line near the top declares the fields, and the columns follow it
  • The W3C extended format, and anything else that borrowed the idea
  • You name the directive, so it does not have to be spelled #Fields
  • Nothing to keep in sync — the file is the source of the column list
Fixed widthcolumns by position
20260910041107 web-01 ERROR
  • Give each column a name, a start and a length — no separator needed
  • The right answer for files where a field can contain the delimiter
  • Still what mainframes, controllers and a lot of industrial kit write
  • Padding is trimmed, so a value does not arrive wrapped in spaces
Ask for a built-inif it is a format others have too

Describing it yourself is the fast route. If it is a format plenty of people run, we would rather ship it as a built-in profile that the Agent pins for everyone — the connector links straight to a request, and it arrives with an Agent update.

Worth being straight about what backs this one. The built-in profiles on the other log pages were each built and tested against real captured files, and we say so there. This preset cannot be — the format is the one you describe, so there is no fixed shape for us to have proven in advance. What stands in for that is the preview: Test connection runs your description against your own files and shows you what parsed and what did not, before you commit to anything.
Describe it once, use it everywhere you run
Every site

Describe it once, use it everywhere you run

A description is part of the connector, not part of a machine. The same five parts that read the audit file at head office read it at the plant and in a second region — and each site runs a Network Agent that reads the files where they are and dials out to Query Streams. As far as your firewall is concerned that is an ordinary outbound connection, so there is nothing to open and no VPN to build.

Head office A line-of-business app writing its own audit file dials out
Plant floor A controller writing fixed-width batch records dials out
eu-west The same app, the same description, a second folder dials out

Three outbound connections, one place to query them — no inbound port, no VPN, no firewall change

You name the identity

Every row is labelled with where it came from. By default that is the file name, and if the machine or site is a field in the line — or a piece of the filename — you can point the label at that instead, and group by it.

file name · a column · part of the filename

Nothing to open

The Agent makes one encrypted outbound connection, and the request out and the rows back both travel over it. No inbound port, no VPN, no firewall change to go and ask for — and no collector daemon added to a server that is already running one.

one connection, both directions

One query across regions

Every source in a federated query names its own Agent, so one statement can put a bespoke log beside the database that explains it — the batch file next to the order it belongs to. Included on Business and Enterprise.

1 log + 1 database → 1 result set

Written once, not per site

The work of describing a format is a one-off. A second folder in the same shape is a second connector with the same description, so rolling it out to another location is a setup step rather than another investigation.

1 description → every folder like it

The four questions a text file will not answer
The hard part

The four questions a text file will not answer

Parsing the line is the easy half. These are the things a log file does not tell you about itself, and each of them is something you state rather than something we guess.

Where a record starts and ends

A stack trace is one event across twenty lines, and reading it as twenty events makes the numbers wrong. Six ways to mark the boundary: one line, a pattern that opens a record, a blank line, an opening and closing marker, a continuation rule, or a fixed count.

1 record is not always 1 line

The encoding has no default

This is the one field with nothing filled in, on purpose. Assuming UTF-8 on a Latin-1 file does not fail — it quietly mangles every accented character and you find out months later. So it is asked, once, before the first sync, rather than guessed.

chosen, never assumed

The year, when the line has none

Plenty of formats print a month and a day and stop. Three ways to settle it: the year is in the line, it comes from the file’s own modification time, or it is read out of the filename — which is where a dated archive usually keeps it.

in the line · from the file · from the name

Nothing counted twice

Each file is fingerprinted by its first 4 KB, so a renamed or rotated file is recognised rather than read again from the top. Ten rotation patterns are recognised by name — plus none, for files that simply grow.

4 KB prefix hash · 20,000-record window

Five parts, and then it is a table
The description

Five parts, and then it is a table

This is the whole of what you fill in. Every built-in profile on the other log pages is these same five parts, filled in by us and pinned by the Agent — here they are yours.

Record boundarywhat counts as one event
one lineopening patternblank linestart and end markerscontinuation rulefixed count
Grammarhow a record splits up
delimitedkey=valuegrokJSON linesheader directivefixed width
Timestampwhich part is the time
fieldformattimezoneyear modecomposed from two fieldsvalues that mean null
Columnsnames and types
stringintbigintdoublebooltimestamptimestamptzdatejson
Growthhow the files change
appendsrotation patterncompressed suffixesring of filesmaximum line length
The defaults are deliberately modest, and that is the design: one message column per line, and the event time taken from the file’s modification time. That is enough for a first sync to succeed on almost anything — then you refine against the preview until the columns are the ones you wanted. Every row also carries the file it came from, and everything lands in one table, fileset.events.
What this deliberately does not do

What this deliberately does not do

Say the biggest one first: try the built-in profiles before you reach for this. Run Test connection and the Agent offers the closest built-in it recognises — and a built-in is pinned by the Agent and was tested against real files, which a description you wrote this afternoon has not been. This preset is the escape hatch for the formats nobody else runs, not the default starting point.

Nothing reads your logs to work the format out for you. That offer of a closest match is your file’s shape compared against the shipped profiles — pattern matching against known formats, not a model reading your data. If none of them match, the description is yours to write, and the preview is how you check it.

It is not a live tail. Query Streams reads the files as they are on disk when you run the query, so a failing job shows up as rows rather than as a stream you watch. If you need sub-second alerting on a line as it is printed, a log pipeline is the right tool for that job.

It reads text, and only text. Archives and binary databases in the folder — .zip, .cab, a raw journal file — are left alone rather than opened and guessed at. Compressed rotations of a text log are read; a container full of something else is not.

And the description can be wrong, which is why nothing is thrown away. A line the grammar does not match is kept as a raw row with a reason attached, so a mistake shows up as something you can see and query rather than as a silent gap. Fix the description and re-sync. The rest follows the usual design: the Agent connects outbound and reads files, nothing is installed on the server, nothing is written back, and access is read-only.

From connection to useful results.

  1. Install the Network Agent where it can see the folder, and point the connector at it.
  2. Run Test connection first. The agent fingerprints what it finds and offers the closest built-in profile - if one fits, take it and you are done describing anything.
  3. If none fits, describe the format yourself: where a record starts, which of the six grammars it uses, where the timestamp is and what to call the columns.
  4. Choose the rotation pattern and, for formats whose timestamps omit the year, where the year should come from. That description is then pinned.
  5. Query it. The table behaves exactly like every other log connector's, because it is the same driver reading a format you supplied instead of one we shipped.

Put your Custom Logs results to work.

One connection serves your spreadsheets, apps and AI tools. Share permission to run a saved query; recipients see the allowed results, with the SQL and source login hidden.

For your team, vendors and partners

Ask Nova about Custom Logs

Ask a question in plain English. Nova already knows your Custom Logs tables, writes and runs the SQL, explains the answer and charts it — no query to write, no credentials to hand over.

How Nova works

Custom Logs to Data Viewer

Share a live Custom Logs result with anyone — a colleague, a vendor, a partner — as a bookmarkable grid that re-runs fresh when opened, on a phone, a tablet or a desktop. They see the results, never the SQL or the files.

How the Data Viewer works

For apps and systems

Custom Logs REST API

Publish a Custom Logs query as an authenticated JSON endpoint any application can call, with an OpenAPI 3.1 spec and ready-made Postman, Insomnia and Hoppscotch collections. No database port is opened.

Explore REST API

Custom Logs to Power BI

Paste the generated Power Query M into the Power BI Advanced Editor and the report reads live Custom Logs results over HTTPS — no ODBC driver, no database port opened.

Explore Power BI

For AI tools

Delivered to people

Join Custom Logs to the rest of your data.

Bring databases, business APIs, files and logs into one read-only query. Every connector supports Federated Queries.

Name the connections and tables. Query Streams handles the source queries and combines the results.

Explore Federated Queries
Illustrative cross-source query
-- nothing copied, nothing merged, nothing scheduled
SELECT   c.region, COUNT(*) AS orders, SUM(i.amount_due) AS invoiced
FROM     customlogs.fileset.orders1 f
JOIN     pg_crm.public.customers2   c ON c.id = f.customer_id
JOIN     billing.stripe.invoices3   i ON i.customer = c.stripe_id
GROUP BY c.region
ORDER BY invoiced DESC;

Use your own connection, schema, table and column names.

Shared across the connector family.

Every log connector also gives you

  • events — one row per parsed line, with the timestamp, the fields the grammar named, and _source_file naming the file it came from.
  • events_raw_lines — lines the grammar did not match, kept verbatim rather than dropped, so a format you did not anticipate is a question you can still answer.
  • files_current and files_events — the live inventory and the audit trail: what appeared, what rotated, what vanished, and anything held back, with the filename and the reason.
  • Rotation is understood, not worked around — ten named rotation patterns plus plain unrotated files, so yesterday's log and its compressed archive are the same table as today's.

Good to know

  • Read-only, enforced — one statement at a time, SELECT and friends only. Your logs are never written, moved, rotated or truncated by us.
  • Nothing is uploaded — lines are parsed and cached, encrypted, on your own machine beside the agent. Only the result of a query leaves your network.
  • Only new bytes are read — the agent resumes where it left off in each file rather than re-reading it, so a busy log does not cost more every day.
  • The SQL dialect is DuckDB — the same SQL you would write against any other connector.

Questions about Custom Logs

Where can I use Custom Logs query results?

Use Custom Logs results in Excel, Google Sheets, the Data Viewer, your apps through the Database REST API, and AI tools through MCP or Nova. Saved results also feed supported Automation destinations, reports and alerts. Destination availability and allowances depend on your plan.

Do I need to open an inbound port for Custom Logs?

No. Install the Network Agent on a machine that can reach Custom Logs. The agent connects outward to Query Streams; no inbound port or VPN is needed. Your source must still allow the agent machine to connect under its own access rules.

Are my Custom Logs files uploaded?

The Network Agent reads files where they are and keeps its cache on the agent machine. Source files are not uploaded, moved or rewritten. Only requested query results leave your network over the outbound encrypted connection.

What do I need to connect Custom Logs?

Folder: Any folder of text logs - local disk or a network share typed as \\server\share (never a mapped drive letter; on Linux or a Mac, mounted first); subfolders included, include and exclude patterns supported. Grammars: Delimited, key = value, grok, JSON lines, a #Fields header directive, and fixed width. Record boundary: Yours to set - one line per record, or a pattern, a fixed count, a blank line or a sentinel, so multi-line records stay whole. Timestamps: Position and format are described; where the line omits the year it can be taken from the file's modified time or its filename

Can I join Custom Logs to other connected sources?

Yes. Federated Queries support every connector type, including databases, business APIs, files and logs. Name the connections and tables in one read-only query. Query Streams translates the source portions as needed and combines their results. See how Federated Queries work.

When should I use this instead of a built-in profile?

When nothing else fits. The built-in profiles cover machine controllers, web servers, containers, application frameworks and system logs, and each one was written and tested against real files — so if your log is one of those, picking it is both less work and better tested than describing it yourself.

Run Test connection before you decide. The Agent compares the shape of your file against the shipped profiles and offers the closest one it recognises, so the answer comes from your actual files rather than from a guess. Reach for a description when that comes back with nothing.

Do I need to know regular expressions?

Usually not. Five of the six grammars need no pattern at all: a delimited file needs a separator, a key=value file describes itself, JSON lines describe themselves, a header directive names its own columns, and fixed width needs a start and a length per column.

Grok is the sixth, and it is the escape hatch for genuinely awkward lines. Even then you are mostly assembling named pieces — a timestamp, a level, the rest — and the pattern libraries behind the built-in profiles are there to borrow from rather than starting on a blank line.

Do I have to get the description right the first time?

No, and the defaults are set so you do not have to. Out of the box a first sync succeeds on almost any text file: one message column per line, with the event time taken from the file’s modification time. That gets you a working table you can actually look at.

From there you refine. The preview shows what your description does to your own lines, so you can pull a severity or a job number out into its own column and see the result before you commit. Change it later and re-sync — nothing was thrown away in the meantime.

My events span several lines. Can it handle a stack trace?

Yes, and this is the setting people most often need. A record does not have to be a line: you can say that a record starts whenever a line matches a pattern — a timestamp at the left margin is the usual one — and everything after it belongs to that record until the next match.

There are five other ways to draw the boundary if that is not the shape: a blank line between records, an opening and closing marker, a rule for what counts as a continuation line, or a fixed number of lines. A twenty-line exception then arrives as one row with the whole trace in it.

What if the lines have no timestamp at all?

That is the default case, and it works. When no field is named as the time, the event time comes from the file’s modification time — so the rows still sort and still respect a retention window, they are just accurate to the file rather than to the line.

If the time is in the line but split across two fields — a date column and a time column, which is common in fixed-width and delimited files — you can name both and have them composed into one timestamp.

Which encoding should I choose?

UTF-8 is the right answer for most files written this decade. It is the one field with no default, and that is deliberate: assuming UTF-8 on a Latin-1 file does not throw an error, it silently mangles every accented character, and a silent corruption is much worse than a question asked once.

If you are not sure, the preview will tell you — a wrong encoding shows up immediately as mojibake in the sample rows. Older Windows software and industrial kit are the usual reasons the answer is not UTF-8.

Can you just add my format as a built-in instead?

Ask us. If it is a format a reasonable number of people run — a commercial product, a common framework, a piece of industrial kit — we would much rather ship it as a built-in profile than have every customer describe the same thing. The connector links straight to a request.

A built-in is better for you as well as for us: it is pinned by the Agent, tested against real captures, and it arrives on your servers with an Agent update rather than needing anyone to touch a connector.

What happens to lines that do not match?

They are kept, not dropped. A line the grammar does not match is retained as a raw row with a reason attached, so an odd entry, a format that changed under you, or a description that is not quite right yet shows up as something you can see and query rather than as a silent gap in your numbers.

That is the safety net that makes refining a description reasonable. You are never choosing between a strict parser that throws data away and a loose one that gives you a wall of text — the parsed rows are typed, and whatever did not parse is still there beside them.

Put Custom Logs where the work happens.

Start with Free: every connector type, 2 data sources and 1 Network Agent. Add plan allowances as your needs grow.

Get the free planSee plan allowances