syslog & journald New

Connect System Logs to Excel, Sheets and AI

Q: Which format do I have?

Look at the start of a line in /var/log/syslog or /var/log/messages. If it begins with a three-letter month - Sep 10 04:11:07 - that is RFC 3164, and it is what most Linux servers still write by default. If it begins with a number in angle brackets followed by a full ISO 8601 timestamp, that is RFC 5424. You do not have to be sure before you start. Test connection reports which profile it recognised and how much of the file matched, so the answer comes from your actual files rather than from a guess.

Every server already writes this down — query all of them at once. Syslog in either RFC, and the systemd journal once you have exported it. Point Query Streams at the folder and every line becomes a row — carrying the host it came from, the program that wrote it and the process id — so a question that used to mean four SSH sessions and grep becomes one statement.

Start free 11 destinations

1connection

0inbound ports

read-onlyenforced

3 system log formats RFC 3164 · RFC 5424 · journald Every row carries its host Read-only · nothing installed on the server

The big idea

A folder of system logs becomes one table.

Query Streams reads the files rsyslog, syslog-ng or systemd already wrote, pins the format, and gives you typed columns with the host named beside every row. No collector to run, no index to maintain, no second copy of the data — and a line the parser cannot read is kept as a row with a reason rather than silently dropped.

Log folderon the server

syslogRFC 3164

auth.log.2.gzrotated

journal.exportjournald

Rotations, compressed archives and re-read files — handled

noisy-programs.sqlSQL

SELECT host,
       program,
       count(*) AS lines
FROM   fileset.events
WHERE  message LIKE '%error%'
GROUP BY host, program
ORDER BY lines DESC

Resulterrors by host

hostprogramlines

db-01postgres412

web-01sshd288

app-01systemd96

mail-01cron11

Four servers in one result, because host is a column

Illustrative figures — the shape of the answer, not a benchmark

Sources

Two RFCs and the journal

Each card names the exact files Query Streams reads and where they live. You do not reconfigure rsyslog to make this work — whichever format your server already writes is the one to pick.

Supported system logs

3 formats

SyslogRFC 3164, the BSD file format

The default on most Linux servers — Mon DD HH:MM:SS host program[pid]: message
Host, program and process id become their own columns
/var/log/syslog, auth.log, kern.log — messages on RHEL
The periodic -- MARK -- filler is dropped by declaration, not by a filter you write
Appends; dated and numbered rotations and .gz followed

SyslogRFC 5424, the structured one

The modern format — a real ISO 8601 stamp, so no year to guess
Priority, app, process id and message id each get a column
The structured-data block is kept whole rather than flattened away
The RFC’s nil value is read as null, not as the text -
Built to the RFC, not yet proven against a real capture

systemd journaljournalctl -o export

Files you made with journalctl -o export
Microsecond timestamps, so ordering survives a busy second
The unit and the transport come through as columns, not as text
Every other journal field is kept in data rather than discarded
One entry per record, however many lines it spans

More comingthese are the start, not the ceiling

New system log formats are added with each Agent release. Because Agents keep themselves up to date, a format added later shows up in your connector without anyone touching a server.

Worth knowing which of these has met a real file. The RFC 3164 parser and the journal export parser were both built and tested against real captured logs, rotations and compressed archives included. RFC 5424 has not been — it is written to the specification and covered by tests, but no real capture has come through it yet. If that is your format, Test connection will tell you what it recognised before you commit to anything, and we would rather you knew now.

Every server

One query, every machine you run

System logs are the case this was built for. Every row carries the host that wrote it, so a folder collected from four servers already answers questions across all four — and each site you add runs a Network Agent that reads the files where they are and dials out to Query Streams. As far as your firewall is concerned that is an ordinary outbound connection, so there is nothing to open and no VPN to build.

Head office A rack of Linux servers writing plain /var/log/syslog dials out

eu-west Cloud instances, journal exports written on a schedule dials out

us-east A second region, the same files in the same shape dials out

Three outbound connections, one place to query them — no inbound port, no VPN, no firewall change

One Agent per location

An Agent covers a whole site rather than a single server, and a folder tree collected from many hosts is one connector — every row already carries its host. The free tier runs a single Agent and higher tiers run more.

1 site = 1 Agent = many hosts

Nothing to open

The Agent makes one encrypted outbound connection, and the request out and the rows back both travel over it. No inbound port, no VPN, no firewall change to go and ask for — and no collector daemon added to a server that is already running one.

one connection, both directions

One query across regions

Every source in a federated query names its own Agent, so a single statement can put one region’s auth log beside another’s — or beside the application database that says which of those logins mattered. Included on Business and Enterprise.

2 regions → 1 result set

They keep themselves current

Agents update themselves, so a log format added in a later release reaches every server without anyone doing a round of SSH. That is what makes the wall above a starting point rather than a fixed list.

new formats arrive with the Agent

The hard part

The year, the timezone and the bytes

Syslog is one of the oldest formats still in daily use, and it shows. Four things routinely go wrong when you try to treat one as data, and each is handled here by declaration rather than by guesswork.

The line has no year

An RFC 3164 stamp is Sep 10 04:11:07 — month, day, time, and nothing else. Read naively that puts every archived line in the current year. Query Streams infers the year from the file’s own modification time instead, so a rotation from December still lands in December.

Sep 10 04:11:07 → year from the file

And no timezone either

The old format carries no offset, so a stamp is only meaningful next to the machine that wrote it. It is read as the Agent’s local time and the raw text is kept beside it. RFC 5424 and the journal both carry real UTC, and are treated as such.

RFC 3164 local · RFC 5424 and journald UTC

Binary values in a text file

The journal export format writes any value containing a newline as a name, a byte length, then the raw bytes. Query Streams reads those by their declared length rather than by looking for the end — so a message with a newline in it stays one entry instead of becoming several.

read by length, never by content

Nothing counted twice

Each file is fingerprinted by its first 4 KB, so a rotated file is recognised as the same file rather than read again from the top. A rolling duplicate window catches the overlap when a writer replays its tail.

4 KB prefix hash · 20,000-record window

The schema

The columns you actually get

Not a blob of text with a timestamp beside it. Each format is parsed into typed columns you can filter, group and aggregate on directly — and the newer the format, the more of them there are.

Syslog, RFC 3164BSD file format

programpidmessage

Syslog, RFC 5424structured syslog

priappprocidmsgidsdmessage

systemd journal exportjournalctl -o export

programpidpriorityunittransportmessagedata

Every row also carries event_time, the raw stamp it was read from, and the file it came from. All three formats share one identity column — host — which is what lets a single query span every server in the folder: the two syslog formats take it from the line, the journal from its own _HOSTNAME field. Everything lands in one table, fileset.events.

Power Query M

Paste the generated Power Query M into the Power BI Advanced Editor and the report reads live system log results over HTTPS — no ODBC driver, no database port opened.

How Power BI works

Supported

System logs alerts and reports

Slack · Discord · Email · Webhook

Put a system log query on a schedule and have the rows delivered to Slack, Discord, email or a signed webhook — or hold the message until a row count, threshold or percentage change crosses the line you set.

How alerts and reports work

The alerts card is the one worth a second look for system logs: a scheduled query against failed logins or disk errors needs nobody watching a terminal, so the thing you would have found on Monday reaches Slack on Friday night. Log-specific step-by-step guides are not written yet — the linked pages cover how each surface works.

What this deliberately does not do

Say the biggest one first: system logs contain personal data. auth.log records usernames and the IP addresses they connected from; a program that logs an email address puts it in the message text. Whatever is in the file is what lands in the column — there is no header map to exclude fields the way there is for a web server log. Treat the resulting table with exactly the care you would treat the file, and think about who you share a query with.

It is not a live tail. Query Streams reads the files as they are on disk when you run the query, so a failing service shows up as rows rather than as a stream you watch. If you need sub-second alerting on a line as it is printed, a log pipeline is the right tool for that job.

We do not read the binary journal. The systemd profile reads files written by journalctl -o export, not /var/log/journal directly — so exporting is a step you schedule, not something that happens by itself.

Three things are written to the specification rather than proven against a real file, and they are worth naming: the whole RFC 5424 profile, the ISO-8601 stamp variant that rsyslog and syslog-ng can write instead of the classic one, and the handling of a journal value that spans several lines. All three are covered by tests. None has met a real capture yet.

Finally, the family is called System logs but today it means Linux and Unix. There is no Windows Event Log profile and no macOS log show profile, and we would rather say that than let the name imply them. The rest follows the usual design: the Agent connects outbound and reads files, nothing is installed on the server, nothing is written back, and access is read-only. The retention field starts at 60 days for this preset and is yours to change.

How it works

Three steps, and nothing gets installed on the server.

Point at the log folder

Give a Query Streams Agent read access to where the logs already land — /var/log on the server itself, or the share you already collect them onto from several machines.

It recognises the format

Query Streams identifies the format from the content of the lines, not the filename — so messages, syslog and a renamed archive all read, and a file that does not match the pinned format is parked with a reason instead of corrupting the table.

Query it, or join it

Run SQL from the portal, land it live in Microsoft Excel or Google Sheets, or use a federated query to join the auth log to the users in your application database — in one statement.

System log FAQ

Which format do I have?

Look at the start of a line in /var/log/syslog or /var/log/messages. If it begins with a three-letter month — Sep 10 04:11:07 — that is RFC 3164, and it is what most Linux servers still write by default. If it begins with a number in angle brackets followed by a full ISO 8601 timestamp, that is RFC 5424.

You do not have to be sure before you start. Test connection reports which profile it recognised and how much of the file matched, so the answer comes from your actual files rather than from a guess.

Do I have to reconfigure rsyslog or syslog-ng?

No. Both formats are supported, so whichever your server already writes is the one to pick — and the classic default is the format with the most testing behind it.

The one change worth considering is unrelated to the format: if you already forward logs from several machines to one collector, point the Agent at that collector’s folder and you get every host in a single connector, because each row carries the host it came from.

How do you handle the missing year in an old syslog line?

The year is taken from the file’s modification time. A rotated archive last written in December is read as December, rather than every line being dropped into the current year the way a naive parser would.

The limit is worth knowing: if a file’s timestamps have been flattened — restored from a backup that did not preserve them, or copied in a way that reset them — the inference works from what the filesystem says. The original text is always kept beside the parsed value, so you can see what was actually on the line.

Can you read the systemd journal directly?

No, and this is the setup step people miss. The journal on disk is a binary database, and Query Streams does not open it. What it reads is the export format: journalctl -o export > journal.export, in any folder the Agent can read.

In practice that means a small scheduled job on the server — a timer or a cron entry that writes the export somewhere the Agent already looks. Everything after that is automatic, and gzipped exports are read too.

What about Windows Event Log or macOS?

Neither has a profile here. The three formats on this page are Linux and Unix, and we would rather name that gap than let the words “system logs” imply coverage that is not there.

There is a route for the Windows case, though. Reading the live Event Log is a different problem from reading a text file — it is a structured channel rather than a folder of lines — and it belongs to a separate connector rather than this one. But if you already export those events to text or CSV on a schedule, that folder is readable today: describe the layout once with a custom log connector and the exports become a table like anything else on this page.

My logs have usernames and IP addresses in them. Should I worry?

Yes, and it is worth thinking about before you share a query rather than after. auth.log is a record of who connected from where, which is personal data in most jurisdictions, and a syslog message is freeform text that can contain anything a program chose to print.

Query Streams does not change what is in the file, so the honest framing is that the table inherits the file’s sensitivity. Two things help: a saved query shares a result, not the connection, so a recipient never gets the whole log; and filters let you narrow a query to the programs someone actually needs before you share it with them.

I collect logs from many servers into one folder. Does that work?

That is the case this is best at. All three formats carry the host as a column — the two syslog formats read it out of the line, and the journal export takes it from its own hostname field — so a folder collected from forty machines is one connector, and grouping by host is an ordinary GROUP BY.

Servers in different buildings or regions work too. Each location runs its own Agent, and a federated query can read across them in a single statement on Business and Enterprise plans.

What about a line the parser cannot read?

It is kept, not dropped. A line that does not match the pinned format is retained as a raw row with a reason attached, so a malformed entry or an unexpected format change shows up as something you can see and query rather than as a silent gap in your numbers.

The same applies to a journal value the format marks as binary: it is parked beside its entry with a reason rather than being guessed at, and the entry itself stays whole.

Your servers have been writing this down for years

Connect a log folder and run your first query against it in a few minutes. Free tier, no credit card, nothing installed on the server.

Get Started Free See all connectors

Read-only · outbound connection only · your logs stay on your servers