JSON & app logs New

Connect Application Logs to Excel, Sheets and AI

Q: Why are dpkg and apt in with application logs?

Because they answer the question you ask immediately after 'when did this start failing?' - namely 'what changed?'. They are the package managers' own logs, in the same folder tree, read the same way, and they are the most useful neighbour an error log has. apt's history is the readable one: one block per transaction with the exact command that ran and what it installed, upgraded or removed. dpkg's log is the fine-grained one, a row per package step. Between them you can put a date against a change instead of trying to remember it.

Q: What about Red Hat, Fedora or Alpine package history?

There is no profile for dnf, yum or apk today, and no Windows equivalent. The two package formats here are dpkg's log and apt's history, which covers Debian and Ubuntu. Your application's own logs are unaffected by any of that - the JSON and text formats do not care what distribution they are sitting on. And if you are unsure what a folder of yours will be recognised as, Test connection reports what it found before you commit to anything.

Your application’s log, stack traces still attached. JSON lines from ECS, Serilog, pino or Bunyan. Plain timestamp [LEVEL] logger: message text, where a forty-line stack trace stays joined to the line that raised it instead of becoming forty useless rows. And, on Debian and Ubuntu, the two logs that record what changed on the box.

Start free 11 destinations

1connection

0inbound ports

read-onlyenforced

4 application log formats Stack traces kept with their message dpkg and apt package history Read-only · nothing added to your app

The big idea

Your log folder becomes a table you can group by.

Not grep, and not a log stack. Query Streams reads the files your application already writes, pins the format, and gives you typed columns — so “which logger is throwing the most errors this week” is one query rather than an afternoon. A line the parser cannot read is kept as a row with a reason rather than silently dropped.

Log folderon your server

shopfront-api/app.jsonlJSON

shopfront-worker/app.logtext

history.logapt

Rotations, compressed archives and re-read files — handled

noisiest-loggers.sqlSQL

SELECT app,
       logger,
       count(*) AS errors
FROM   fileset.events
WHERE  level = 'ERROR'
GROUP BY app, logger
ORDER BY errors DESC

Resulterrors by logger

apploggererrors

shopfront-apiPaymentClient412

shopfront-apiOrderRepo96

shopfront-workerEmailSender31

shopfront-webSessionFilter4

Join it to your application database in the same query

The files your app already writes, now a table you can report on

Sources

Two shapes your app writes, two the box writes

Each card names the exact files Query Streams reads and what you get out of them. You do not adopt a logging library or change a format to make this work — the formats supported are the ones these tools already produce by default.

Supported application logs

4 formats

JSON linesECS, Serilog compact, pino, Bunyan

One JSON object per line in .jsonl, .ndjson or *-json.log
ECS field names map straight through — log.level, host.name, service.name
Serilog compact, pino and Bunyan read too: same keys, different spellings
Nested objects flatten to dotted columns; anything unmapped lands in data
Appends; dated rotations and .gz followed

Text logtimestamp, level, logger, message

The default shape: a stamp, a level in brackets, a logger, a message
A stack trace is joined to the line that raised it — one row, not forty
at …, Caused by: and Traceback all read as continuation
Reads .log, .out and .txt; JSON files are excluded by declaration
Comma or dot milliseconds; local clock read as the Agent’s

dpkgpackage log

One row per package step: install, upgrade, remove, configure
package, version and new_version as their own columns
status lines carry the state dpkg moved the package into
/var/log/dpkg.log and its numbered .gz rotations
No zone in the file; read as the Agent’s local clock

apthistory log

One Key: value block per transaction, blank line between blocks
The exact Commandline that ran, and what it changed
Start-Date is the event time; End-Date is its own column
An unfinished transaction waits rather than being split in half
/var/log/apt/history.log plus its .gz rotations

More comingthese are the start, not the ceiling

New application and package log formats are added with each Agent release. Because Agents keep themselves up to date, a format added later shows up in your connector without anyone touching a server.

All four parsers were built and tested against real captured log files, rotations and compressed archives included. The two package logs are Debian and Ubuntu only — there is no profile for RPM-based distributions or for Windows package history, and we would rather say that than let you find out during setup.

Everywhere it runs

Your application’s log from every region, read where it runs

One application is rarely one server. An API in one cloud region, workers in another, something still on a box in the office. Each location runs a Network Agent that reads the log files where they already are and dials out to Query Streams. As far as your firewall is concerned that is an ordinary outbound connection — nothing to open, no VPN to build, and no logging library to add to your app.

eu-west The API writing JSON lines, one file per service dials out

us-east Workers writing text logs, stack traces and all dials out

Head office An older box, plus the dpkg and apt history beside it dials out

Three outbound connections, one place to query them — no inbound port, no VPN, no firewall change

One Agent per location

An Agent covers a whole location rather than a single log: the JSON lines, the text log and the package history at that site each become their own File Set connector on the same Agent. The free tier runs a single Agent and higher tiers run more.

1 location = 1 Agent = many connectors

Nothing to open

The Agent makes one encrypted outbound connection, and the request out and the rows back both travel over it. No inbound port, no VPN, no firewall change to go and ask for — and nothing added to your application to make it work.

one connection, both directions

One query across regions

Every source in a federated query names its own Agent, so a single statement can put the API’s errors in one region beside the workers’ in another — or beside the package history that explains when it started. Included on Business and Enterprise.

2 regions → 1 result set

They keep themselves current

Agents update themselves, so a log format added in a later release reaches every location without a deploy of your own. That is what makes the wall above a starting point rather than a fixed list.

new formats arrive with the Agent

The hard part

Stack traces, clocks and lookalikes

An application log is the messiest file on the server: one event can span forty lines, two unrelated formats can look identical, and half of these files carry no timezone at all. Each of those is handled by declaration rather than by guesswork.

One event, not forty rows

A line that does not open with a timestamp belongs to the line above it. That is how a stack trace stays attached to the message that raised it, instead of shattering into dozens of fragments that no GROUP BY can put back together.

continuation lines → joined, up to 200

Two logs that look alike

A JSON application line, a Docker envelope and a web server’s JSON access line are all “JSON with a timestamp in it”. The format is decided by what the line contains, so each one lands on its own profile — never by branching on a filename.

content decides, not the file name

A clock with no timezone

Text logs, dpkg and apt all write the host’s local time with no offset attached. Rather than guess, those profiles declare that the stamp is local and read it as the Agent’s timezone — and the original string is kept in its own column either way.

local clock → declared, never inferred

A transaction still running

apt writes the start of an upgrade immediately and the end line only when dpkg finishes, which can be minutes later. A block that has not been closed yet waits, instead of being cut in half and turning up as an event with no beginning.

waits for End-Date, up to 30 min

In the file5 lines

2026-09-08 14:02:11,431 [ERROR] PaymentClient: charge failed java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.read at PaymentClient.charge(PaymentClient.java:88) … 24 more

In the table1 row

event_time: 2026-09-08 14:02:11.431
level: ERROR
logger: PaymentClient
message: charge failed ↵ java.net.SocketTimeoutException: Read timed out ↵ at …

Count the exception, not the lines it printed

The schema

The columns you actually get

Not a blob of text with a timestamp beside it. Each format is parsed into typed columns you can filter, group and aggregate on directly — and where your application logs structured JSON, its own fields are unwrapped into columns too.

JSON linesECS · Serilog · pino · Bunyan

levelloggermessagehostservicestreamdata

Text logstack traces joined in

levelloggermessage

dpkg package logDebian · Ubuntu

actionstatepackageversionnew_versiondetail

apt history logone block per transaction

commandlinerequested_byinstallupgraderemovepurgeerrorend_datedata

Every row also carries event_time, the raw stamp it was read from, and the file it came from. Your application logs are identified by app and the two package logs by host — both taken from the folder the file sits in, so a folder per service or per machine gives you a clean column to group by. The three-column text log is deliberate: rather than guess at fields a freeform message does not reliably have, the message is kept whole, with its stack trace inside it.

Power Query M

Paste the generated Power Query M into the Power BI Advanced Editor and the report reads live application log results over HTTPS — no ODBC driver, no database port opened.

How Power BI works

Supported

Application logs alerts and reports

Slack · Discord · Email · Webhook

Put an application log query on a schedule and have the rows delivered to Slack, Discord, email or a signed webhook — or hold the message until a row count, threshold or percentage change crosses the line you set.

How alerts and reports work

The alerts card is the one worth a second look here: a scheduled query over your error count needs nobody watching a terminal, so a spike reaches Slack or an inbox on its own. Log-specific step-by-step guides are not written yet — the linked pages cover how each surface works.

What this deliberately does not do

It is not a live tail. Query Streams reads the files as they are on disk when you run the query, so an error spike shows up as rows rather than as a stream you watch. If you need sub-second alerting on a line as it is printed, a log pipeline is the right tool and we will say so.

Your application’s log contains whatever your application prints, and that is worth saying plainly: if it logs a token, a customer email or a full request body, that string is already in the file and it will be in the column too. There is no field map to exclude the way there is for a web server log. Treat the resulting table with exactly the care you would treat the file.

Some of what the package logs can record is recognised by shape rather than proven by capture. dpkg documents a configuration-file line and purge and disappear actions; apt documents Requested-By, Error, Purge, Reinstall and Downgrade. None of those appeared in the logs these profiles were built against, so they are declared, not demonstrated — and neither package log covers RPM-based distributions or Windows.

One setup detail rather than a limit: the identity column comes from the folder a file sits in below the folder you point at. Point a connector at a directory that holds one subfolder per service or per machine and you get useful names for free; point it straight at a directory of loose files and every row is named after that one directory. Worth thirty seconds when you set it up.

The rest follows the usual design. The Agent connects outbound and reads files; nothing is installed alongside your application, nothing is written back, no process is attached to or instrumented, and access is read-only.

How it works

Three steps, and no change to how your application logs.

Point at the log folder

Give a Query Streams Agent read access to where the logs already land — the folder beside your application, a share you collect them onto, or /var/log for the package history.

It recognises the format

Query Streams identifies the format from the content of the lines, not the filename — so a renamed copy still reads, and a file that does not match the pinned format is parked with a reason instead of corrupting the table.

Query it, or join it

Run SQL from the portal, land it live in Microsoft Excel or Google Sheets, or use a federated query to join your error log to the orders and users in your application database — in one statement.

Application log FAQ

Which logging library do I need to be using?

None in particular. If you write JSON lines, the Elastic Common Schema field names map straight through, and Serilog’s compact format, pino and Bunyan read as well because they use the same top-level keys with different spellings — the timestamp is taken from whichever of the usual names your line actually carries.

If you write plain text, the ordinary timestamp [LEVEL] logger: message shape is a supported format in its own right. There is nothing to adopt and nothing to reconfigure.

What actually happens to a stack trace?

It stays with the error that produced it. A record starts at a line that opens with a timestamp, and every following line that does not is treated as a continuation of it — which is exactly the shape a Java, .NET or Python trace has. The exception class, the at … frames, a Caused by: chain and a Python Traceback header are all recognised as belonging to the line above.

The practical difference is that counting errors counts errors. Without this, one exception inflates your numbers by however many frames it happened to print, and the message you actually want to read is in a different row from the class name.

Why are dpkg and apt in with application logs?

Because they answer the question you ask immediately after “when did this start failing?” — namely “what changed?”. They are the package managers’ own logs, in the same folder tree, read the same way, and they are the most useful neighbour an error log has.

apt’s history is the readable one: one block per transaction with the exact command that ran and what it installed, upgraded or removed. dpkg’s log is the fine-grained one, a row per package step. Between them you can put a date against a change instead of trying to remember it.

Can I join my errors to the upgrade that caused them?

Yes, with one thing to know: a connector reads one format, so your application log and /var/log/apt/history.log are two connectors rather than one. A federated query joins them in a single statement, the same mechanism that joins a log to a production database.

That is the query worth building on this family — errors per hour beside the transactions that ran that day, both of them read from files nobody had to ship anywhere.

Is this real-time?

It is on-demand rather than streaming. The files are read as they stand when the query runs, so new lines appear as your application writes them and the file is picked up — not on a fixed nightly schedule, but not a live tail either.

For asking a question of the last few weeks and getting the answer into a spreadsheet or a scheduled report, that is the useful shape and considerably less machinery than a log stack.

My log has fields you have not listed. Are they lost?

No. On the JSON and apt formats, anything that does not map to a named column is kept in a data column rather than discarded, so a request id, a tenant or a duration you log is still queryable. Nested objects are flattened into dotted names on the way in.

A line that does not match the pinned format at all is also kept — as a raw row with a reason attached, so an unexpected format change shows up as something you can see rather than as a silent gap in your numbers.

What about Red Hat, Fedora or Alpine package history?

There is no profile for dnf, yum or apk today, and no Windows equivalent. The two package formats here are dpkg’s log and apt’s history, which covers Debian and Ubuntu.

Your application’s own logs are unaffected by any of that — the JSON and text formats do not care what distribution they are sitting on. And if you are unsure what a folder of yours will be recognised as, Test connection reports what it found before you commit to anything.

Your application already wrote it down

Connect a log folder and run your first query against it in a few minutes. Free tier, no credit card, no change to how you log.

Get Started Free See all connectors

Read-only · outbound connection only · your logs stay on your server