fix(logs): a line with no RouterOS id had no protection against re-insertion (v0.24.4-beta)

Follows #137. The v0.24.1 fix made the count honest and stopped a thousand
round trips per poll, but it assumed the unique index on (device_id, log_id)
would reject lines already stored. For most devices it does. For a device
whose log entries carry no `.id` it never could: the column was set to NULL,
and PostgreSQL treats NULLs as *distinct* in a unique index, so every NULL
conflicts with nothing. Those devices re-inserted their entire buffer on
every poll, for ever, and the constraint that was supposed to stop it was
structurally incapable of firing.

A line with no id now gets a key derived from its own content -- timestamp,
topics and message -- so the existing index does the work. The `#` prefix
keeps it out of the space of real RouterOS ids (`*` plus hex), so the two
cannot collide, and parseRosId already rejects it, which matters because the
watermark is a max over real ids and must not be poisoned by a synthetic one.

The trade, stated rather than buried: two genuinely separate lines sharing a
timestamp, topic and message collapse into one. RouterOS timestamps are
second-resolution, so that can happen. Losing a repeat of an identical
message is a much smaller loss than growing the table by the whole buffer
every minute without bound.

This was found by reading a comment on #137 I had not read before closing it
-- log output from a device with eleven lines total, reporting eleven new
entries every minute for forty-five minutes. My own fleet cannot reproduce
it: every event here carries a real `.id`, with no nulls and no duplicates,
which is exactly why the hole was invisible from here. So this closes a
defect demonstrable from the schema; whether it is the whole of that
reporter's problem is still an open question and the issue is reopened to ask.

Verified no regression where ids are healthy: no surrogate keys generated, no
nulls, no duplicates, and counts that vary with real activity.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2GT-Rich
2026-09-08 08:45:29 -07:00
co-authored by Claude Opus 5
parent 6bc7480b89
commit d9fa120abc
7 changed files with 98 additions and 11 deletions
@@ -30,7 +30,7 @@ const UPDATE_DOWNLOAD_TIMEOUT_MS = 10 * 60_000;
* learned to abandon a timed-out connection, desynchronised the session (#137).
*/
const LOG_READ_TIMEOUT_MS = 120_000;
import { selectNewLogLines, highestStoredId, type RawLogLine } from '../../utils/logDedup';
import { selectNewLogLines, highestStoredId, surrogateLogId, type RawLogLine } from '../../utils/logDedup';
import { alertService } from '../AlertService';
/** DB column limits for topology_links (see migrate.ts); reject oversize rows instead of silent truncation. */
@@ -852,17 +852,23 @@ export class DeviceCollector {
const pending: unknown[][] = [];
for (const log of fresh) {
const logId = (log['.id'] || '') as string;
const rawId = (log['.id'] || '') as string;
const topics = (log['topics'] as string) || '';
const message = (log['message'] as string) || '';
const time = this.parseLogTime(log['time'] || '');
const severity = this.mapLogSeverity((log['topics'] as string) || '');
const severity = this.mapLogSeverity(topics);
// Never store a null log_id: the unique index that stops re-insertion
// does not constrain nulls, so a device whose lines carry no `.id`
// would accumulate the same entries for ever (#137).
const logId = rawId || surrogateLogId(log['time'] || '', topics, message);
pending.push([
this.device.id,
time.toISOString(),
severity,
(log['topics'] as string) || null,
(log['message'] as string) || '',
topics || null,
message,
JSON.stringify(log),
logId || null,
logId,
]);
}