fix(logs): a line with no RouterOS id had no protection against re-insertion (v0.24.4-beta)
Follows #137. The v0.24.1 fix made the count honest and stopped a thousand round trips per poll, but it assumed the unique index on (device_id, log_id) would reject lines already stored. For most devices it does. For a device whose log entries carry no `.id` it never could: the column was set to NULL, and PostgreSQL treats NULLs as *distinct* in a unique index, so every NULL conflicts with nothing. Those devices re-inserted their entire buffer on every poll, for ever, and the constraint that was supposed to stop it was structurally incapable of firing. A line with no id now gets a key derived from its own content -- timestamp, topics and message -- so the existing index does the work. The `#` prefix keeps it out of the space of real RouterOS ids (`*` plus hex), so the two cannot collide, and parseRosId already rejects it, which matters because the watermark is a max over real ids and must not be poisoned by a synthetic one. The trade, stated rather than buried: two genuinely separate lines sharing a timestamp, topic and message collapse into one. RouterOS timestamps are second-resolution, so that can happen. Losing a repeat of an identical message is a much smaller loss than growing the table by the whole buffer every minute without bound. This was found by reading a comment on #137 I had not read before closing it -- log output from a device with eleven lines total, reporting eleven new entries every minute for forty-five minutes. My own fleet cannot reproduce it: every event here carries a real `.id`, with no nulls and no duplicates, which is exactly why the hole was invisible from here. So this closes a defect demonstrable from the schema; whether it is the whole of that reporter's problem is still an open question and the issue is reopened to ask. Verified no regression where ids are healthy: no surrogate keys generated, no nulls, no duplicates, and counts that vary with real activity. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
6bc7480b89
commit
d9fa120abc
@@ -30,7 +30,7 @@ const UPDATE_DOWNLOAD_TIMEOUT_MS = 10 * 60_000;
|
||||
* learned to abandon a timed-out connection, desynchronised the session (#137).
|
||||
*/
|
||||
const LOG_READ_TIMEOUT_MS = 120_000;
|
||||
import { selectNewLogLines, highestStoredId, type RawLogLine } from '../../utils/logDedup';
|
||||
import { selectNewLogLines, highestStoredId, surrogateLogId, type RawLogLine } from '../../utils/logDedup';
|
||||
import { alertService } from '../AlertService';
|
||||
|
||||
/** DB column limits for topology_links (see migrate.ts); reject oversize rows instead of silent truncation. */
|
||||
@@ -852,17 +852,23 @@ export class DeviceCollector {
|
||||
|
||||
const pending: unknown[][] = [];
|
||||
for (const log of fresh) {
|
||||
const logId = (log['.id'] || '') as string;
|
||||
const rawId = (log['.id'] || '') as string;
|
||||
const topics = (log['topics'] as string) || '';
|
||||
const message = (log['message'] as string) || '';
|
||||
const time = this.parseLogTime(log['time'] || '');
|
||||
const severity = this.mapLogSeverity((log['topics'] as string) || '');
|
||||
const severity = this.mapLogSeverity(topics);
|
||||
// Never store a null log_id: the unique index that stops re-insertion
|
||||
// does not constrain nulls, so a device whose lines carry no `.id`
|
||||
// would accumulate the same entries for ever (#137).
|
||||
const logId = rawId || surrogateLogId(log['time'] || '', topics, message);
|
||||
pending.push([
|
||||
this.device.id,
|
||||
time.toISOString(),
|
||||
severity,
|
||||
(log['topics'] as string) || null,
|
||||
(log['message'] as string) || '',
|
||||
topics || null,
|
||||
message,
|
||||
JSON.stringify(log),
|
||||
logId || null,
|
||||
logId,
|
||||
]);
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user