[BUG] `serverOnlyFields` causes false conflicts on push when field not present on stored document

[fadnincx]

Description

When using addReplicationEndpoint with serverOnlyFields, push requests that create new documents succeed, but all subsequent pushes (update/delete) on those documents fail with false conflicts. The conflict response returns a document identical to the assumedMasterState, making this very confusing to debug.

Versions

• rxdb: 16.21.1
• rxdb-server: 16.21.1

Root Cause

Two bugs in mergeServerDocumentFieldsMonad in src/plugins/server/helper.ts:

Bug 1: New documents don't get server-only fields initialized

When a document is created via push, serverDoc is undefined and the function returns the client doc as-is:

if (!serverDoc) {
    return clientDoc; // ← server-only fields never set!
}

The document is stored in the database without the server-only field property.

Bug 2: Copying undefined creates phantom keys that break deepEqual

On subsequent pushes, serverDoc exists but doesn't have the server-only field (it was never initialized). The code does:

ret[field] = serverDoc[field]; // serverDoc.myServerField → undefined

This creates a property with value undefined on assumedMasterState. JavaScript's Object.keys() counts this as a key, but JSON.stringify() omits it.

In masterWrite, the actual master state (fetched from storage) has no such property. The deepEqual comparison fails at:

if (length !== Object.keys(b).length) return false; // e.g. 5 !== 6

Result: false conflict. The conflict response goes through JSON.stringify, stripping the undefined property, so it looks identical to the assumedMasterState.

Minimal Reproduction

import express from 'express';
import { createRxDatabase } from 'rxdb';
import { getRxStorageMemory } from 'rxdb/plugins/storage-memory';
import { createRxServer } from 'rxdb-server/plugins/server';
import { RxServerAdapterExpress } from 'rxdb-server/plugins/adapter-express';

const db = await createRxDatabase({
    name: 'testdb',
    storage: getRxStorageMemory(),
});

await db.addCollections({
    items: {
        schema: {
            version: 0,
            primaryKey: 'id',
            type: 'object',
            properties: {
                id:        { type: 'string', maxLength: 100 },
                name:      { type: 'string' },
                updatedAt: { type: 'number' },
                // This field exists in the schema but is server-only:
                serverTag: { type: ['string', 'null'] },
            },
            required: ['id', 'name', 'updatedAt'],
        },
    },
});

const server = await createRxServer({
    database: db,
    adapter: RxServerAdapterExpress,
    port: 3000,
    authHandler: async () => ({
        data: {},
        validUntil: Date.now() + 999999999,
    }),
});

server.addReplicationEndpoint({
    name: 'items',
    collection: db.items,
    serverOnlyFields: ['serverTag'],
    changeValidator: () => true,
    queryModifier: (_auth, query) => query,
});

await server.start();

// --- Step 1: Create a document (succeeds) ---
const createRes = await fetch('http://localhost:3000/items/0/push', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify([{
        newDocumentState: {
            id: 'doc1',
            name: 'Test',
            updatedAt: 1000,
            _deleted: false,
        },
        // No assumedMasterState for new document
    }]),
});
console.log('Create response:', await createRes.json());
// Expected: [] (no conflicts)

// --- Step 2: Delete the document (FALSE CONFLICT!) ---
const deleteRes = await fetch('http://localhost:3000/items/0/push', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify([{
        assumedMasterState: {
            id: 'doc1',
            name: 'Test',
            updatedAt: 1000,
            _deleted: false,
        },
        newDocumentState: {
            id: 'doc1',
            name: 'Test',
            updatedAt: 1000,
            _deleted: true,
        },
    }]),
});
const conflicts = await deleteRes.json();
console.log('Delete response:', JSON.stringify(conflicts, null, 2));
// BUG: Returns conflict with document identical to assumedMasterState:
// [{ "id": "doc1", "name": "Test", "updatedAt": 1000, "_deleted": false }]
//
// Expected: [] (no conflicts — the delete should succeed)

await server.close();
await db.close();

Why the conflict response looks identical

The conflict goes through writeDocToDocState(masterState) → JSON.stringify().
The assumedMasterState has { ..., serverTag: undefined } (11 keys).
The master state has { ... } (10 keys).

deepEqual sees different key counts → conflict.
But JSON.stringify strips the undefined property → both look the same in the HTTP response.

Proposed Fix

In src/plugins/server/helper.ts, mergeServerDocumentFieldsMonad:

export function mergeServerDocumentFieldsMonad<RxDocType>(serverOnlyFields: string[]) {
    let useFields = serverOnlyFields.slice(0);
    useFields = uniqueArray(useFields);

    return (
        clientDoc: RxDocType | RxDocumentData<RxDocType>,
        serverDoc?: RxDocType | RxDocumentData<RxDocType>
    ) => {
        const ret = flatClone(clientDoc);
        if (!serverDoc) {
            // Initialize server-only fields to null for new documents
            useFields.forEach(field => {
                (ret as any)[field] = null;
            });
            return ret;
        }
        useFields.forEach(field => {
            // Only copy if field exists on serverDoc to avoid creating
            // properties with undefined value (which break deepEqual key count)
            (ret as any)[field] = field in serverDoc ? (serverDoc as any)[field] : null;
        });
        return ret;
    };
}

This fixes both bugs:

1. New documents get server-only fields initialized to null
2. Existing documents without the field get null instead of undefined

Proposed Test Cases

These tests reproduce the bug and verify the fix. They extend the existing mergeServerDocumentFieldsMonad test in test/unit/server.test.ts (which only covers the happy path where serverDoc has the field).

// Add to test/unit/server.test.ts, inside the existing
// describe('.mergeServerDocumentFieldsMonad()', () => { ... })
// Uses the same imports already present: assert, RxDocumentData, mergeServerDocumentFieldsMonad
// Additionally import deepEqual:
// import { deepEqual } from 'rxdb/plugins/utils';

it('should initialize server-only fields to null when serverDoc is undefined (new document)', () => {
    const clientDoc = { id: 'foobar' };
    const result = mergeServerDocumentFieldsMonad<any>(['private'])(clientDoc, undefined);

    assert.strictEqual(result.private, null);
    assert.strictEqual(result.id, 'foobar');
    // Must not mutate the original
    assert.strictEqual('private' in clientDoc, false);
});

it('should not create undefined properties when serverDoc lacks the field', () => {
    // Simulates: document was created via push without the server-only field,
    // stored in DB without it, then a subsequent push references it as serverDoc
    const serverDoc: any = {
        _attachments: {},
        _deleted: false,
        _meta: { lwt: 2000 },
        _rev: '1-rev',
        id: 'foobar',
        // note: no 'private' field at all
    };
    const clientDoc = { id: 'foobar' };
    const result = mergeServerDocumentFieldsMonad<any>(['private'])(clientDoc, serverDoc);

    // Must be null, not undefined
    assert.strictEqual(result.private, null);
    // Must survive JSON roundtrip (undefined would be stripped)
    assert.strictEqual(JSON.parse(JSON.stringify(result)).private, null);
});

it('should not cause false conflicts with deepEqual when server-only field is missing from stored doc', () => {
    // This is the exact scenario that causes the bug in the push endpoint:
    // masterWrite compares writeDocToDocState(masterState) with row.assumedMasterState
    // using deepEqual. If mergeServerDocumentFields creates an undefined property,
    // Object.keys().length differs and deepEqual returns false.
    const merge = mergeServerDocumentFieldsMonad<any>(['private']);

    const storedDoc: any = { id: 'foobar', name: 'test', _deleted: false };
    // ^ no 'private' — was stored via push without server-only field

    const clientAssumedMaster = { id: 'foobar', name: 'test', _deleted: false };
    const mergedAssumed = merge(clientAssumedMaster, storedDoc);

    // Simulate what writeDocToDocState returns (the raw stored doc without _meta/_rev)
    const masterState = { id: 'foobar', name: 'test', _deleted: false };

    // BUG: with the old code, this fails because mergedAssumed has
    // { ..., private: undefined } (4 keys) vs masterState (3 keys)
    assert.strictEqual(
        Object.keys(mergedAssumed).length === Object.keys(masterState).length + 1,
        true,
        'merged doc should have exactly one extra key (the server-only field)'
    );
    assert.strictEqual(
        Object.values(mergedAssumed).every(v => v !== undefined),
        true,
        'no property should be undefined (breaks deepEqual key count)'
    );
});