fix: Tombstone MDN before sending it (#8252)

[iequidoo]

Otherwise, when it appears on IMAP, it will mark chat messages as seen/noticed even if
markfresh_chat() is called meanwhile.

Fix #8252

[WofWca]

Sounds like this should work but I'm probably not the right person to ask.

IDK about introducing different behavior for tests though, it doesn't seem like something we often do at least. At least some comments about could be useful? But maybe it's obvious to someone who knows what they're reading.

[iequidoo]

IDK about introducing different behavior for tests though, it doesn't seem like something we often do at least.

The smtp code (send_mdn_rfc724_mid() particularly) isn't tested at all in Rust currently. It's better to test it somehow than not test it at all, and running an SMTP server in Rust tests isn't what we want.

[link2xt]

IDK about introducing different behavior for tests though

This is already discussed in #8008, we indeed don't want to have conditional compilation or hooks just for tests. Just checking that MDN is in smtp_mdns table would be enough.

[iequidoo]

For the record, i've removed conditional compilation, but just in case if changes still look complex, i'm going to replace the changes to smtp and the exisitng Rust tests with a JSON-RPC Python test, it should fail because MDNs aren't appearing on IMAP instantly.

[iequidoo]

I've tried to add a Python test, but it fails because #7758 is still unmerged and that affects testing because the message state is "synced back" from IMAP, doing anything with MDNs is not helpful at the moment. This PR should be merged after #7758

So this is flaky, sometimes get_fresh_message_count() returns 0:

def test_mark_fresh_vs_self_mdn(acfactory) -> None:
    alice, bob = acfactory.get_online_accounts(2)
    bob.set_config("bcc_self", "1")

    alice_contact_bob = alice.create_contact(bob)
    alice_chat = alice_contact_bob.create_chat()
    alice_chat.send_text("Hello!")

    event = bob.wait_for_incoming_msg_event()
    bob.wait_for_event(EventType.IMAP_INBOX_IDLE)
    chat_id = event.chat_id
    msg_id = event.msg_id

    bob_chat = bob.get_chat_by_id(chat_id)
    message = bob.get_message_by_id(msg_id)
    bob.clear_all_events()
    bob.mark_seen_messages([message])
    bob_chat.mark_fresh()
    assert bob_chat.get_fresh_message_count() == 1
    bob.wait_for_event(EventType.IMAP_INBOX_IDLE)
    assert bob_chat.get_fresh_message_count() == 1

[iequidoo]

Can improve this in a follow-up PR if you want, or create an issue. We don't need to interrupt the inbox loop that often, e.g. in imap::prefetch_should_download(). markseen_on_imap_table() even interrupts it unconditionally: it does INSERT OR IGNORE, but it doesn't check if a row was inserted.

[link2xt]

Currently the test looks like it will get stuck if we decrease the number of interruptions or even if the network is slow and two interruptions collapse into one. I'd at least put a TODO here.

[link2xt]

There is probably a way to make it non-flaky by not relying on inbox IDLE already without modifications to how often we interrupt IDLE.

E.g. set up a second Bob and wait for MDN to arrive there (the chat marked as seen on second Bob's device). Then send another message from second Bob to self messages and wait for it to arrive on first Bob's device. Then check the status of the chat with Alice on Bob's first device.

[link2xt]

For Rust tests dedicated MDN rendering function as described in #8008 (comment) is likely also fine if it is just a factored out code that is also used by the code running outside the tests.

[iequidoo]

There is probably a way to make it non-flaky by not relying on inbox IDLE already without modifications to how often we interrupt IDLE.

I've modified the test by waiting for the MDN on alice's side and sending a new message to bob, to not complicate the test too much. Have checked that it fails w/o the fix.

Still, should i create an issue for reduction of number of inbox loop interruptions?

[iequidoo]

https://github.com/chatmail/core/actions/runs/27877188488/job/82498958716?pr=8257 -- the same issue as in #8222 (comment). Not related to the PR.

[link2xt]

Currently the test looks like it will get stuck if we decrease the number of interruptions or even if the network is slow and two interruptions collapse into one. I'd at least put a TODO here.

[WofWca]

Still looks OK to my untrained eye.

[iequidoo]

There was an unrelated failure of tests/test_1_online.py::test_archived_muted_chat, https://github.com/chatmail/core/actions/runs/27966845289/job/82763940033?pr=8257 , the end of log is:

9.68 [events-ac1] INFO src/imap/idle.rs:67: Transport 1: IDLE entering wait-on-remote state in folder "INBOX".
65.82 [events-ac1] WARNING src/smtp/connect.rs:154: SMTP failed to connect to ci-chatmail.testrun.org (49.12.217.82:465): timed out.
65.82 [events-ac1] WARNING src/smtp.rs:148: SMTP failed to connect and authenticate: SMTP failed to connect: All connection attempts failed: timed out.
65.82 [events-ac1] INFO src/smtp.rs:133: SMTP trying to connect to ci-chatmail.testrun.org:587:starttls.
65.82 [events-ac1] INFO src/net/dns.rs:134: Using memory-cached DNS resolution for ci-chatmail.testrun.org.
65.82 [events-ac1] INFO src/net/dns.rs:180: Resolved ci-chatmail.testrun.org into 49.12.217.82.
65.83 [events-ac1] INFO src/smtp/connect.rs:109: Attempting SMTP connection to ci-chatmail.testrun.org (49.12.217.82:587).
66.82 [events-ac1] DC_EVENT_SMTP_CONNECTED data1=0 data2=SMTP-LOGIN as ci-ktq5rj@ci-chatmail.testrun.org ok
66.82 [events-ac1] INFO src/smtp.rs:391: Try number 1 to send message Msg#13 (entry 1) over SMTP.
66.82 [events-ac1] DC_EVENT_CONNECTIVITY_CHANGED data1=0 data2=0
67.20 [events-ac1] INFO src/smtp/send.rs:62: Message len=3159 was SMTP-sent to 1 recipients..
67.20 [events-ac1] DC_EVENT_SMTP_MESSAGE_SENT data1=0 data2=Message len=3159 was SMTP-sent to 1 recipients.
67.20 [events-ac1] DC_EVENT_MSG_DELIVERED data1=12 data2=13
67.21 [events-ac1] DC_EVENT_CHATLIST_ITEM_CHANGED data1=12 data2=0
67.21 [events-ac1] INFO src/smtp.rs:639: Sending MDNs.
67.21 [events-ac1] INFO src/scheduler.rs:581: SMTP fake idle started.
67.21 [events-ac1] DC_EVENT_CONNECTIVITY_CHANGED data1=0 data2=0
67.21 [events-ac1] INFO src/scheduler.rs:609: SMTP has no messages to retry, waiting for interrupt.

Not clear why it failed after the successful reconnection, we have a 300s timeout in the CFFI tests.