[core] Fix hang up on not enough space in the RCV buffer.

[yomnes0]

When there is space available in the receiving buffer after it has been fulled, send an ack to allow the transmitter to resume transmission

[yomnes0]

Naming convention is now respected and explicit check on true removed

[yomnes0]

I pushed some modification that are not intended to be merged as is. They are just a "quickfix" to allow for further debug with Maxim

[maxsharabayko]

I feel like m_bBufferWasFull should be set back to false only upon successful sending of ACK.
Something like this:

diff --git a/srtcore/core.cpp b/srtcore/core.cpp
index 0e3cce0..dac1f03 100644
--- a/srtcore/core.cpp
+++ b/srtcore/core.cpp
@@ -7904,6 +7904,7 @@ int srt::CUDT::sendCtrlAck(CPacket& ctrlpkt, int size)
     SRT_ASSERT(ctrlpkt.getMsgTimeStamp() != 0);
     int nbsent = 0;
     int local_prevack = 0;
+    bool sendAckAgain = false; //Send ack again after a buffer full was detected
 
 #if ENABLE_HEAVY_LOGGING
     struct SaveBack
@@ -7927,12 +7928,16 @@ int srt::CUDT::sendCtrlAck(CPacket& ctrlpkt, int size)
     // The TSBPD thread may change the first lost sequence record (TLPKTDROP).
     // To avoid it the m_RcvBufferLock has to be acquired.
     UniqueLock bufflock(m_RcvBufferLock);
+    if (m_bBufferWasFull && getAvailRcvBufferSizeNoLock() > 0)
+    {
+        sendAckAgain = true;
+    }
 
     int32_t ack;    // First unacknowledged packet sequence number (acknowledge up to ack).
     if (!getFirstNoncontSequence((ack), (reason)))
         return nbsent;
 
-    if (m_iRcvLastAckAck == ack)
+    if (m_iRcvLastAckAck == ack && !sendAckAgain)
     {
         HLOGC(xtlog.Debug,
               log << CONID() << "sendCtrl(UMSG_ACK): last ACK %" << ack << "(" << reason << ") == last ACKACK");
@@ -7941,7 +7946,7 @@ int srt::CUDT::sendCtrlAck(CPacket& ctrlpkt, int size)
 
     // send out a lite ACK
     // to save time on buffer processing and bandwidth/AS measurement, a lite ACK only feeds back an ACK number
-    if (size == SEND_LITE_ACK)
+    if (size == SEND_LITE_ACK && !sendAckAgain)
     {
         bufflock.unlock();
         ctrlpkt.pack(UMSG_ACK, NULL, &ack, size);
@@ -8101,7 +8106,7 @@ int srt::CUDT::sendCtrlAck(CPacket& ctrlpkt, int size)
     // [[using locked(m_RcvBufferLock)]];
 
     // Send out the ACK only if has not been received by the sender before
-    if (CSeqNo::seqcmp(m_iRcvLastAck, m_iRcvLastAckAck) > 0)
+    if (CSeqNo::seqcmp(m_iRcvLastAck, m_iRcvLastAckAck) > 0 || sendAckAgain)
     {
         // NOTE: The BSTATS feature turns on extra fields above size 6
         // also known as ACKD_TOTAL_SIZE_VER100.
@@ -8117,9 +8122,7 @@ int srt::CUDT::sendCtrlAck(CPacket& ctrlpkt, int size)
         data[ACKD_RTT] = m_iSRTT;
         data[ACKD_RTTVAR] = m_iRTTVar;
         data[ACKD_BUFFERLEFT] = (int) getAvailRcvBufferSizeNoLock();
-        // a minimum flow window of 2 is used, even if buffer is full, to break potential deadlock
-        if (data[ACKD_BUFFERLEFT] < 2)
-            data[ACKD_BUFFERLEFT] = 2;
+        m_bBufferWasFull = (data[ACKD_BUFFERLEFT] == 0);
 
         if (steady_clock::now() - m_tsLastAckTime > m_tdACKInterval)
         {
diff --git a/srtcore/core.h b/srtcore/core.h
index e7ca57c..eb3e73f 100644
--- a/srtcore/core.h
+++ b/srtcore/core.h
@@ -934,7 +934,7 @@ private: // Receiving related data
     int32_t m_iAckSeqNo;                         // Last ACK sequence number
     sync::atomic<int32_t> m_iRcvCurrSeqNo;       // (RCV) Largest received sequence number. RcvQTh, TSBPDTh.
     int32_t m_iRcvCurrPhySeqNo;                  // Same as m_iRcvCurrSeqNo, but physical only (disregarding a filter)
-
+    bool m_bBufferWasFull; // Indicate that RX buffer was full last time a ack was sent
     int32_t m_iPeerISN;                          // Initial Sequence Number of the peer side
 
     uint32_t m_uPeerSrtVersion;

[yomnes0]

With the help of @maxsharabayko , this bug should be fixed by my last commit. Reschedule was done before m_iSndLastAck was updated, this could cause issues when the flightspan was close to the available size on the receiver