Issue

When TCP session is idle for more than 15 mins (900 secs), IPVS connection times out and gets cleared from the connection table.

There are two different timeouts — one for IPVS and another one for TCP.

Default IPVS timeout value:

• ipvsadm -l --timeout
• Timeout (tcp tcpfin udp): 900 120 300

Default TCP timeout value:

• tcp_keepalive_time = 7200 (seconds)
• tcp_keepalive_intvl = 75 (seconds)
• tcp_keepalive_probes = 9 (number of probes)

When IPVS timesout, it clears the connection from the table.

Sample issues:

https://github.com/moby/moby/issues/31208

Resolution

To fix the issue, the tcp_keepalive_time has to be set to less than 900 secs. Values between 600 to 800 are optimal.

To set the values are runtime:

sysctl -w net.ipv4.tcp_keepalive_time=600

The above command is not persistent across reboots. To make the change persistent these values has to be put into the file sysctl.conf.

For the fix to be effective, application should enable the keepalive on the socket.

Example in C/C++:

` /* Set the option active */ optval = 1; optlen = sizeof(optval); if(setsockopt(s, SOL_SOCKET, SO_KEEPALIVE, &optval, optlen) < 0)

From Linux kernel 4.13 onwards, sysctl default values can be modified per container basis. Container will not inherit changes from the host sysctl modified values.