Socat child processes not terminating after client disconnect

Question

I'm using socat to forward traffic from a local port to a remote server over SSL. My command is as follows:

socat -d -d TCP-LISTEN:1234,fork,reuseaddr OPENSSL:192.168.1.2:1234,verify=0

However, I'm experiencing an issue where the child processes created by socat are not terminating even after the client disconnects. When I close the client connection (using nc localhost 1234 to establish the connection and then press ctrl-C), I see the following in the socat debug log:

2024/07/06 22:37:12 socat[146196] N listening on AF=10 [0000:0000:0000:0000:0000:0000:0000:0000]:1234
2024/07/06 22:37:14 socat[146196] N accepting connection from AF=10 [0000:0000:0000:0000:0000:ffff:7f00:0001]:41648 on AF=10 [0000:0000:0000:0000:0000:ffff:7f00:0001]:1234
2024/07/06 22:37:14 socat[146196] N forked off child process 146210
2024/07/06 22:37:14 socat[146196] N listening on AF=10 [0000:0000:0000:0000:0000:0000:0000:0000]:1234

Despite this, the child processes remain active. Here is a pgrep -a socat output of hanging processes:

146196 socat -d -d TCP-LISTEN:1234,fork,reuseaddr OPENSSL:192.168.1.2:1234,verify=0
146210 socat -d -d TCP-LISTEN:1234,fork,reuseaddr OPENSSL:192.168.1.2:1234,verify=0

What I've Tried:

1. Adding keepaliveand it's settings options (based on this question): No improvement.
2. -t and -T parameters: No improvement.

Observations:

1. when I let the client end the session properly, the child process on the socat side shuts down properly
2. by default, ten seconds after the client sends the EOF, socat prints this to logs, but the child process is still active.

N inactivity timeout triggered
N exiting with status 0

3. here is the netstat output after 4 established connections:

tcp        0      1 10.0.0.21:37192         192.168.1.2:1234        SYN_SENT    217732/socat
tcp        0      1 10.0.0.21:44948         192.168.1.2:1234        SYN_SENT    217655/socat
tcp        0      1 10.0.0.21:44934         192.168.1.2:1234        SYN_SENT    217645/socat
tcp        0      1 10.0.0.21:44920         192.168.1.2:1234        SYN_SENT    217630/socat
tcp6       0      0 :::1234                 :::*                    LISTEN      217531/socat

I create many forked connections and over time the server creates hundreds of hanging processes that allocate hundreds of megabytes of memory until the server crashes.

Any help or suggestions would be greatly appreciated!

Answer 1

I haven't found a solution using socat. But I achieved the same objective using nginx ssl proxy. Here is my Docker setup:

docker-compose.yaml:

version: '3'
services:
  nginx:
    image: nginx:latest
    ports:
      - "1234:1234"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro

nginx.conf:

events { }

http {
    server {
        listen 1234;

        location / {
            proxy_pass https://destionation-on-lan:1234;
            proxy_ssl_verify off;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

Attention! Be careful using this configuration, I disable SSL certificate verification, which can lead to serious security problems - a potential malicious actor on the network can gain access to the communication between the proxy and destionation server.

Answer 2

Your netstat output shows many SYN_SENT entries, this indicates that Socats child processes try to connect but get no answer from the server 192.168.1.2:1234; there may be IP filters, a firewall, routing problems, or the server might be down. You should try to fix this issue.

However, to solve your resource exhaustion on Socat side, the option connect-timeout might help to prevent too many child processes from hanging and consuming resources:

socat -d -d TCP-LISTEN:1234,fork,reuseaddr OPENSSL:192.168.1.2:1234,verify=0,connect-timeout=1

Explanation: Each child process tries to connect to the target server but gets no response (on Linux see man 7 tcp and search for "tcp_syn_retries" to understand timing of unsuccessful connection attemts); Socat option -T only helps when the connection already has successfully been established (see Socat man page).