Bash: How to continue for loop when condition X is satisfied

Question

I'm trying to start a process on multiple VMs using GitHub Actions. However, the command doesn't complete (as it needs to continue running) so my for loop just kind of hangs.

What I need, is a way to inspect the command's log output (which is in a known file) and then continue the loop whenever a specific sentence appears in the log file.

This is a minimal example of what I currently have:

#! /bin/bash

VMS=("10.10.789.001" "10.10.789.002" "10.10.789.003")
for VM in "${!VMS[@]}"; do
command.sh "$VM"
done

command.sh:

#! /bin/bash
vmip=$1
ssh user@"$vmip" -T "myprogram -flag &> output.log"

This works in a sense that the connection is made, myprogram is started and its output redirected to output.log. However, since the myprogram continues running (as is intended), the for loop never reaches done.

I've tried setting a timeout like this:

#! /bin/bash

VMS=("10.10.789.001" "10.10.789.002" "10.10.789.003")
for VM in "${!VMS[@]}"; do
timeout 5m command.sh "$VM"
done

That however ends the whole script because it effectively throws an exit after the timeout is reached, and that still doesn't continue the for loop.

I know that myprogram will write setup complete to the output.log file, so I could check for that by doing something like:

#! /bin/bash
cat vault.log | grep "setup complete"
if [ $? = 0 ]; then
<do something>
fi

If the condition above satisfies, the program has completed setting up and is now just running, which is what I want. However, I don't know what the <do something> part needs to do, in order to ensure the original for loop can continue.

Note I do not have access to the contents of the myprogram program. It's simply an existing program which I need to run, not something I can alter myself.

Answer 1

Background process each ssh command which starts myprogram. Then do some kind of loop which checks the log.

command.sh:

#! /bin/bash

main()
{
    local ip="$1"

    # Run the program in the background and redirect output to the log file
    ssh user@"$ip" -T "myprogram -flag &> output.log" &
    local ssh_pid=$!

    local max_iterations=100
    local iter_sleep_seconds=2


    # Wait until the setup complete message appears in the log file
    for ((i=1; i<=max_iterations; i++))
    do
        if ssh user@"$ip" -T "grep 'setup complete' output.log"
        then
            echo "Setup complete on $ip."
            break
        else
            echo "Waiting for setup to complete on $ip..."
            sleep $iter_sleep_seconds
        fi
    done

    if (( i > max_iterations ))
    then
        echo "Maximum number of iterations reached for $ip. Killing the instance."
        kill $ssh_pid
        return 1
    fi

    return 0
}

main "$@"

Background process each call of command.sh

main.sh:

VMS=("10.10.789.001" "10.10.789.002" "10.10.789.003")

for VM in "${VMS[@]}"
do
    ./command.sh "$VM" &
done

wait
echo "All VM processes are done. Check log above if successful."