This is a Nvidia GPU training monitoring tool using python. This script can kill a training process when timeout.
You can adjust the time threshold on line 34 as you want (Default 5 min).