google cloud platform - Possibility to catch a dataproc kill signal on a spark streaming job -

i looking way catch dataproc job kill signal in python spark streaming job: have 1 specific job on dataproc opens several connections postgresql db, has limited amount of connections in pool. currently, if job restarted, connections not closed , result next instance of job not have enough connections available operate correctly. if able catch kill signal in job somehow, still ensure connections closed eventually.

i suspect best can register atexit handler in python driver; whether gets called depends on cause of restart or failure, can verify whether seems work testing intended restart case first.

otherwise, if there's way force cleanup of orphaned connections somehow through other means, might easier them on startup , issue necessary cleanup calls explicitly.