Add cron job to do HTTP PUT /jobtimeoutcheck that checks for timeout conditions and performs corrective action

[ericpassmore]

Timeout check needs to

Tasks

Beta Give feedback

1. set status to TIMEOUT when threshold exceeded on last_updated or start_time when last_update is not set
2. call /hosts?reap=all to kill off hosts associated with the timeout job
3. re-inits job back to status WAITING_4_WORKER
4. post /hosts?count= to start new host to take the place of hosts terminated above

[ericpassmore]

To support AWS spot instances we need to recover jobs that are orphaned. This is accomplished via a timeout check. Jobs status and times are updated every few minutes with most recent block. So every WORKING job should have a recent update time.
Add cron job to do HTTP PUT /jobtimeoutcheck that checks for timeout conditions and performs corrective action

[ericpassmore]

Tagged with faster-replay because spot instances should allow 2x more nodes.