Overlapping cron jobs solution using Flock

While working on an in-house project we’ve hit an interesting view on the execution of the concurrent task. To be more concrete, it’s a small segment of an Amazon hosted project. We need a python script in cron running every minute doing some calculations, using Kubernetes and Dynamodb APIs. What can happen in real-life scenarios is for a script to take longer than a minute (this time should be always within but, you need to count on edge cases) in which case we need to make sure we are not corrupting the data due to overlapping jobs. This is how we have handled overlapping cron jobs using Flock

What we figured to be a good solution is using a tool called Flock (interesting words play, probably, referring to managing a flock one head at the time) to ensure escaping the overlapping jobs scenario thus, keeping data clean and safe for the next round of cron execution.

This particular project is deployed on CentOS 7. Below steps took place to ensure overlapping jobs step by step:

1. Installing Flock
   $ yum install -y util-linux
   
   The util-linux package contains a large variety of low-level system utilities that are necessary for a Linux system to function. Among others, Util-linux contains Flock. Flock manages locks from within shell scripts or the command line.
   
2. Creating a lock file
   $ touch /path/to/your_script_name.lock
   
3. Creating cron job
   $ crontab -e
   * * * * * /usr/bin/flock -w 0 /path/to/your_script_name.lock python3.7 /path/to/your_script.py
   
4. Verification
   $ fuser -v /path/to/your_script_name.lock

The moment flock starts, it locks the lock-file you specify in the command. You can see that by requesting the user/script that is having the lock on that file.

Thank you, ladies and gentlemen, see you soon with the next show and tell 😉
Oh, and make sure to check out our other tech blog posts!