Scheduler jobs in backend (Local Development)

Local Development Scheduler for Flask



In first post (https://angular-python-salesforce.blogspot.com/2020/04/scheduler-jobs-in-backend.html) I've described common approach to implement Scheduler Job. Now I'm going to make it works in local environment for development purposes.

I've updated cli commands structure and now task group contains "scheduler" and "run" scripts:
- scheduler is designed to call from cron and run tasks processing.
- run script is designed to process one particular task

./backend/scripts/task.py
import os
import sys
import click
from flask import Blueprint
from models import db, Task

task_scripts = Blueprint('task_scripts', __name__, cli_group='task')


def run_task_helper(task):
   if current_app.config['is_development_mode']::
        print('RUN TASK LOCALY')
        import subprocess
        subprocess.Popen(['flask', 'task', 'run', str(task.id)])
    else:
        print('RUN TASK IN PRODUCTION ...')


@task_scripts.cli.command("scheduler")
def start_scheduler():

    # EXAMPLE: flask task scheduler

    print('SCHEDULER STARTED')

    tasks = Task.query.all()
    print('==== Tasks:')
    print(tasks)

    for task in tasks:
        run_task_helper(task)

    print('SCHEDULER FINISHED')


@task_scripts.cli.command("run")
@click.argument("task_id")
def run_task(task_id):

    # EXAMPLE: flask task run <TASK_ID>

    print('TASK PROCESSING STARTED:', task_id)

    task = Task.query.filter_by(id=task_id).first()
    print('==== Task:')
    print(task)

    if task is None:
        sys.exit('Task not found')

    task.status = 'Finished'

    db.session.commit()

    print('TASK PROCESSING FINISHED')
The main idea here is usage of subprocess.Popen. It allows to run separate process in background (it doesn't block the main thread) that looks close to what we want to see in production.

And now if we run flask task scheduler it will run N task processes (for all Tasks, but next I will add more complex logic). This command we will use in Heroku Scheduler. 

Also flask task run <task_id> will be used directly to run task processing immediately (with button from UI). For this purpose I've added new route handler

./backend/controllers/task.py
...
@task_bp.route('/run', methods=['POST'])
@login_required
def run_tasks():

    task_payload = request.get_json()
    task = Task.query.filter_by(id=task_payload['id']).first_or_404()

    run_task_helper(task)

    return jsonify({'status': 'ok'})

One more interesting step to emulate real scheduler in local environment is this bash script - it will fire our tasks processing every 10 seconds.

./run_scheruler.sh
#!/bin/bash

echo "Starting Scheruler"

while true; do
    echo "=== Start processing ==="
    pipenv run flask task scheduler
    echo "=== Finish processing ==="
    sleep 10
done

Comments

Post a Comment

Popular posts from this blog

HTTPS in local environment for Angular + Flask project.

Image
Sometimes it is important to run project in local environment with HTTPS. For example when you use third party authentication system in our app. In my case I prefer to have local instance similar as mush as possible to the production. Moreover, it’s easy to implement. First what you need is the ssl key. You can follow this article to create it https://devcenter.heroku.com/articles/ssl-certificate-self Create your ssl key in ./key project folder. Finally it should look like this [dmnbrest@~/Documents/myPython/blog2/project/key]$ ls server.crt server.csr server.key Add HTTPS support to the FLASK: Update init section in your backend server file ( ./backend/app.py ) this way if __name__ == "__main__":     context = ('./key/server.crt', './key/server.key')     app.run(ssl_context=context, host='0.0.0.0', debug=True) But now we need to run backend with other command python ./backend/app.py This time you should see this output in ...
Read more

Task schedule configuration (Cron-like)

Image
I've rethink schedule run flow and decide to focus on Cron like style. User can still select from simple options like 10 mins, 1 hour, 1 day but they will be stored in Cron format. I found great python lib to simplify this work - Croniter ( https://github.com/kiorky/croniter ). It allows us to validate cron expression and calculate next run time. In scheduler we will check next run time and start task processing if time is less or equal and task status is "waiting". ./backend/controllers/task.py ... @task_bp.route('/start', methods=['POST']) @login_required def start_tasks(): task_payload = request.get_json() task = Task.query.filter(Task.id == task_payload['id'], Task.user_id == current_user.id).first_or_404() if not task.config or not croniter.is_valid(task.config['trigger_value']) : abort(400, 'Invalid Trigger Setup') task.status = 'waiting' base = datetime.now...
Read more

Salesforce Lightning Design System (SLDS)

Image
At this moment our UI looks ugly without any styles. To make it looks better we can to use any of existing CSS frameworks. I tried Bootstrap ( https://getbootstrap.com/ ), Foundation ( https://get.foundation/ ), UIkit ( https://getuikit.com/ ). But as a Salesforce developer I found SLDS ( https://www.lightningdesignsystem.com/ ) is closer to my heart. Let's add it to our project. Download last SLDS (current version is 2.11.6) from https://www.lightningdesignsystem.com/downloads/ Extract it to the ./backend/static/slds folder. There are few other ways to add SLDS to the project: - from NPM in the Angular subproject. But in this case it can be accessible by Angular only and is compiled as part of the result bundle. But we have Flask project as global playground and few pages (like /auth/login) should work out of the angular context but have the same look. - external CDN (like  https://cdnjs.com/ ). I like this approach and use it often during development time, but ...
Read more