design patterns - Workflow System with Azure Table Storage -


i have system need run simple workflow. example:

  1. on jan 1st 08:15 trigger task object z
  2. when triggered run code (implementation details not important)
  3. schedule task b object z run @ jan 3rd 10:25 (and on)

the workflow simple, need run 500.000+ instances , that's tricky part.

i know windows workflow foundation , same reason have chosen not use that.

my initial design use azure table storage , appreciate feedback on design.

the system consist of 2 tables

table "jobs"   partitionkey: objectid   rowkey: processon (utc ticks in reverse newest on top)   attributes: state (pending, processed, error, skipped), etc...  table "timetable"   partitionkey: yyyymmdd   rowkey: yyyymmddhhmm_<guid>   attributes: job_partitionkey, job_rowkey

the idea runs table have complete history of jobs per object , timetable have list of jobs run in future.

some assumptions:

  • a job never span more 1 object
  • there ever 1 pending job per object
  • the "job" lightweight e.g. posting message queue

the system must able perform these tasks:

  • execute pending jobs

    1. query records in "timetable" "partition <= today" , "rowkey <= today"
    2. for each record (in parallel)
      1. lookup job in jobs table via partitionkey , rowkey
      2. if "not exists" or state != pending skip
      3. execute "logic". if fails => log , maybe retry logic
      4. submit "next run date in timetable"
      5. submit "update state = processed" , "new job record (next run)" single transaction
    3. when finished => delete processed timetable records

    concern: 2 of 3 records modifications in transaction. overcome in way?

  • stop workflow stop/pause workflow object z

    1. query top 1 jobs in jobs table partitionkey
    2. if , state == pending update "cancelled"
    3. (no need bother cleaning timetable clean "when time comes")

  • start workflow

    1. create pending record in jobs table
    2. create record in timetable

in terms of "executing thing" using azure function or scheduler-thing execute pending jobs every 5 minutes or so.

any comments or suggestions highly appreciated.

thanks!

how using service bus instead? brokeredmessage class has property called scheduledenqueuetimeutc. can schedule when want jobs run via scheduledenqueuetimeutc property, , fuggedabouddit. can have triggered webjob monitors service bus messaging queue, , triggered near when job message enqueued. i'm big fan of relying on existing services minimize coding needed.


Comments

Post a Comment

Popular posts from this blog

javascript - Clear button on addentry page doesn't work -

this addentry blog page have made scratch , there underlying problems code particularly clear button supposed clear text entered when button clicked on doesn't work have saved html file problem. also 1 other problem submit button works doesn't pass of text index file have written php , reason whenever click on submit button redirects me index page without posting entry have made. <!doctype html> <html> <head> <script> function message() { alert("clear form?"); } </script> <style> body {background-color: beige;} h1 {color: black;} p {color: black;} textarea { width: 50%; height: 150px; padding: 12px 20px; box-sizing: border-box; border: 2px solid #ccc; border-radius: 4px; background-color: white; font-size: 16px; resize: none; } div { padding-left: 20px; } </style> </head> <meta charset="utf-8"> <link rel="stylesheet" href=".
Read more

c# - Selenium Authentication Popup preventing driver close or quit -

i have selenium automated task setup login website retrieve information. it's basic in automation. logging in , clicking 2 links. running issue proving problem. the site in question uses authentication required popup not javascript popup rather regular windows auth popup. have read isn't handled out of box selenium. i using " http://username:password@url.com " site every , site has issues on end prompt me again username/password randomly. have enter username/password twice in row when navigating there hand. since task loops continuously few hours picking needed info figured use webdriverwait verify element present on page. if authentication popup appears, element isn't present. no need care run, start over. the problem having when authentication popup displayed , hit timeout limit want close driver/browser window can start task on again. however, driver.close() , driver.quit() not doing anything. browser remains open , unloaded page still there login p
Read more

tensorflow when input_data MNIST_data , zlib.error: Error -3 while decompressing: invalid block type -

i'm geting started tensorflow,followed tutorials,but cannot download mnist data. import tensorflow tf tensorflow.examples.tutorials.mnist import input_data mnist=input_data.read_data_sets("mnist_data/",one_hot=true) print("download done!") error information: extracting mnist_data/train-images-idx3-ubyte.gz traceback (most recent call last): file "/home/sj/program/tesorflow_try.py", line 6, in <module> mnist=input_data.read_data_sets("mnist_data/",one_hot=true) file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 213, in read_data_sets train_images = extract_images(f) file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 60, in extract_images buf = bytestream.read(rows * cols * num_images) file "/usr/lib/python2.7/gzip.py", line 268, in read self._read(readsize) file
Read more