New ask Hacker News story: Ask HN: Best distributed job processing system in 2019?
Ask HN: Best distributed job processing system in 2019?
3 by sharmi | 5 comments on Hacker News.
I have around 10 million network I/O related jobs that I would like to do in a short period. So I hope to use a job queue to run it distributed on several servers. I have used celery in the past but it is not quite reliable. Features most important to me are multiple retries, restarting workers that are not responding, ability to monitor status of the queue and workers. Nice to have features - cron scheduling, task chaining, high throughput. Which is the most stable, reliable job queue out there? It would be preferable to support workers in multiple languages. The ones I would prefer are Python and Go. I have used celery in the past, but workers often hang
3 by sharmi | 5 comments on Hacker News.
I have around 10 million network I/O related jobs that I would like to do in a short period. So I hope to use a job queue to run it distributed on several servers. I have used celery in the past but it is not quite reliable. Features most important to me are multiple retries, restarting workers that are not responding, ability to monitor status of the queue and workers. Nice to have features - cron scheduling, task chaining, high throughput. Which is the most stable, reliable job queue out there? It would be preferable to support workers in multiple languages. The ones I would prefer are Python and Go. I have used celery in the past, but workers often hang
No comments