Best way to queue or thread CPU intensive tasks so Node server doesn't crash?

Question

I wrote code that forks and queues up and stops child processes but the processes themselves use up quite a bit of CPU.

Is there a library that's particularly helpful with building these types of async/cron-like Node services? Wasn't finding anything in my search.

Should I consider offloading that to another server or is it just time to up the CPU of my virtual machine?

Hugo · Accepted Answer

There's cool UIs to monitor the queues as well, I've got an example of a working setup at https://github.com/HugoDF/express-bull-es6 (post explaining what's going on at https://codewithhugo.com/bring-redux-to-your-queue-logic-an-express-setup-with-es6-and-bull-queue/).

Bull is the only queue with a modern interface (ie Promises) that I found when digging around.

Let me know if you get stuck anywhere.

Hugo · Answer

You should try using a queue eg. Bull queue. 
You can run the workers on the same server or somewhere else (it's backed by redis).
Not sure if it 100% fits your use case since as far as I understand you seem to want to wait for the job to complete before sending a response back or something? In which case you should wait on the output of the queued job (that writes to a db or something)

Corey Gwin · Answer

Bull looks like exactly what I need. I don't have redis setup for my project but shouldn't be too tough. Thanks dude!!! 🙌 Will let you know how it goes.

akshay kadam (a2k) · Answer

Although I have never used it or anything, I've heard about forking child processes. I think this talk covers it - https://www.youtube.com/watch?v=K8spO4hHMhg

Not sure if it'll help though. You should also try asking in the subreddit /r/nodejs

Go to Homepage	`g` `h`
Go to Done Todos	`g` `d`
Go to Questions	`g` `q`
Compose a New Todo	`n`
Go to Search	`/`
Show this dialog	`?`

Best way to queue or thread CPU intensive tasks so Node server doesn't crash?

Keyboard Shortcuts