What is the best way to organize a lazy image processing queue?

  • 0
    I get images via php. Then they need to be cut / reduced.
    I want to make the thresher on a separate server with the possibility of making a pool of such threshers in the future. As well as a pool of servers that receive images and process logic. And so that you can pull / insert any server at any time.
    How would you advise to organize processing?
    To pump out the logic from the server to a random thresher on the crown? Or send a task to a control server and then transfer images to them?

    It is a little scary to use and administer several ready-made solutions (such as a queue in radishes or a group).
    I think it's an overhead.

    Maybe just slap the server for java processing? In general, I am grateful for any advice on organization / software
    Java Arabella Beard, Jun 3, 2020

  • 2 Answers
  • 0
    1) RabbtiMQ | Kafka | ActiveMQ | whatever. We put messages into the queue for processing, remember about the restrictions on the size of the message.

    2) Self-written solution on crutches and scotch tape, using a database.

    3) Make a decoupling in the form of a data storage service and a rest interface. PHP accepts the image, saves it, sends a rest request to process the message lying on the server. LB dynamically distributes requests to threshers.

    4) Any variants.

    But there is actually a question. Are there many of those images? Maybe you shouldn't bother?

  • 0
    I would make some kind of dispatcher server with storage and processing queue (it doesn't matter, MQ or something self-written with the base). His tasks:

    1. Interaction with clients - accept input data and put them in a queue for processing (in response, we issue some id), display the processing status (queue, in progress, ready, error) according to the passed id and the result when ready

    2. Interaction with processing servers - transfer received and ready-to-start tasks one by one to requests that the server is ready to process a task, receive results or error messages, monitor the operation of servers (if a server takes too long to process a task or does not transmit readiness requests , notify the admin, and set the hung task as starting over so that another server can intercept it).

    In principle, you can put several dispatchers so that the processing server polls them in turn (for example, the main and the backup).

Your Answer
To place the code, please use CodePen or similar tool. Thanks you!