How to properly index a large amount of information in elasticsearch?

  • 0
    Created a command in Laravel, I get all the goods in it
    $ prod = Prod :: cursor ();
    I iterate over and write to elasticsearch
    $this->elasticsearch->index([
      'index' => 'prods',
      'type' => 'prod',
      'id' => $prod->prods_id,
      'body' => [
        'site_id' => $prod->site_id,
        'category_id' => $prod->category_id,
        'fields' => $prod->field,
        'priority' => $prod->priority,
        'prod_updated_at' => $prod->updated_at_p,
        'prod_created_at' => $prod->created_at_p,
        'site_rating' => $prod->site_rating,
        'prod_rating' => $prod->prod_rating,
      ],
    ]);

    More than 2 million records in the database
    On average, elasticsearch records 23,000 products per hour, which means that it will take about 4 days to write the entire database.

    Is it possible to write in a faster way?
    Laravel Paige Duarte, Oct 22, 2020

  • 3 Answers
  • 0
    Although I didn't work much with ES and it seemed strange to me that the index should be built on such a number of fields. What is the practical point of this? Why not store in ES only what needs to be searched and / or analyzed? After all, for other cases, you can use a relational DBMS.

    Mash. translation about indices and cardinality
    Anonymous

  • 0
    More than 2 million records in the database

    On average, elasticsearch records 23,000 products per hour, which means that it will take about 4 days to write the entire database.




    I have two questions, one of which is a bit off-topic.

    1) what server is it running on and what does top show?

    2) why are you not satisfied with 4 days for primary filling the database? Next, will you add only new entries?
    Anonymous

  • 0
    Use Bulk API to load into dozens times faster than one document at a time.



    True, it's also not worth throwing everything in one bulk, practice shows that packages of the order of hundreds of documents at a time are quite, although it also depends on the size of one document.
    Anonymous

Your Answer
To place the code, please use CodePen or similar tool. Thanks you!