How to properly index a large amount of information in elasticsearch?

• 0
  Created a command in Laravel, I get all the goods in it
  $ prod = Prod :: cursor ();
  I iterate over and write to elasticsearch
  

  $this->elasticsearch->index([
    'index' => 'prods',
    'type' => 'prod',
    'id' => $prod->prods_id,
    'body' => [
      'site_id' => $prod->site_id,
      'category_id' => $prod->category_id,
      'fields' => $prod->field,
      'priority' => $prod->priority,
      'prod_updated_at' => $prod->updated_at_p,
      'prod_created_at' => $prod->created_at_p,
      'site_rating' => $prod->site_rating,
      'prod_rating' => $prod->prod_rating,
    ],
  ]);

  
  More than 2 million records in the database
  On average, elasticsearch records 23,000 products per hour, which means that it will take about 4 days to write the entire database.
  
  Is it possible to write in a faster way?

  Laravel Paige Duarte, Oct 22, 2020

---

3 Answers
• 0
  Although I didn't work much with ES and it seemed strange to me that the index should be built on such a number of fields. What is the practical point of this? Why not store in ES only what needs to be searched and / or analyzed? After all, for other cases, you can use a relational DBMS.
  
  Mash. translation about indices and cardinality

  Anonymous

---

• 0

  More than 2 million records in the database
  
  On average, elasticsearch records 23,000 products per hour, which means that it will take about 4 days to write the entire database.

  
  
  
  
  I have two questions, one of which is a bit off-topic.
  
  1) what server is it running on and what does top show?
  
  2) why are you not satisfied with 4 days for primary filling the database? Next, will you add only new entries?

  Anonymous

---

• 0
  Use Bulk API to load into dozens times faster than one document at a time.
  
  
  
  True, it's also not worth throwing everything in one bulk, practice shows that packages of the order of hundreds of documents at a time are quite, although it also depends on the size of one document.

  Anonymous

---