Since Elasticsearch was the center of the long Intercom, the update is a difficult problem. For all versions of the upgrade, you need to be completely invisible to you. To quote the AaronBrady of Shopify, "Our customers, our that to upgrade the infrastructure should not be punished by  our decision."

There are many best elasticsearch course available. These are clusters of single-purpose product teams independently owned. The team believes that there is a need for its infrastructure. This gives you control over indexing, mapping, type, and other decisions without being blocked by the bureaucracy for approval, allowing you to act quickly and optimize in the long run (of the value of core engineering). 

We focus on delivering great products at high speeds, and our product teams must be empowered to do so. We strive to ensure that these product teams are always quick, safe and easy to build great things using our infrastructure services. For this reason, we decided to focus this upgrade work on a single infrastructure team.

So as decided to upgrade all Elasticsearch clusters, starting with the oldest cluster that was running version 2.3.3. There were 54 versions of Elasticsearch between versions 2.3.3 and 6.3.0. The long jump had to land perfectly.

Reasons to keep Elasticsearch up to date:

Before starting this project, I asked myself, "Why do I need to keep Elasticsearch up to date?" 

• Running different versions of Elasticsearch is not optimal for several reasons.

• It is difficult for engineers to debug problems with individual clusters because they cannot transfer their knowledge and context of how one cluster works to another cluster.

• It is not possible to have a generic tool common to all clusters because all clusters support different APIs.  The older the cluster, the harder it is to upgrade.

Running older versions of Elasticsearch (and popular software) is not OK for the following reasons:

• Security: Building an intercom with software or systems at risk of misuse is unacceptable to our customers.

• Productivity: Most software improves over time as it adds features, fixes bugs, and supports new approaches. Many software tends to get faster over time.

• Maintainability: Providing software updates ensures that your team understands how to create, test, and deploy their software.

The benefits of continuous integration at Elasticsearch Upgrade

An amazing continuous integration (CI) pipeline that ships more than 100 times to production on a good day. To achieve this safely, all code changes are exposed to an ever-growing number of units and integration test batteries. At the time of this writing, we are running 47,080 tests per CI build. By upgrading Elasticsearch for the first time with CI, we were able to remove most of the significant changes that affected a particular use of Elasticsearch. This helped me find mapping changes, invalid cluster settings, old query syntax, and more. I provided a fix and repeated it here until CI passed both Elasticsearch 2.3.3 and Elasticsearch 6.3.0.

I know CI test coverage is really good, but it's not perfect, and it's never perfect. So, before moving on, I took a week of actual ingestion and search traffic and repeated them for a test cluster using Elasticsearch 6.3.0. We used this mechanism to verify that all requests from old and new versions of Elasticsearch responded the same but if you are still not able to understand ES then take elasticsearch training.

Step-by-step upgrade process

Elasticsearch has excellent documentation on version upgrades. However, major release upgrades before Elasticsearch 6.0 indicate that the cluster must be completely restarted. We needed a zero downtime, minimal risk upgrade that would allow us to revert to the old cluster as soon as we saw the first signs of the problem.

Our process is a two-step process. First updated from 2.3.3 to 5.6.9, then from 5.6.9 to 6.3.0. This was necessary to use snapshots and restores. How to update Elasticsearch from version 2.3.3. Up to 6.3.0.

1. Set up a new 6.3.0 cluster on the same hardware and activate dual writes.

2. Take a snapshot of the old 2.3.3 cluster.

3. Set up a temporary 5.6.9 cluster.

• Restore from the 2.3.3 snapshot created in the previous step.

• This has been reindexed in Elasticsearch 5x format.

• Take another snapshot.

4. Cluster should be deleted

• Restore to the new temporary index from the 5.6.9 snapshot taken in the previous step.

• Delete the temporary index.

5. At this point, you have two identical Elasticsearch clusters. They have the same hardware, the same data, and both are kept in sync. You can now switch the read load to the new cluster and dismantle the old cluster at any time.

6. Clear double write and create a new cluster with reading and write operation permissions. We can now delete all "deleted" documents whose Boolean field is set to "true". We were using the delete_by_query API for this.

Finally, we can withdraw the old cluster.

Elasticsearch 5x versions focus on ingestion and search performance. The Elastic blog promises to "increase indexing throughput by 25-80%," which is exactly what we saw after applying the two fixes above. Most clusters have improved average indexing and search latency by more than 50% and reduced average CPU utilization by 40%.