Elasticsearch with Spring-Data-Elasticsearch Application

Overview 

Elasticsearch is a real-time distributed and open source full-text search and analytics engine. It is document-based search platform with fast searching capabilities. It’s optimized for needle-in-haystack problems rather than consistency or atomicity.
Elasticserach is a huge topic so in this blog i am going to keep it very simple at 101 level. In this blog I will cover how to download Elasticsearch and set it up. Also, how you can use Spring Boot and Spring Data ElasticSearch projects to integrate with Elasticsearch engine.

Setup Elasticsearch

Elasticsearch latest version is 6.5.x as of today. However, I am going to download only 2.4.4 version only. The reason is that spring-data-elasticsearch project is compatible only with 2.4.4 version as of now. If you want to know more details on the compatibility, please check this link - Spring-Data-Elasticsearch---Spring-Boot---version-matrix
If you still want to use the latest Elasticsearch 5.x or 6.x, you would need to use Spring  with Transport Client or Node client apis directly instead of spring-data-elasticsearch. Follow below steps to download and setup elasticsearch engine.
Step 1 - Go to elastic official website https://www.elastic.co/downloads/past-releases
Step 2 - Select Elasticsearch in drop down and then version as 2.4.4 and click on Download button.
Step 3 - It will give you options if you want to download as Zip, TAR, RPM. You can choose as per your convenience. I selected the Zip format as using it on windows.
Step 4 - unzip the downloaded content and go to  bin  folder. There will be a file named elasticsearch.bat.
Step 5 - Run this file on windows OS through command prompt and it will start the elasticsearch engine for you. Once started, it will start listening to port 9200. So the url will be http://localhost:9200/ Also, 9300 port is exposed as cluster node. 9200 port is for REST communication and can be used by Java or any other language and 9300 is for Elasticsearch Cluster Node communication with each other as well Java can also connect to this cluster node using Transport protocol.
Step 6 - Test to verify if it started properly by launching the url using curl command. You can use PowerShell on windows.
curl http://localhost:9200/

Develop Spring Boot Application -

     Now, we will develop a Spring Boot application which will showcase ElasticsearchTemplate and ElasticsearchRepository way of accessing the Elasticsearch engine and do CRUD operations.
    Before we develop the application, let's first understand how ElasticsearchTemplate and ElasticsearchRepository works.
ElasticsearchTemplate - It is a Template class which implements the ElasticsearchOperations. It is more powerful than ElasticsearchRepository as it can do more than CRUD operations. It has operations to create, delete indexes, bulk upload. It can do aggregated search as well.
ElasticsearchRepository -  If we define an interface which extends the ElasticsearchRepositorywhich is provided by Spring data Elasticsearch, it will provide the CRUD operations automatically for that Document. e.g. UserRepository interface is defined below with "User" document by extending ElasticsearchRepository. Now all the find, save, delete, update default operations can be done on User document. If you don't know what is document in Elasticsearch, read below in this blog for more explanation. 

It extends the ElasticsearchCrudRepository which extends eventually the Repository interface. This repository interface is standard feature of Spring data. No need to provide implementation of this interface. You can write customized queries as well by using @Query annotation.
Both of these uses Transport client or Node Client

    Prerequisites

  •   Java 1.8 installed
  •   Eclipse or Visual Studio Code IDE
  •   Maven

 Maven Dependencies   


Configuration -

   application.properties  needs to have spring data properties which is used by ElasticsearchTemplate and ElasticsearchRepository to connect the engine. I have used Transport client properties like cluster-nodes, index name to connect the elasticsearch engine.

Mappings

In Elasticsearch, Index is like a DB in RDBMS and Mappings/Type is similar to a table in RDBMS. Document is a collection of fields belonging to a type and resides in index. Spring data provides annotation like @Document to create document. Here, we define User as a document which has index as "my_index" and type as "user".

Develop Controllers -

First Controller is UserController. It will use UserDAOImpl to have ElasticserachTemplate interacting with Elasticsearch Engine. 

UserDAOImpl - This class initialize elasticsearchtemplate and use queryForList method to retrieve data.

Another Controller is UserRepositoryConroller. This is using UserRepository to interact with elasticsearch engine.

This Repository class extends the ElasticsearchRepository class which internally extends  ElasticsearchCrudRepository ->  PagingAndSortingRepository


You can find the full code at github link -https://github.com/RajeshBhojwani/spring-boot-elasticsearch.git

Build Application


  Application can be build using Maven commands.
 mvn clean install  will build the code and create elasticsearch-0.0.1-SNAPSHOT.jar  file.

Run the application


     java -jar  target/elasticsearch-0.0.1-SNAPSHOT.jar  will start the application. And application will listen to port 8102 as defined in application.properties file.

Test the application -

Test UserController flow which uses ElasticsearchTemplate.

Step 1 - Add new Users. Use this REST API url to add new User  http://localhost:8102/new 

Add Json data in Request body.



Step 2 - Check the response. You will see new User is created with userId generated which is Unique Id for this document. Output will be as below:


Step 3 - Retrieve all Users. Use   http://localhost:8102/all 


Test UserRepositoryController flow which uses ElasticsearchRepository.

Step 1 - Add new Users. Use this REST API url to add new User  http://localhost:8102/repo/new 

Add Json data in Request body as we did with earlier test case.

Step 2 - Check the response. You will see new User is created with userId generated which is Unique Id for this document.

There are many other apis i have put in the code which you can play around. It will showcase how ElasticsearchTemplate can work with indexes, How you can search by field name. With ElasticsearchRepository, you can search by userId as well.

There are many other ways we can connect to Elasticsearch Engine. I will just provide brief introduction about them here as it would require whole new blog to explain in detail.

Spring Boot with Transport Client - As explained above, if you want to use latest elasticsearch engine 5.x or 6.x, spring data elasticsearch would not support that as of now. But Elasticsearch provides RestHighLevelClient  class which works well with latest version and uses RestClient  class to configure the host and port for the cluster. It uses index() method with IndexRequest as input to create a document. It uses get() method with GetRequest as input to search a document. It does have update() and delete() method as well.

Nodejs - If you are not Java/Spring fan, then you have an option of Nodejs as well. What you need to do is download  elasticsearch and get-json module using npm. elasticsearch has Client module which can configure host and port of the elasticsearch server cluster. This Client now have methods like create() to create the index; index() method to put document in that index.

That's all for Elasticsearch integration with Spring data project. Please do let me know your views through comments. Happy to address your queries if I can.

6 comments:

  1. Thank you for this great article!
    I had been trying from a very long time to learn this elastic-search and its integration through spring data.
    Very good working examples!
    :)

    ReplyDelete
  2. Spring data elasticsearch compatible with 2.4.4 elasticsearch version, how to provide security feature(I mean use username/password to connect)? TransportClient available in spring data elasticsearch for spring boot version 1.5.2_RELEASE, doesn't have option. Please help on this.

    ReplyDelete
    Replies
    1. Hi,
      I believe you can use Shield or XPack for securing your Elastic Search connectivity.
      There are some threads on stackoverflow which covers that part.
      https://stackoverflow.com/questions/42838459/connecting-to-xpack-enabled-elasticsearch-5-x-via-spring-data-elasticsearch

      https://stackoverflow.com/questions/33628214/elasticsearch-http-authentication-in-spring

      Delete
  3. I am getting 'The method findOne(Example<>) in the type QueryByExampleExecutor is not applicable for the arguments (String)' error in UserRepositoryController class in findOne() method, it is suggesting me to use findById() method.

    User user = userRepository.findOne(userId);

    Please help me on this.

    ReplyDelete
    Replies
    1. you can use either of findById or findOne. It depends on which version of Spring data elasticsearch you are using. If you go for the higher version, findOne are converted to findOneBy.

      But ensure you read the compatibility issues I shared in the other blog (link above).

      I hope it helps.

      Delete
  4. When I originally commented I clicked the -Notify me when new comments are added- checkbox and now each time a comment is added I get four emails with the same comment. Is there any way you can remove me from that service? Thanks!
    New

    ReplyDelete

Most Viewed Posts