Use an index name defined by the entity to store data in Spring Data Elasticsearch 4.0

When using Spring Data Elasticsearch (I am referencing the current version 4.0.2), normally the name of the index where the documents are stored is taken from the @Document annotation of the entity class – here it’s books:

@Document(indexName="books")
public class Book {
  // ...
}

Recently in a discussion of a Pull Request in Spring Data Elasticsearch, someone told that she needed a possibility to extract the name from the entity itself, as entities might go to different indices.

In this post I will show how this can be done by using Spring Data Repository customization by providing a custom implementation for the save method. A complete solution would need to customize saveAll and other methods as well, but I will restrict this here to just one method.

The Hotel entity

For this post I will use an entity describing a hotel, with the idea that hotels from different countries should be stored in different Elasticsearch indices. The index name in the annotation is a wildcard name so that when searching for hotels all indices are considered.

Hotel.java

package com.sothawo.springdataelastictest;

import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;
import org.springframework.lang.Nullable;

/**
 * @author P.J. Meisch (pj.meisch@sothawo.com)
 */
@Document(indexName = "hotel-*", createIndex = false)
public class Hotel {
    @Id
    @Nullable
    private String id;

    @Field(type = FieldType.Text)
    @Nullable
    private String name;

    @Field(type = FieldType.Keyword)
    @Nullable
    private String countryCode;

    // getter/setter ...
}

The custom repository

We need to define a custom repository interface that defines the methods we want to implement. Since we want to customize the save method that ElasticsearchRepository has by extending CrudRepository, we need to use the very same method signature including the generics:

CustomHotelRepository.java

package com.sothawo.springdataelastictest;

/**
 * @author P.J. Meisch (pj.meisch@sothawo.com)
 */
public interface CustomHotelRepository<T> {
    <S extends T> S save(S entity);
}

The next class to provide is an implementation of this interface. It is important that the implementation class is named like the interface with a Impl suffix:

CustomHotelRepositoryImpl.java

package com.sothawo.springdataelastictest;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.data.elasticsearch.core.ElasticsearchOperations;
import org.springframework.data.elasticsearch.core.IndexOperations;
import org.springframework.data.elasticsearch.core.document.Document;
import org.springframework.data.elasticsearch.core.mapping.IndexCoordinates;
import org.springframework.lang.NonNull;
import org.springframework.lang.Nullable;

import java.util.concurrent.ConcurrentHashMap;

/**
 * @author P.J. Meisch (pj.meisch@sothawo.com)
 */
@SuppressWarnings("unused")
public class CustomHotelRepositoryImpl implements CustomHotelRepository<Hotel> {

    private static final Logger LOG = LoggerFactory.getLogger(CustomHotelRepositoryImpl.class);

    private final ElasticsearchOperations operations;

    private final ConcurrentHashMap<String, IndexCoordinates> knownIndexCoordinates = new ConcurrentHashMap<>();
    @Nullable
    private Document mapping;

    @SuppressWarnings("unused")
    public CustomHotelRepositoryImpl(ElasticsearchOperations operations) {
        this.operations = operations;
    }

    @Override
    public <S extends Hotel> S save(S hotel) {

        IndexCoordinates indexCoordinates = getIndexCoordinates(hotel);
        LOG.info("saving {} to {}", hotel, indexCoordinates);

        S saved = operations.save(hotel, indexCoordinates);

        operations.indexOps(indexCoordinates).refresh();

        return saved;
    }

    @NonNull
    private <S extends Hotel> IndexCoordinates getIndexCoordinates(S hotel) {

        String indexName = "hotel-" + hotel.getCountryCode();
        return knownIndexCoordinates.computeIfAbsent(indexName, i -> {

                IndexCoordinates indexCoordinates = IndexCoordinates.of(i);
                IndexOperations indexOps = operations.indexOps(indexCoordinates);

                if (!indexOps.exists()) {
                    indexOps.create();

                    if (mapping == null) {
                        mapping = indexOps.createMapping(Hotel.class);
                    }

                    indexOps.putMapping(mapping);
                }
                return indexCoordinates;
            }
        );
    }
}

This implementation is a Spring Bean (no need for adding @Component) and so can use dependency injection. Let me explain the code.

Line 22: the ElasticsearchOperations object we will use to store the entity in the desired index, this is autowired by constructor injection in lines 29-31

Line 24-26: As we want to make sure that the index we write to exists and has the correct mapping, we keep track of which indices we already know. This is used in the getIndexCoordinates method explained later.

Line 34 to 44: This is the actual implementation of the save operation. First we call getIndexCoordinates which will make sure the index exists. We pass the indexCoordinates into the save method of the ElasticsearchOperations instance. If we would use ElasticsearchOperations.save(hotel), the name from the @Document annotation would be used. But when passing an IndexCoordinates as second parameter, the index name from this is used to store the entity. In line 41 there is a call to refresh, this is the behaviour of the original ElasticsearchRepository.save() method, so we do the same here. If you do not need the immediate refresh, omit this line.

Line 47 to 76: As Spring Data Elasticsearch does not yet support index templates (this will come with version 4.1) this method ensures, that when the first time that an entity is saved to an index, this index is created if necessary and writes the mappings to the new created index.

The HotelRepository to use in the application

We now need to combine our custom repository with the ElasticsearchRepository from Spring Data Elasticsearch:

HotelRepository.java

package com.sothawo.springdataelastictest;

import org.springframework.data.elasticsearch.core.SearchHits;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

/**
 * @author P.J. Meisch (pj.meisch@sothawo.com)
 */
public interface HotelRepository extends ElasticsearchRepository<Hotel, String>, CustomHotelRepository<Hotel> {
    SearchHits<Hotel> searchAllBy();
}

Here we combine the two interfaces and define an additional method that returns all hotels in a SearchHits object.

Use the repository in the code

The only thing that’s left is to use this repository, for example in a REST controller:

HotelController.java

package com.sothawo.springdataelastictest;

import org.springframework.data.elasticsearch.core.SearchHits;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * @author P.J. Meisch (pj.meisch@sothawo.com)
 */
@RestController
@RequestMapping("/hotels")
public class HotelController {

    private final HotelRepository repository;

    public HotelController(HotelRepository repository) {
        this.repository = repository;
    }

    @GetMapping()
    public SearchHits<Hotel> all() {
        return repository.searchAllBy();
    }

    @PostMapping()
    public Hotel save(@RequestBody Hotel hotel) {
        return repository.save(hotel);
    }
}

This is a standard controller which has a HotelRepository instance injected (which Spring Data Elasticsearch will create for us). This looks exactly how it would without our customization. The difference is that the call to save() ends up in our custom implementation.

Conclusion

This post shows how easy it is to provide custom implementations for the methods that are normally provided by Spring Data Repositories (not just in Spring Data Elasticsearch) if custom logic is needed.

mapjfx 2.15.0 and 1.33.0 released adding circles and OpenLayers 6.4.2

I just released mapjfx versions 1.33.0 and 2.15.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.33.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.15.0</version>
  </dependency>

1.33.0 is built using Java 8 and 2.15.0 uses Java 11.

Circles can now be added to a map, giving the center coordinates and the radius in meters with custom coloring and transparency, thanks to Hanwoo Kim for this contribution!

The OpenLayers version now is 6.4.2.

How to provide a dynamic index name in Spring Data Elasticsearch using SpEL

In Spring Data Elasticsearch – at the time of writing, version 4.0 is the current version – the name of an index is normally defined by the @Document annotation on the entity class. For the following examples let’s assume we want to write some log entries to Elasticsearch with our application. We use the following entity:

@Document(indexName = "log")
public class LogEntity {
    @Id
    private String id = UUID.randomUUID().toString();

    @Field(type = FieldType.Text)
    private String text;

    @Field(name = "log-time", type = FieldType.Date, format = DateFormat.basic_date_time)
    private ZonedDateTime logTime = ZonedDateTime.now();

    public String getId() {
        return id;
    }

    public String getText() {
        return text;
    }

    public void setText(String text) {
        this.text = text;
    }

    public ZonedDateTime getLogTime() {
        return logTime;
    }

    public void setLogTime(ZonedDateTime logTime) {
        this.logTime = logTime;
    }
}

Here the index name is the fixed name log.

it is possible to use a dynamically defined name for an index by using Spring Expression Language SpEL. Important: We need to use a SpEL template expression, that is an expression enclosed in #{}. This allows for the following setups:

Use a value from the application configuration

Let’s assume we have the following entry in the application.properties file:

index.prefix=test

We then use this code

@Document(indexName = "#{@environment.getProperty('index.prefix')}-log")

and the index name to use changes to test-log.

Use a value provided by a static method of some class

The second example shows how to call a static function to get a dynamic value. We use the following definition to add the current date to the index name:

@Document(indexName = "log-#{T(java.time.LocalDate).now().toString()}")

Currently this would provide an index name of log-2020-07-28.

Use a value provided by a Spring bean

For the third case we provide a bean that will give us a dynamically created string to be used as part of the index name.

@Component
public class LogIndexNameProvider {

    public String timeSuffix() {
        return LocalTime.now().truncatedTo(ChronoUnit.MINUTES).toString().replace(':', '-');
    }
}

This bean, named logIndexNameProvider, can return a String that contains the current time as hh-mm (I would not use this for naming indices, but this is just an example).

Changing the definition to

@Document(indexName = "log-#{@logIndexNameProvider.timeSuffix()}")

will now create index names like log-08-25 or log-22-07.

Of course we can mix all of these together: add a prefix from the configuration, append the current date.

Important notice:

The evaluation of SpEL for index names is only done for the index names defined in the @Document annotation. It is not done for index names that are passed as a IndexCoordinates parameter in the different methods of the ElasticsearchOperations or IndexOperations interfaces. If it were allowed on these, it would be easy to set up a scenario, where an expression is read from some outside source. And then someone might send something like "log-#{T(java.lang.Runtime).getRuntime().exec(new String[]{'/bin/rm', '/tmp/somefile'})}" which will not provide an index name, but delete files on your computer.

mapjfx 1.32.0 and 2.14.0 adds the ability to rotate markers and labels

I just released mapjfx versions 1.32.0 and 2.14.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.32.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.14.0</version>
  </dependency>

1.32.0 is built using Java 8 and 2.14.0 uses Java 11.

Markers and Labels on a map now have a rotation property which will rotate the corresponding HTML Element. The values goes from 0 to 360 and defines the rotating angle clockwise.

mapjfx 1.31.1 and 2.13.1 fixing event triggering regression

I just released mapjfx versions 1.31.1 and 2.13.1, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.31.1</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.13.1</version>
  </dependency>

1.31.1 is built using Java 8 and 2.13.1 uses Java 11.

These versions fix a regression where events for markers and labels (mouse enter/leave, context click etc) where not created anymore due to enabling scroll/zoom behaviour on markers and labels.
Now all events are dispatched again as they should.

mapjfx 1.30.0 and 2.11.0 with new Bing Maps MapTypes

I just released mapjfx versions 1.30.0 and 2.11.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.30.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.11.0</version>
  </dependency>

1.30.0 is built using Java 8 and 2.11.0 uses Java 12.

In these version 4 new variants for the Bing Map mapType are added (thanks to https://github.com/MalteBahr)

mapjfx 2.5.0 and 1.24.0 released

I just released mapjfx versions 1.24.0 and 2.5.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.24.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.5.0</version>
  </dependency>

Now slf4j is used as logging framework and zooming by using the scrollwheel now works when the mouse is over a marker or label.

Version 1.24.0 is built using OracleJDK8, whereas version 2.5.0 is built with OpenJDK11.

support for alternate map projections in mapjfx

The latest versions of mapjfx – 1.22.0 and 2.3.0 – have the possibility to have different projections for the map used in OpenLayers. The possible projections are:

  • Web Mercator (EPSG:3857); this is used if no other projection is specified and is the projection that was hardcoded up to now
  • WGS84 (EPSG:4326); this can be set when initializing the MapView object.

Check the mapjfx-demo description and the source code. The actual versions are available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.22.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.3.0</version>
  </dependency>

The two screenshots show the different map projections:

EPSG:4326

EPSG:3857

mapjfx now supports filled polygons

I just released mapjfx versions 1.21.0 and 2.2.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.21.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.2.0</version>
  </dependency>

Now it is possible to close CoordinateLines to polygons with a fill color:


Comments and contributions welcome.

mapjfx 2.1.1 and 1.20.1 using OpenLayers 5.3.0

I just released mapjfx versions 1.20.1 and 2.1.1, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.20.1</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.1.1</version>
  </dependency>

mapjfx now uses OpenLayers version 5.3.0.

Version 1.20.1 is built using OracleJDK8, whereas version 2.1.1 is built with OpenJDK11.