How to provide a dynamic index name in Spring Data Elasticsearch using SpEL

In Spring Data Elasticsearch – at the time of writing, version 4.0 is the current version – the name of an index is normally defined by the @Document annotation on the entity class. For the following examples let’s assume we want to write some log entries to Elasticsearch with our application. We use the following entity:

@Document(indexName = "log")
public class LogEntity {
    @Id
    private String id = UUID.randomUUID().toString();

    @Field(type = FieldType.Text)
    private String text;

    @Field(name = "log-time", type = FieldType.Date, format = DateFormat.basic_date_time)
    private ZonedDateTime logTime = ZonedDateTime.now();

    public String getId() {
        return id;
    }

    public String getText() {
        return text;
    }

    public void setText(String text) {
        this.text = text;
    }

    public ZonedDateTime getLogTime() {
        return logTime;
    }

    public void setLogTime(ZonedDateTime logTime) {
        this.logTime = logTime;
    }
}

Here the index name is the fixed name log.

it is possible to use a dynamically defined name for an index by using Spring Expression Language SpEL. Important: We need to use a SpEL template expression, that is an expression enclosed in #{}. This allows for the following setups:

Use a value from the application configuration

Let’s assume we have the following entry in the application.properties file:

index.prefix=test

We then use this code

@Document(indexName = "#{@environment.getProperty('index.prefix')}-log")

and the index name to use changes to test-log.

Use a value provided by a static method of some class

The second example shows how to call a static function to get a dynamic value. We use the following definition to add the current date to the index name:

@Document(indexName = "log-#{T(java.time.LocalDate).now().toString()}")

Currently this would provide an index name of log-2020-07-28.

Use a value provided by a Spring bean

For the third case we provide a bean that will give us a dynamically created string to be used as part of the index name.

@Component
public class LogIndexNameProvider {

    public String timeSuffix() {
        return LocalTime.now().truncatedTo(ChronoUnit.MINUTES).toString().replace(':', '-');
    }
}

This bean, named logIndexNameProvider, can return a String that contains the current time as hh-mm (I would not use this for naming indices, but this is just an example).

Changing the definition to

@Document(indexName = "log-#{@logIndexNameProvider.timeSuffix()}")

will now create index names like log-08-25 or log-22-07.

Of course we can mix all of these together: add a prefix from the configuration, append the current date.

Important notice:

The evaluation of SpEL for index names is only done for the index names defined in the @Document annotation. It is not done for index names that are passed as a IndexCoordinates parameter in the different methods of the ElasticsearchOperations or IndexOperations interfaces. If it were allowed on these, it would be easy to set up a scenario, where an expression is read from some outside source. And then someone might send something like "log-#{T(java.lang.Runtime).getRuntime().exec(new String[]{'/bin/rm', '/tmp/somefile'})}" which will not provide an index name, but delete files on your computer.

Using geo-distance sort in Spring Data Elasticsearch 4

The release of Spring Data Elasticsearch in version 4.0 (see the documentation) brings two new features that now enable users to use geo-distance sorts in repository queries: The first is a new class GeoDistanceOrder and the second is a new return type for repository methods SearchHit<T>. In this post I will show how easy it is to use these classes to answer questions like “Which pubs are the nearest to a given location?”.

The source code

The complete runnable code used for this post is available on GitHub. In order to run the application you will need Java 8 or higher and a running instance of Elasticsearch. If this is not accessible at localhost:9200 you need to set the correct value in the src/main/resources/application.yaml file.

Update 12.09.2020

The original code was a little extended for the follow-up post Search entities within a geographic distance with Spring Data Elasticsearch 4

The sample data

For this sample application I use a csv file with POI data from OpenStreetMap that contains POIs in Germany which are categorized as kind of food, like restaurants, pubs, fast-food and more. All together there are 826843 records.

When the application is started, the index in Elasticsearch is created and loaded with the data if it does not yet exist. So the first startup takes a little longer, the progress can be seen on the console. Within the application, these POIs are modelled by the following entity:

@Document(indexName = "foodpois")
public class FoodPOI {
    @Id
    private String id;
    @Field(type = FieldType.Text)
    private String name;
    @Field(type = FieldType.Integer)
    private Integer category;
    private GeoPoint location;
    // constructors, getter/setter left out for brevity
}

The interesting properties for this blog post are the location and the name.

The Repository

In order to search the data we need a Repository Definition:

public interface FoodPOIRepository extends ElasticsearchRepository<FoodPOI, String> {
    List<SearchHit<FoodPOI>> searchTop3By(Sort sort);
    List<SearchHit<FoodPOI>> searchTop3ByName(String name, Sort sort);
}

We have two functions defined, the first we will use to search any POI near a given point, with the second on we can search for the POIs with a name. Defining these methods in the interface is all we need as Spring Data Elasticsearch will under the hood create the implementation for these methods by analyzing the method names and parameters.

In Spring Data Elasticsearch before version 4 we could only get a List<FoodPOI> from a repository method. But now there is the SearchHit<T> class, which not only contains the entity, but also other values like a score, highlights or – what we need here – the sort value. When doing a geo-distance sort, the sort value contains the actual distance of the POI to the value we passed into the search.

The Controller

We define a REST controller, so we can call our application to get the data. The request parameters will come in a POST body that will be mapped to the following class:

public class RequestData {
    private String name;
    private double lat;
    private double lon;
    // constructors, getter/setter ...
}

The result data that will be sent to the client looks like this:

public class ResultData {
    private String name;
    private GeoPoint location;
    private Double distance;

  // constructor, gette/setter ...
}

The controller has just one method:

@RestController
@RequestMapping("/foodpois")
public class FoodPOIController {

    private final FoodPOIRepository repository;

    public FoodPOIController(FoodPOIRepository repository) {
        this.repository = repository;
    }

    @PostMapping("/nearest3")
    List<ResultData> nearest3(@RequestBody RequestData requestData) {

        GeoPoint location = new GeoPoint(requestData.getLat(), requestData.getLon());
        Sort sort = Sort.by(new GeoDistanceOrder("location", location).withUnit("km"));

        List<SearchHit<FoodPOI>> searchHits;

        if (StringUtils.hasText(requestData.getName())) {
            searchHits = repository.searchTop3ByName(requestData.getName(), sort);
        } else {
            searchHits = repository.searchTop3By(sort);
        }

        return searchHits.stream()
            .map(searchHit -> {
                Double distance = (Double) searchHit.getSortValues().get(0);
                FoodPOI foodPOI = searchHit.getContent();
                return new ResultData(foodPOI.getName(), foodPOI.getLocation(), distance);
            }).collect(Collectors.toList());
    }
}

In line 15 we create a Sort object that specifies that Elasticsearch should return the data ordered by the geographical distance to the given value which we take from the request data. Then, depending if we have a name, we call the corresponding method and get back a List<SearchHit<FoodPOI>>.

We then in the lines 27 to 29 extract the information we need from the returned objects and build our result data object.

Check the result

After starting the application we can hit the endpoint. I use curl here and pipe the output through jq to have it formatted:

$curl -X "POST" "http://localhost:8080/foodpois/nearest3" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{
  "lat": 49.02,
  "lon": 8.4
}'|jq

[
  {
    "name": "Cantina Majolika",
    "location": {
      "lat": 49.0190808,
      "lon": 8.4014792
    },
    "distance": 0.14860088197123017
  },
  {
    "name": "Waldgaststätte FSSV",
    "location": {
      "lat": 49.023578,
      "lon": 8.3954656
    },
    "distance": 0.5173117164589114
  },
  {
    "name": "Hatz",
    "location": {
      "lat": 49.0155358,
      "lon": 8.3975457
    },
    "distance": 0.5276800664204232
  }
]

And the Pubs?

curl -X "POST" "http://localhost:8080/foodpois/nearest3" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{
  "lat": 49.02,
  "lon": 8.4,
  "name": "pub"
}'|jq
[
  {
    "name": "Scruffy's Irish Pub",
    "location": {
      "lat": 49.0116335,
      "lon": 8.3950194
    },
    "distance": 0.998711100164643
  },
  {
    "name": "Irish Pub “Sean O'Casey's”",
    "location": {
      "lat": 49.0090639,
      "lon": 8.4028365
    },
    "distance": 1.2335132790824628
  },
  {
    "name": "Oxford Pub",
    "location": {
      "lat": 49.0086149,
      "lon": 8.4129781
    },
    "distance": 1.5806674447458173
  }
]

And that’s it

Without even needing to know how these request are sent to Elasticsearch and what Elasticsearch sends back, we can easily use these features in our Spring application. Hope you enjoyed it!

 

 

 

mapjfx display problems update

For the last two years a problem was coming up occasionally with some users, that only the top left area of the map is displayed, and the rest is not loading:

Thanks to the analysis of Martin Stiel in this comment and Victor Ewert in issue 81 it seems that this can be traced to a problem when running the application on a high resolution display.

Alas I cannot reproduce this as I have no hardware with a resolution that might be high enough. So if you have this problem you might try the solution that Victor mentions in the issue linked above: start the application with -Dprism.allowhidpi=false.

So currently I cannot support this as for em this problem is not reproducible, But I’d be glad for feedback or any new information.

 

Update:

https://www.sothawo.com/2020/09/mapjfx-display-problem-on-windows-10-seems-solved/

mapjfx 1.32.0 and 2.14.0 adds the ability to rotate markers and labels

I just released mapjfx versions 1.32.0 and 2.14.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.32.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.14.0</version>
  </dependency>

1.32.0 is built using Java 8 and 2.14.0 uses Java 11.

Markers and Labels on a map now have a rotation property which will rotate the corresponding HTML Element. The values goes from 0 to 360 and defines the rotating angle clockwise.

mapjfx 1.31.1 and 2.13.1 fixing event triggering regression

I just released mapjfx versions 1.31.1 and 2.13.1, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.31.1</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.13.1</version>
  </dependency>

1.31.1 is built using Java 8 and 2.13.1 uses Java 11.

These versions fix a regression where events for markers and labels (mouse enter/leave, context click etc) where not created anymore due to enabling scroll/zoom behaviour on markers and labels.
Now all events are dispatched again as they should.

mapjfx 2.13.0 switches back to use Java 11 (LTS)

I just released mapjfx version 2.13.0, it will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.13.0</version>
  </dependency>

I switched back to use Java 11 (LTS) as SceneBuilder is only available for Java 11 (https://gluonhq.com/products/scene-builder/) and libraries built with version 12 or 13 cannot be used in SceneBuilder 11.

mapjfx 1.30.0 and 2.11.0 with new Bing Maps MapTypes

I just released mapjfx versions 1.30.0 and 2.11.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.30.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.11.0</version>
  </dependency>

1.30.0 is built using Java 8 and 2.11.0 uses Java 12.

In these version 4 new variants for the Bing Map mapType are added (thanks to https://github.com/MalteBahr)

mapjfx 1.29.0 and 2.10.0 now with OpenLayers 6.0.1 and extension constrains

I just released mapjfx versions 1.29.0 and 2.10.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.29.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.10.0</version>
  </dependency>

1.29.0 is built using Java 8 and 2.10.0 uses Java 12.

This version updates the OpenLayers library to version 6.0.1 (https://github.com/sothawo/mapjfx/issues/66).
Additionally it is now possible to constrain the panning and zooming of a map to a given extent (https://github.com/sothawo/mapjfx/issues/62).

mapjfx 2.9.0 and 1.28.0 with offline cache improvements

I just released mapjfx versions 1.28.0 and 2.9.0, they will be available in maven central:

  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>1.28.0</version>
  </dependency>
  <dependency>
    <groupId>com.sothawo</groupId>
    <artifactId>mapjfx</artifactId>
    <version>2.9.0</version>
  </dependency>

1.28.0 is bilt using Java 8 and 2.9.0 uses Java 12.

These releases add the possibility to pass a list of URLs (map tiles) to the OfflineCache to have the cache populated in the background with the data from these URLs (https://github.com/sothawo/mapjfx/issues/64)

Example:

private void initOfflineCache() {
    final OfflineCache offlineCache = OfflineCache.INSTANCE;
    offlineCache.setCacheDirectory(FileSystems.getDefault().getPath("tmpdata/cache"));
    offlineCache.setActive(true);
    offlineCache.setNoCacheFilters(Collections.singletonList(".*\\.sothawo\\.com/.*"));

    LinkedList<String> urls = new LinkedList<>();
    urls.add("https://c.tile.openstreetmap.org/14/8572/5626.png");
    urls.add("https://b.tile.openstreetmap.org/14/8571/5626.png");
    urls.add("https://a.tile.openstreetmap.org/14/8572/5625.png");
    urls.add("https://c.tile.openstreetmap.org/14/8571/5625.png");
    urls.add("https://b.tile.openstreetmap.org/14/8570/5625.png");
    urls.add("https://a.tile.openstreetmap.org/14/8572/5625.png");
    urls.add("https://a.tile.openstreetmap.org/14/8570/5626.png");
    urls.add("https://a.tile.openstreetmap.org/14/8571/5627.png");
    urls.add("https://a.tile.openstreetmap.org/14/8573/5626.png");
    urls.add("https://a.tile.openstreetmap.org/14/8574/5627.png");
    urls.add("https://b.tile.openstreetmap.org/14/8571/5626.png");
    urls.add("https://b.tile.openstreetmap.org/14/8573/5625.png");
    urls.add("https://b.tile.openstreetmap.org/14/8572/5627.png");
    urls.add("https://b.tile.openstreetmap.org/14/8574/5626.png");
    urls.add("https://c.tile.openstreetmap.org/14/8572/5626.png");
    urls.add("https://c.tile.openstreetmap.org/14/8570/5627.png");
    urls.add("https://c.tile.openstreetmap.org/14/8574/5625.png");
    urls.add("https://c.tile.openstreetmap.org/14/8573/5627.png");

    offlineCache.preloadURLs(urls);
}