4.5. Analysis

Procedure 4.1. The Analyzer Process

1. The CharFilters process the character input.
2. Tokenizer converts the character input into tokens.
3. The tokens are the processed by the TokenFilters.

Procedure 4.2. @AnalyzerDef and the Solr framework

1. Configure the CharFilter

   Define a CharFilter by factory. In this example, a mapping CharFilter is used, which will replace characters in the input based on the rules specified in the mapping file.

   @AnalyzerDef(name = "customanalyzer",
       charFilters = {
           @CharFilterDef(factory = MappingCharFilterFactory.class, params = {
               @Parameter(name = "mapping",
                   value = 
                       "org/hibernate/search/test/analyzer/solr/mapping-chars.properties")
           })
       },

   Copy to Clipboard Toggle word wrap

2. Define the Tokenizer

   A Tokenizer is then defined using the StandardTokenizerFactory.class.

   @AnalyzerDef(name = "customanalyzer",
       charFilters = {
           @CharFilterDef(factory = MappingCharFilterFactory.class, params = {
               @Parameter(name = "mapping",
                   value = 
                       "org/hibernate/search/test/analyzer/solr/mapping-chars.properties")
           })
       },
     
       tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class)

   Copy to Clipboard Toggle word wrap

3. List of Filters

   Define a list of filters by their factories. In this example, the StopFilter filter is built reading the dedicated words property file. The filter will ignore case.

   @AnalyzerDef(name = "customanalyzer",
       charFilters = {
           @CharFilterDef(factory = MappingCharFilterFactory.class, params = {
               @Parameter(name = "mapping",
                   value =
                       "org/hibernate/search/test/analyzer/solr/mapping-chars.properties")
           })
       },
     
       tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
       filters = {
                    
           @TokenFilterDef(factory = ISOLatin1AccentFilterFactory.class),
           @TokenFilterDef(factory = LowerCaseFilterFactory.class),
           @TokenFilterDef(factory = StopFilterFactory.class, params = {
               @Parameter(name = "words",
                   value= "org/hibernate/search/test/analyzer/solr/stoplist.properties" ),
               @Parameter(name = "ignoreCase", value = "true")
           })
       })
   public class Team {
   }

   Copy to Clipboard Toggle word wrap

Procedure 4.3. Configure the @AnalyzerDiscriminator

1. Predefine Dynamic Analyzers

   The @AnalyzerDiscriminator requires that all analyzers that are to be used dynamically are predefined via @AnalyzerDef. The @AnalyzerDiscriminator annotation can then be placed either on the class, or on a specific property of the entity, in order to dynamically select an analyzer. An implementation of the Discriminator interface can be specified using the @AnalyzerDiscriminator impl parameter.

   @Indexed
   @AnalyzerDefs({
       @AnalyzerDef(name = "en",
           tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
           filters = {
               @TokenFilterDef(factory = LowerCaseFilterFactory.class),
               @TokenFilterDef(factory = EnglishPorterFilterFactory.class)
           }),
       @AnalyzerDef(name = "de",
           tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
           filters = {
               @TokenFilterDef(factory = LowerCaseFilterFactory.class),
               @TokenFilterDef(factory = GermanStemFilterFactory.class)
           })
       })
   public class BlogEntry {
   
       @Field
       @AnalyzerDiscriminator(impl = LanguageDiscriminator.class)
       private String language;
     
       @Field
       private String text;
     
       private Set<BlogEntry> references;
     
       // standard getter/setter    
   }

   Copy to Clipboard Toggle word wrap

2. Implement the Discriminator Interface

   Implement the getAnalyzerDefinitionName() method, which is called for each field added to the Lucene document. The entity being indexed is also passed to the interface method.
   The value parameter is set if the @AnalyzerDiscriminator is placed on the property level instead of the class level. In this example, the value represents the current value of this property.

   public class LanguageDiscriminator implements Discriminator {
       public String getAnalyzerDefinitionName(Object value, Object entity, String field) {
           if (value == null || !(entity instanceof Article)) {
               return null;
           }
           return (String) value;
       }
   }

   Copy to Clipboard Toggle word wrap