Searching with NOT equals with special characters

Question

Hi everyone

Question: Is there a way to search for documents that do not match an exact value with special characters in Yuuvis?

Background Info:
I am trying to search for documents which do not have a specific value set. As far as I could figure out from the documentation, the standard filters or search does not support the NOT operator.

In the expert search however, I can query for NOT str_type_field:"value". This works perfectly fine if there are no special characters in my value. But the value I need to search for has forward slashes, brackets and potentially other characters inside.
I was able to figure out that if I simply delete the forward slashes and the value contains no brackets, the search will work. But as soon as the value contains brackets I could not figure out a way to search for it and get results.

Examples
Document Value: /Europe/Switzerland/Zürich/
Failing Search Query: str_type_field:"/Europe/Switzerland/Zürich/"
Successful Search Query: str_type_field:"Europe Switzerland Zürich"

Document Value: /Europe/Switzerland/Zürich (ZH)/
Failing Search Query: str_type_field:"/Europe/Switzerland/Zürich (ZH)/"
Failing Search Query: str_type_field:"Europe Switzerland Zürich (ZH)"
Failing Search Query: str_type_field:"Europe Switzerland Zürich ZH"

The special characters work fine when doing an exact "eq" search in the normal search mode.

I'm figuring that it probably has something to do with how the Elastic Search Index is made on Yuuvis side. I would probably want to prevent this behavoir by adding a "not_analyzed" to a field: https://www.elastic.co/guide/en/elasticsearch/guide/current/_finding_exact_values.html

My desired behavoir would be to search for the exact value (including special characters) in the field, but with NOT equals.

We are currently testing Yuuvis RAD 6.0 BETA, but I had the same behavior with Yuuvis RAD 5.20 (and was hoping that it will change with the BETA version).

Kind regards
Nicole

Answer 1

Hello Nicole,

we don't support NOT-filter, but you can use for it our expert search. If you need to use any of the characters which function as operators in your expert query itself (and not as operators), then you should escape them with a leading backslash. The reserved characters are: + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /

For instance, to search for „/Europe/Switzerland/Zürich/“ , you would need to write your query as „\/Europe\/Switzerland\/Zürich\/“. When using JSON for the request body, two preceding backslashes (\) are required; the backslash is a reserved escaping character in JSON strings.

Kind regards
Irina

Comments

Hello Irina

I'm sorry, but your example does not work. So staying with the same example I can expand my tests as follows:
Document Value: /Europe/Switzerland/Zürich/
Failing Search Query: str_type_field:"\/Europe\/Switzerland\/Zürich\/"
Failing Search Query: str_type_field:"/Europe/Switzerland/Zürich/"
Successful Search Query: str_type_field:"Europe Switzerland Zürich"

Isn't escaping only relevant when the searched value is not quoted? Otherwise searching for str_type_field:"Europe Switzerland Züri*" would return a result aswell, but it does not as the * is evaluated literally to my knowledge.

Kind regards
Nicole

---

Hello Nicole,

you don’t need to use the double-quote. For instance, in your case:  

str_type_field:\/Europe\/Switzerland\/Zürich\/

It’s more complecate with wildcard in the expert query because  the special field should be used. For instance,

str_type_field.idxs:\/Europe\/Switzerland\/Zür*  

You can use this field for the first search too:

str_type_field.idxs:\/Europe\/Switzerland\/Zürich\/

Kind regards
Irina

PS I tested your query with Yuuvis RAD 6.0 BETA

---

Hello Irina

When I search for str_type_field:\/Europe\/Switzerland\/Zürich\/ this also finds the documents tagged with other values.
For example with 3 documents:
Document 1: /Europe/Switzerland/Zürich/
Document 2: /Europe/Switzerland/
Document 3: /Europe/Switzerland 2/
I would only expect it to find Document 1, but all three documents are found.

But the special field is a lot of help! When I use .idxs I can search as expected, also using quotes without escaping anything. Is there any documentation for this special field ".idxs"? What exactly is in this field?

Thanks
Nicole

---

Hello Nicole,

the expert search is the fulltext search and  it’s correct that you found  other documents too. The search against idxs-field is the exact diacritics-normalized search that we use for index data search. If you need such search you can use our standard query, in this case you don’t need to escape anything. For instance,

{"filters": { "type.field " : "/Europe/Switzerland/Zürich/" }}

Best regards
Irina

---

Hi Irina

When using wildcards, I am still experiencing problems. Using the following example documents:
Document 1 Value: /Europe/Switzerland/Zürich/
Document 2 Value: /Europe/Switzerland/Zürich (ZH)/
Document 3 Value: /Europe/Switzerland/Zürich/Winterthur/
Document 4 Value: /something/different/

I would like to search for all paths starting with /Europe/Switzerland/Zürich/
Expected: Document 1 and 3

If I try to search with the expert search in the .idxs field with wildcards:
Query: str_dmsdoc_marketpaths.idxs:(\/Europe\/Switzerland\/Zürich\/*)
Result: All documents (and without the brackets a lot more is returned as I have other documents (and types) in the system)

Query: str_dmsdoc_marketpaths.idxs:"/Europe/Switzerland/Zürich/*"
Returns: No results

Exact matching would work fine:
Query: str_dmsdoc_marketpaths.idxs:\/Europe\/Switzerland\/Zürich\/
Result: Document 1, as expected

Query: str_dmsdoc_marketpaths.idxs:"/Europe/Switzerland/Zürich/"
Result: Document 1, as expected


With the standard query, I have similar problems with special characters. Using the same example documents:
Search for /Europe/Switzerland/Zürich/* -> "filters":{"dmsdoc.marketpaths":{"o":"in","v1":["/Europe/Switzerland/Zürich/*"]}}
Result: Finds all documents

Search for "/Europe/Switzerland/Zürich/*" -> "filters":{"dmsdoc.marketpaths":{"o":"in","v1":["\"/Europe/Switzerland/Zürich/*\""]}}
Result: Finds document 1, 2 and 3


I have found no way so far, how I can just retrieve document 1 and 3.

Kind regards
Nicole