ZQuery1.SortedFields: Not correctly sorted Russian character

[alexs75]

ZQuery1.SortedFields: Not correctly sorted Russian characters in UTF8

Lazarus 1.3 r44337M FPC 2.6.2 i386-win32-win32/win64
ZEOS_VERSION = '7.2.0-alfa';

[EgonHugeist]

Hi Boris,

This is a thread i was waiting for.. The problem is NOT the russian characters, nope the problem is a {LAZARUSUTF8HACK} define (exists since years) and a problem of UTF8 'em selves. If you OS is a Windows with codepage 1251 f.e. than the AnsiStrCompare functions do return a wrong results for UTF8 encoded strings. Simply the byteorder doesn't match your expected results. IMO the only way to get it perfectly runnning: Converty to Wide/Unicode up and make a Wide-Compare...

Other proposals?

I'll check this the next weeks.

Michael

[alexs75]

I'm create fix for this error.
On Windows for FPC i'm use compare procedure from LazUTF8 - work fine.

[EgonHugeist]

Did check you patch!

Inital idea sound great but this compare is LCL available only. I did check the UTF8 functions the LCL guys have been made.. Nice, fast but incomplete for the asiann character sets. IMHO do we run into next issue again.

Propose we make a UTF8 to Wide conversion, call WideUpper/LowerCase if required(casesensitive or not) and make a WideCompareStr. AFAIK most compile targets have these algorithmns. Do you agree?

[alexs75]

Won't this slow down? Lazarus works in UTF8. The conversion will be slow.

[EgonHugeist]

Yes it will a bit. But this is also happens with with the LCL units.

I can't apply this patch "as is" since it's simply wrong. Zeos isn't made for LCL+FPC only. You can also use all other charsets with FPC. It's just the evel LCL which allways assumes all strings are UTF8.
If i would apply the idea than all other sorts using different codepages would fail. I'll invest some more time next days (since our quicksort could also be faster if we use a hybrid algorithm f.e.).

And as i wrote the LCL units are incomplete for all fareast ideoms... The code you're using only handles UTF8 untail plain 3 and no surrogates are included...

[EgonHugeist]

Just a note:

Did commit a fix R3302 \testing-7.2 (SVN). Hope you test it?

[alexs75]

Not compiled:

Code: Select all

Free Pascal Compiler version 2.7.1 [2014/09/15] for x86_64
Copyright (c) 1993-2014 by Florian Klaempfl and others
(1002) Target OS: Linux for x86-64
/usr/local/share/lazarus/components/zeos/src/core/ZSysUtils.pas(3049,7) Hint: (4055) Conversion between ordinals and pointers is not portable
/usr/local/share/lazarus/components/zeos/src/core/ZSysUtils.pas(71,33) Hint: (5024) Parameter "SortList" not used
/usr/local/share/lazarus/components/zeos/src/core/ZSysUtils.pas(3042,52) Fatal: Internal error 200706094
Fatal: (1018) Compilation aborted
Error: /usr/local/bin/ppcx64 returned an error exitcode
Компиляция пакета zcore 7.2: stopped with exit code 256

[bigchimp]

That's an internal error in the FPC compiler which should basically not happen.... so it has nothing to doe with Zeos (except that the Zeos code triggers the error condition).

If it hasn't been reported in the FPC bug tracker, it's a good idea to do so, so the FPC devs can take a look at it.

[EgonHugeist]

Did add a define in ZeosLazarus.inc {$DEFINE WITH_IE200706094} and implemented a workaround again.

Patch done R3313 \testing-7.2 (SVN)

@BigChimp

Might be true. But we are trying to handle all FPC related bugs as well as we do it for Delphi (here we've only problems with inline code, and some issues with D7/D2005).

Just study ZeosLazarus.inc i could point you to some more issues, if you doubt.

[alexs75]

Thank you. Works correctly.