Full Unicode/Ansi-Support in /testing branch

[Kosvalery]

EgonHugeist,
Thanks for help. TZReadOnlyQuery and TZQuery.Options.doPreferPrepared working.
But аs before, if TZQuery.Options.doPreferPreparedResolver=true and TZConnection.ClientCodepage=utf8, an attempt to post the edited string date causes exception "Unsupported data type". If ClientCodepage is cp1251, then doPreferPreparedResolver=true working.

[EgonHugeist]

Kosvalery,
hopefully last round of your reported problems..

doPreferPreparedResolver fixed for Unicode. And you're right. This should speed up the functionality!!

If you found other issues: Report them!!!!

Can you confirm the unicode fix for this option?

King regards,

EgonHugeist

btw. thank you for investing your time...

[Kosvalery]

EgonHugeist,
Now working doPreferPreparedResolver with utf8 ClientCodepage with MySQL. But, sometimes happens exceptions "AccessViolation" in ZeosLib if TZConnection.ClientCodapge=cp1251 and doPreferPreparedResolver and doPreferPrepared is set. Most often this occurs in TZGenericSQLWordState.NextToken. Sometimes exceptions happens when colosing application.

Btw. Not work TZQuery.Prepare and UnPrepare procedures with MySQL. They needed for batch insert/update operations by TZQuery.ExecSQL. This becomes understandable with TZSQLMonitor.

[EgonHugeist]

Wow random problems alike, hmpff.

Any ideas? Some better debug results?

Try to figure out what happens with the Prepare/UnPrepare.. wondering that our testsuites don't tell us more here..
And i can't rebuild this issue. So i need the sequence you're using concerning theses bug's.

[s]I'll chage this post if i can tell you more.
[/s]
Proposal: Can you write a pm(click pm below ) and attach some code snippets? Makes the job easier..

Michael

[ism]

Maybe do all operations in zeos only in utf8 ?
In Delphi 9 must be tools to work with unicode

Get with ansi but convert for work inside in zeos

For example crossplatform ZUtf8String and functions

ZUTF8Copy
ZUTF8Length

like this

http://wiki.freepascal.org/LCL_Unicode_Support

[EgonHugeist]

That's absolutly unwanted for me. Complete unicode connections mean's allways speed decrease for the servers and workstations. My patch makes native ansi and utf8 connections possible for all delphi compilers. For FPC i can't say if it would be possible. Actually this isn't because of the LzarusUTF8Hack define in the sources. I've never tried to undefine this hack because i don't know about the LCL behavior for native 1Byte character sets...

[Sorien]

Hi, don't know if its exactly problem with your branch but i have problems to run database in embedded mode after switching to your branch, could you look at it pls...

Edit:
i've probably found it

Code: Select all

ZUrl.pas

procedure TZURL.SetURL(const Value: string);
var
..
begin
  APrefix := '';
  AProtocol := '';
  AHostName := 'localhost'; is preventing to use db in embedded mode, should be changed to AHostName := ''

[EgonHugeist]

Nope the issue was hopefully another one. The 'localhost' should be supported

Seems some '\\' where missed at beginning of the URL if your Localhost or Database wasn't set.

Rev. 1165. If that patch doesn't help. I'll commit yours.

Michael

[Sorien]

Nope the issue was hopefully another one. The 'localhost' should be supported

Seems some '\\' where missed at beginning of the URL if your Localhost or Database wasn't set.

Rev. 1165. If that patch doesn't help. I'll commit yours.

Michael

thx, seems to be working

[Sorien]

next huge bug is in tokenizer's date token detection when you set preparesql to true and system format to "d.M.rrrr" during processing query like "INSERT INTO FOO VALUES ('foo faaf faa ýľ .foo.')" tokenizer will return 'foo faaf faa ýľ .foo.' as ttDate (couse ttDate token detection is based on two "dots") and will not be properly converted to utf8 representaion and fb server returns malformed string exception

[EgonHugeist]

Sorien,
I can't follow you here.

Your example syntax tells me you've a table Foo with one varchr/char column, right? Otherwise your insert fails. In my mind this query is valid and should be handled fine. But you point me to Date/DateTime fields, right too?

I don't want to say there is no bug, but i need a better example and more details about your config. (just give me the real query and tell me which comiler, db-engine, clientcodepage you're using here). Because the plaindriver generic tokonizer shouldn't return a ttData token. That's making me curious.

Michael

[Sorien]

codepage is set to UTF as whole DB

just look at ZGenericSqlToken.pas TZGenericSQLQuoteState.NextToken, ttDate token detection is based on 2 "." which means that string like 'aa.bb.cc' will be detected wrong as date its not a proble until you will use some unicode chars

TZAbstractStatement.GetPrepreparedSQL in ZDbcStatement.pas
will convert only tokens ttWord, ttQuoted, ttQuotedIdentifier, ttKeyword

as far as ttdate token is not converted by your function ZAnsiString query it will be sent to server in malformed reprezentation.

Code: Select all

my version of token detection
ZGenericSqlToken.pas

function TZGenericSQLQuoteState.NextToken(Stream: TStream;
  FirstChar: Char; Tokenizer: TZTokenizer): TZToken;
var
  ReadChar: Char;
  LastChar: Char;
  CountDoublePoint,CountSlash : integer;
  DecodedString: string;
  DummyDateTime: TDateTime;
begin
  Result.Value := FirstChar;
  LastChar := #0;
  CountDoublePoint := 0;
  CountSlash := 0;

  while Stream.Read(ReadChar, SizeOf(Char)) > 0 do
  begin
    if (LastChar = FirstChar) and (ReadChar <> FirstChar) then
    begin
      Stream.Seek(-SizeOf(Char), soFromCurrent);
      Break;
    end;
    Result.Value := Result.Value + ReadChar;
    if (LastChar = FirstChar) and (ReadChar = FirstChar) then
      LastChar := #0
    else LastChar := ReadChar;
  end;

  if FirstChar = '"' then
    Result.TokenType := ttWord
  else Result.TokenType := ttQuoted;

  DecodedString := DecodeString(Result.Value,'"');

  if TryStrToDateTime(DecodedString, DummyDateTime) then
  begin
    Result.TokenType := ttDateTime;
    Result.Value := DecodedString;
  end else
  if TryStrToTime(DecodedString, DummyDateTime) then
  begin
    Result.TokenType := ttTime;
    Result.Value := DecodedString;
  end else
  if TryStrToDate(DecodedString, DummyDateTime) then
  begin
    Result.TokenType := ttDate;
    Result.Value := DecodedString;
  end;
end;

[EgonHugeist]

Sorien,

Your'e right the Plaindrivers giving me the descendant of the TTokenizer back.

If sequences like 'aa.bb.cc' where detected as ttDate/ttTime/ttDateTime then this is a new bug or am i wrong?. Because the result should be ttQuoted or not? Am i wrong here? Are sequences like "d.M.rrrr" db valid? Or shouldn't we also test for [0..9] between the quotes and dots?

I think respecting only ttWord, ttQuoted, ttQuotedIdentifier, ttKeyword to use the UTF8Encode function should be the right way. Thinking on CPU-time. Do you agree?

It is easy to say encode everything which isn't my EscapeState or not?

What do you think? My last proposal should be 5 min work to commit the change.

[Sorien]

it is bug in tokenizer, old or new don't know i'm using zeos lib like few weeks, im just curious why there is so primitive check for date, or why tokenizer tries to detect date field when its not used at any other place in library

[marcov]

That's absolutly unwanted for me. Complete unicode connections mean's allways speed decrease for the servers and workstations. My patch makes native ansi and utf8 connections possible for all delphi compilers. For FPC i can't say if it would be possible. Actually this isn't because of the LzarusUTF8Hack define in the sources. I've never tried to undefine this hack because i don't know about the LCL behavior for native 1Byte character sets...

Lazarusutf8hack is something like using TNT components with D7..d2007. The visual parts of lazarus expect UTF8, just like you must pass unicode to the TNT components.

But instead of TNT components using widestring, Lazarus stores utf8 in ansistrings, so it is less obviously visible.