Full Unicode/Ansi-Support in /testing branch

The alpha/beta tester's forum for ZeosLib 7.0.x series

Report problems concerning our Delphi 2009+ version and new Zeoslib 7.0 features here.

This is a forum that will be removed once the 7.X version goes into stable!!

Moderators: gto, EgonHugeist, olehs

Locked
olehs
Zeos Dev Team
Zeos Dev Team
Posts: 118
Joined: 09.11.2009, 21:05

Post by olehs »

EgonHugeist,
and could you please insert some much more complicated tests, with accedend chars like 'üöäü'? that would be great!
I'll do it.
Do we need now GetServerSetting(), FStandartConfirmingStrings, function EncodeString and so on?? What do you think?
Look at test I gave you earlier (procedure TZTestPostgreSQLTokenizer.TestQuoteState;)

Tokenizer has to know what rules will be applied by the server and parse strings the same.

Look at my modified tokenizer.

Code: Select all

{**
  Checks whether escape syntax is used.

  @return a True if escape syntax is used.
}
function TZPostgreSQLQuoteState.CheckEscapeSyntax(Stream: TStream): Boolean;
var
  ReadChar: Char;
begin
  Result := not FStandardConformingStrings;
  if FStandardConformingStrings then
  begin
    Stream.Seek(-SizeOf(Char), soFromCurrent);
    if Stream.Position >= SizeOf(Char) then
    begin
      Stream.Seek(-SizeOf(Char), soFromCurrent);
      Stream.Read(ReadChar, SizeOf(Char));
      Result := UpperCase(ReadChar) = 'E';
      Stream.Seek(SizeOf(Char), soFromCurrent);
    end;
  end;
end;

{**
  Return a quoted string token from a reader. This method
  will collect characters until it sees a match to the
  character that the tokenizer used to switch to this state.

  @return a quoted string token from a reader
}
function TZPostgreSQLQuoteState.NextToken(Stream: TStream;
  FirstChar: Char; Tokenizer: TZTokenizer): TZToken;
const BackSlash = Char('\');
var
  ReadChar: Char;
  LastChar: Char;
  QuoteCount: Integer;
  EscapeSyntax: Boolean;
begin
  Result.Value := FirstChar;
  QuoteCount := 1;
  EscapeSyntax := False;
  if FirstChar = '"' then
  begin
    Result.TokenType := ttWord;
  end
  else
  begin
    Result.TokenType := ttQuoted;
    EscapeSyntax := CheckEscapeSyntax(Stream);
  end;

  LastChar := #0;

  while Stream.Read(ReadChar, SizeOf(Char)) > 0 do
  begin
    if ReadChar = FirstChar then
      Inc(QuoteCount);

    if (LastChar = FirstChar) and (ReadChar <> FirstChar) then
    begin
      if QuoteCount mod 2 = 0 then
      begin
        Stream.Seek(-SizeOf(Char), soFromCurrent);
        Break;
      end;
    end;
    Result.Value := Result.Value + ReadChar;
    if (LastChar = BackSlash) and EscapeSyntax then
      LastChar := #0
    else if (LastChar = FirstChar) and (ReadChar = FirstChar) then
      LastChar := #0
    else LastChar := ReadChar;
  end;
end;
It does what we just need. All we have to do is set FStandardComformingStrings to a valid value.
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

@olehs,

did commit a fix for your test according Ansi-IDE's and UTF8 Strings.

Also did i manipulate the EscapeString function again. I created an overload for D12_UP and added my ZPlainString function to the Ansi IDE's to avoid bad encoded escape behavior.
Look at my modified tokenizer.
This i understand. Other possibility is you create a ttWord childstate. "E" is a word state. If you check the next char for "\" or"'" then go on reading and change the result state(dependend on StandartConfirmingStrings)...

This detection brings up a new question: Which state will this detected sequence have?

miab3, thank you for your confirmation.

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
olehs
Zeos Dev Team
Zeos Dev Team
Posts: 118
Joined: 09.11.2009, 21:05

Post by olehs »

EgonHugeist,
Other possibility is you create a ttWord childstate. "E" is a word state. If you check the next char for "\" or"'" then go on reading and change the result state(dependend on StandartConfirmingStrings).
Yes, it's also a good algorythm. But it's only "'" you have to check. "\" can't follow E.

Ok. There's even more. How about this query :twisted:

Code: Select all

select $$Test string$$
or even

Code: Select all

select $Nice_name_for_tag$Test string 2$Nice_name_for_tag$
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

olehs,

this is slightly annoying. Hmpf if i do understand all this correctly do you slightly rebuild the whole tokenizer of PostgreSQL. Which is needed for user generated external sended statments.

I don't know and didn't expect such a syntax...

Our issues are now only the external sended Statements after Fixing the Escape behavior for Parameter values, right?

Next thing i don't like: example of stUnicodeStream.
First we do UTF8ToString(in Blob.GetUnicodeString)- then on EscapeString(UTF8Encode(Value), PlainEscaping(AnsiValue), UTF8ToString(AnsiValue))-> then send Statment: UTF8Encode(on sending the query).

What about speed decrease here? Only to be 100% unicode safe? Isn't that a stupid sh...it?

I planned to avoid such stupid things, an overloaded Statement execution for D12_UP. You can find some abstract definitions in ZDbcStatement.pas. But i had not the time to complete this. IF that would be ready then we encode the strings only once and no issues to see. (The reason for my overload of today)

Which doesn't solves the issues from external sended statments, except the user uses this new overload..

Sure you understand what i mean.

What do you think?

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
olehs
Zeos Dev Team
Zeos Dev Team
Posts: 118
Joined: 09.11.2009, 21:05

Post by olehs »

EgonHugeist,
I don't know and didn't expect such a syntax...
It's not a problem, I can add this processing. It's gonna be very useful for DBMS developers etc.
Sure you understand what i mean.
I'm not sure I understand, I need to look deeper...
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

olehs,

It's not a problem, I can add this processing. It's gonna be very useful for DBMS developers etc.
That would be great. We really need this token behavior to stabalize everything. And i've the impression you can help us here!
I'm not sure I understand, I need to look deeper...
Actully do not need to invest time there. This i think is my part(as long i've the time to complete everything). Teamwork...

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
miab3
Zeos Test Team
Zeos Test Team
Posts: 1309
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

@EgonHugeist

Zeos7_branches_ testing_r1637 compiles and runs on D2006, DXE2 32/64, LazarusWin1.1.0/fpc 2.7.1 32/64. (With a small exception on Oracle AQ$_QUEUES in DXE2-32).

Michal
miab3
Zeos Test Team
Zeos Test Team
Posts: 1309
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

@EgonHugeist

Zeos7_branches_ testing_r1640 compiles and runs on D2006, DXE2 32/64, LazarusWin1.1.0/fpc 2.7.1 32/64. (With a small exception on Oracle AQ$_QUEUES in DXE2-32).

Michal
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

miab3,

:lol: Thank you Michal, for reminding us according the Nested issues.

@Michal, Olehs,
I'm daddy since 03.08.2012 and soo happy about. So don't wait for reply the next days, ok? But i'm still coding in the background..

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
miab3
Zeos Test Team
Zeos Test Team
Posts: 1309
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

@Michael

As I suspected something because of your silence. Congratulations.

Michal
miab3
Zeos Test Team
Zeos Test Team
Posts: 1309
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

@EgonHugeist

Zeos7_branches_ testing_r1642 compiles and runs on D2006, DXE2 32/64, LazarusWin1.1.0/fpc 2.7.1 32/64. ( Oracle as usual :) )

Michal
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

miab3,

Thank you Michal... :) The oracle isse is still on the todo's.

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
miab3
Zeos Test Team
Zeos Test Team
Posts: 1309
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

@EgonHugeist

Zeos7_branches_ testing_r1644
http://svn.code.sf.net/p/zeoslib/code-0 ... s/testing/
compiles and runs on (I have tested):

- D2006,
- D2007/C++,
- DXE2 32/64 / C++32,
- Lazarus Win 1.1/fpc 2.7.1 32/64,
- Lazarus Lin 1.1/fpc 2.7.1 64 on Debian 64-bit (LMDE 201204).

I enclose three small patches ( tagged as M.A.) for older versions of C++ Builder.

Michal Abramczyk
You do not have the required permissions to view the files attached to this post.
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

miab3,

thank you for testing the branches. Applyed your patch. Rev. 1646.

Michal can you use TortoisSVN next times instead? Then you can easiely generate a *.patch file and i do not need to compare the files for diff's..

Thank you,

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
markus
Senior Boarder
Senior Boarder
Posts: 58
Joined: 17.10.2011, 12:43
Location: Piotrków Trybunalski, Poland

Post by markus »

Hi,

I've encountered problem with PrepareSQL property of ZConnection.

My query is like this:

Code: Select all

select *,null as typ_upustu_tmp, null as czas_trwania_tmp,null as thumb  from (SELECT * from data.klienci
)q WHERE (imie ~* 'ed') AND (usr_status >= 0)
but when PrepareSQL is set to true PostgreSQL receives query like this:

Code: Select all

2012-08-18 13:52:01 CEST [127.0.0.1(49731)]/prosalon prosalon_apo STATEMENT:  select *,null as typ_upustu_tmp, null as czas_trwania_tmp,null as thumb  from (SELECT * from data.klienci
	)q WHERE (imie ~ 'ed') AND (usr_status >= 0)
difference is imie ~* 'ed' versus imie ~ 'ed'.

whit PrepareSQL=false query send to postgres is correct and i got expected results.

Best regards,
Marek
Locked