Page 22 of 44

Posted: 30.07.2012, 14:16
by olehs
EgonHugeist,
and could you please insert some much more complicated tests, with accedend chars like 'üöäü'? that would be great!
I'll do it.
Do we need now GetServerSetting(), FStandartConfirmingStrings, function EncodeString and so on?? What do you think?
Look at test I gave you earlier (procedure TZTestPostgreSQLTokenizer.TestQuoteState;)

Tokenizer has to know what rules will be applied by the server and parse strings the same.

Look at my modified tokenizer.

Code: Select all

{**
  Checks whether escape syntax is used.

  @return a True if escape syntax is used.
}
function TZPostgreSQLQuoteState.CheckEscapeSyntax(Stream: TStream): Boolean;
var
  ReadChar: Char;
begin
  Result := not FStandardConformingStrings;
  if FStandardConformingStrings then
  begin
    Stream.Seek(-SizeOf(Char), soFromCurrent);
    if Stream.Position >= SizeOf(Char) then
    begin
      Stream.Seek(-SizeOf(Char), soFromCurrent);
      Stream.Read(ReadChar, SizeOf(Char));
      Result := UpperCase(ReadChar) = 'E';
      Stream.Seek(SizeOf(Char), soFromCurrent);
    end;
  end;
end;

{**
  Return a quoted string token from a reader. This method
  will collect characters until it sees a match to the
  character that the tokenizer used to switch to this state.

  @return a quoted string token from a reader
}
function TZPostgreSQLQuoteState.NextToken(Stream: TStream;
  FirstChar: Char; Tokenizer: TZTokenizer): TZToken;
const BackSlash = Char('\');
var
  ReadChar: Char;
  LastChar: Char;
  QuoteCount: Integer;
  EscapeSyntax: Boolean;
begin
  Result.Value := FirstChar;
  QuoteCount := 1;
  EscapeSyntax := False;
  if FirstChar = '"' then
  begin
    Result.TokenType := ttWord;
  end
  else
  begin
    Result.TokenType := ttQuoted;
    EscapeSyntax := CheckEscapeSyntax(Stream);
  end;

  LastChar := #0;

  while Stream.Read(ReadChar, SizeOf(Char)) > 0 do
  begin
    if ReadChar = FirstChar then
      Inc(QuoteCount);

    if (LastChar = FirstChar) and (ReadChar <> FirstChar) then
    begin
      if QuoteCount mod 2 = 0 then
      begin
        Stream.Seek(-SizeOf(Char), soFromCurrent);
        Break;
      end;
    end;
    Result.Value := Result.Value + ReadChar;
    if (LastChar = BackSlash) and EscapeSyntax then
      LastChar := #0
    else if (LastChar = FirstChar) and (ReadChar = FirstChar) then
      LastChar := #0
    else LastChar := ReadChar;
  end;
end;
It does what we just need. All we have to do is set FStandardComformingStrings to a valid value.

Posted: 30.07.2012, 17:31
by EgonHugeist
@olehs,

did commit a fix for your test according Ansi-IDE's and UTF8 Strings.

Also did i manipulate the EscapeString function again. I created an overload for D12_UP and added my ZPlainString function to the Ansi IDE's to avoid bad encoded escape behavior.
Look at my modified tokenizer.
This i understand. Other possibility is you create a ttWord childstate. "E" is a word state. If you check the next char for "\" or"'" then go on reading and change the result state(dependend on StandartConfirmingStrings)...

This detection brings up a new question: Which state will this detected sequence have?

miab3, thank you for your confirmation.

Michael

Posted: 30.07.2012, 18:32
by olehs
EgonHugeist,
Other possibility is you create a ttWord childstate. "E" is a word state. If you check the next char for "\" or"'" then go on reading and change the result state(dependend on StandartConfirmingStrings).
Yes, it's also a good algorythm. But it's only "'" you have to check. "\" can't follow E.

Ok. There's even more. How about this query :twisted:

Code: Select all

select $$Test string$$
or even

Code: Select all

select $Nice_name_for_tag$Test string 2$Nice_name_for_tag$

Posted: 30.07.2012, 20:22
by EgonHugeist
olehs,

this is slightly annoying. Hmpf if i do understand all this correctly do you slightly rebuild the whole tokenizer of PostgreSQL. Which is needed for user generated external sended statments.

I don't know and didn't expect such a syntax...

Our issues are now only the external sended Statements after Fixing the Escape behavior for Parameter values, right?

Next thing i don't like: example of stUnicodeStream.
First we do UTF8ToString(in Blob.GetUnicodeString)- then on EscapeString(UTF8Encode(Value), PlainEscaping(AnsiValue), UTF8ToString(AnsiValue))-> then send Statment: UTF8Encode(on sending the query).

What about speed decrease here? Only to be 100% unicode safe? Isn't that a stupid sh...it?

I planned to avoid such stupid things, an overloaded Statement execution for D12_UP. You can find some abstract definitions in ZDbcStatement.pas. But i had not the time to complete this. IF that would be ready then we encode the strings only once and no issues to see. (The reason for my overload of today)

Which doesn't solves the issues from external sended statments, except the user uses this new overload..

Sure you understand what i mean.

What do you think?

Michael

Posted: 30.07.2012, 21:09
by olehs
EgonHugeist,
I don't know and didn't expect such a syntax...
It's not a problem, I can add this processing. It's gonna be very useful for DBMS developers etc.
Sure you understand what i mean.
I'm not sure I understand, I need to look deeper...

Posted: 30.07.2012, 21:34
by EgonHugeist
olehs,

It's not a problem, I can add this processing. It's gonna be very useful for DBMS developers etc.
That would be great. We really need this token behavior to stabalize everything. And i've the impression you can help us here!
I'm not sure I understand, I need to look deeper...
Actully do not need to invest time there. This i think is my part(as long i've the time to complete everything). Teamwork...

Michael

Posted: 01.08.2012, 13:37
by miab3
@EgonHugeist

Zeos7_branches_ testing_r1637 compiles and runs on D2006, DXE2 32/64, LazarusWin1.1.0/fpc 2.7.1 32/64. (With a small exception on Oracle AQ$_QUEUES in DXE2-32).

Michal

Posted: 04.08.2012, 13:45
by miab3
@EgonHugeist

Zeos7_branches_ testing_r1640 compiles and runs on D2006, DXE2 32/64, LazarusWin1.1.0/fpc 2.7.1 32/64. (With a small exception on Oracle AQ$_QUEUES in DXE2-32).

Michal

Posted: 04.08.2012, 22:01
by EgonHugeist
miab3,

:lol: Thank you Michal, for reminding us according the Nested issues.

@Michal, Olehs,
I'm daddy since 03.08.2012 and soo happy about. So don't wait for reply the next days, ok? But i'm still coding in the background..

Michael

Posted: 05.08.2012, 08:40
by miab3
@Michael

As I suspected something because of your silence. Congratulations.

Michal

Posted: 09.08.2012, 09:40
by miab3
@EgonHugeist

Zeos7_branches_ testing_r1642 compiles and runs on D2006, DXE2 32/64, LazarusWin1.1.0/fpc 2.7.1 32/64. ( Oracle as usual :) )

Michal

Posted: 13.08.2012, 00:10
by EgonHugeist
miab3,

Thank you Michal... :) The oracle isse is still on the todo's.

Michael

Posted: 14.08.2012, 08:57
by miab3
@EgonHugeist

Zeos7_branches_ testing_r1644
http://svn.code.sf.net/p/zeoslib/code-0 ... s/testing/
compiles and runs on (I have tested):

- D2006,
- D2007/C++,
- DXE2 32/64 / C++32,
- Lazarus Win 1.1/fpc 2.7.1 32/64,
- Lazarus Lin 1.1/fpc 2.7.1 64 on Debian 64-bit (LMDE 201204).

I enclose three small patches ( tagged as M.A.) for older versions of C++ Builder.

Michal Abramczyk

Posted: 15.08.2012, 09:37
by EgonHugeist
miab3,

thank you for testing the branches. Applyed your patch. Rev. 1646.

Michal can you use TortoisSVN next times instead? Then you can easiely generate a *.patch file and i do not need to compare the files for diff's..

Thank you,

Michael

Posted: 18.08.2012, 13:08
by markus
Hi,

I've encountered problem with PrepareSQL property of ZConnection.

My query is like this:

Code: Select all

select *,null as typ_upustu_tmp, null as czas_trwania_tmp,null as thumb  from (SELECT * from data.klienci
)q WHERE (imie ~* 'ed') AND (usr_status >= 0)
but when PrepareSQL is set to true PostgreSQL receives query like this:

Code: Select all

2012-08-18 13:52:01 CEST [127.0.0.1(49731)]/prosalon prosalon_apo STATEMENT:  select *,null as typ_upustu_tmp, null as czas_trwania_tmp,null as thumb  from (SELECT * from data.klienci
	)q WHERE (imie ~ 'ed') AND (usr_status >= 0)
difference is imie ~* 'ed' versus imie ~ 'ed'.

whit PrepareSQL=false query send to postgres is correct and i got expected results.

Best regards,
Marek