Full Unicode/Ansi-Support in /testing branch

[EgonHugeist]

miab3,

And only for D2006? Makes me wondering. I removed the TZPostgreSQLTokenizer interface. It should work now i hope. R.1624

Michael

[miab3]

@EgonHugeist

You have to move for protected or public not private then Delphi2006 compile(I think is right):

procedure SetEscapeMarkSequence(const Value: String); <-------------------------
protected

protected
procedure SetEscapeMarkSequence(const Value: String);

I suspect that the D2006 and earlier was that this problem or feature.

Michal

[olehs]

miab3,

I guess you're right. Methods declared in public interface shouldn't be implemented as private.

Btw, why is it in interface at all?

[EgonHugeist]

olehs, miab3,

If nobody did recognize it: These both functions had been in the private section since egonhugeist did exists. So i dont understand it. Anyway i did commit that proposal now. On my side i could run everything with FPC, D7, D2007... Rev 1627.

@olehs,

saw your commit. A good first job. Will test it right now.

Michael

[olehs]

miab3,

Maybe you use some special compile settings (optimization=false etc.)

[olehs]

EgonHugeist,

I'm proceeding with modifying QuoteState tokenizer and I'm gonna need TZPostgreSQLTokenizer back (with a little change).

This is a test I made that uses it.

Code: Select all

procedure TZTestPostgreSQLTokenizer.TestQuoteState;
const
  TokenString1: string = '"a\""\''aa" ''c\'' ''c''''c''';

  TokenTypes1Off: array[0..1] of TZTokenType = (ttWord, ttQuoted);
  TokenValues1Off: array[0..1] of string = ('"a\""\''aa"', '''c\'' ''c''''c''');

  TokenTypes1On: array[0..2] of TZTokenType = (ttWord, ttQuoted, ttQuoted);
  TokenValues1On: array[0..2] of string = ('"a\""\''aa"', '''c\''', '''c''''c''');
begin
  (Tokenizer as TZPostgreSQLTokenizer).SetStandardConformingStrings(False);
  CheckTokens(Tokenizer.TokenizeBuffer(TokenString1,
    [toSkipEOF, toSkipWhitespaces]), TokenTypes1Off, TokenValues1Off);

  (Tokenizer as TZPostgreSQLTokenizer).SetStandardConformingStrings(True);
  CheckTokens(Tokenizer.TokenizeBuffer(TokenString1,
    [toSkipEOF, toSkipWhitespaces]), TokenTypes1On, TokenValues1On);
end;

This test is more important, because depending on StandardConformingStrings queries must be tokenized in different ways.

So, what do you think, can I return that TZPostgreSQLTokenizer back?

[EgonHugeist]

olehs,

The TZPostgreSQLTokenizer does still exist, or not? (:

I don't understand the question right now. What you did was overriding some protected procedures on CppCommentState. No problem. Do you mean you need me interface back i did before?

Generally i so no issue to have it. I'm only wondering why now Michal has compiling issues. But i trust him. He is allways there and does a great job to confirm the running latest commits. But i must admit my packages are not the standarts(i use all available files for all packages), so i testet them with the "normal" packeges again and everything works. Curious...

Accordingly the spezial Tokenizer: That's what i mentioned before. IF he must have different behavior, then there must be a way to inform him about to get him running like expected. In my mind a spezial descendant interface of IZTokenizer is the only way here...

Michael

[olehs]

EgonHugeist,

Yes, you're right. I was talking about IZPostgreSQLTokenizer interface.
I need it to set StandardConformingStrings behaviour.

[EgonHugeist]

olehs,

then add it. Let's see how we get this running later on. Wondering that you do not need the backslash_quote option here. If i understand the documentations right is a \ equal to ' IF backslash_quote=ON. Or am i totaly wrong? So if backslash_quote is ON then the tokenizer should detect \sasf\ as ttQuoted=sasf or not?

Michael

[olehs]

EgonHugeist,

No, actually.
When standard_conforming_strings=off or E'syntax is used, then \' is parsed as '
But when standard_conforming_strings=on, \' is parsed as \'
backslash_quote controls, whether postgres should allow usage of \'
in SCS=off & E'syntax modes or it should raise exception.

Here are examples

Code: Select all

zeoslib=# set standard_conforming_strings=ON;
SET
zeoslib=# select '''' as a, '\'' as b';
 a | ?column?
---+----------
 ' | \' as b
(1 row)

BUT

Code: Select all

zeoslib=# set standard_conforming_strings=off;
SET
zeoslib=# select '''' as a, '\'' as b;
 a | b 
---+---
 ' | '
(1 row)
zeoslib=# set standard_conforming_strings=on;
SET
zeoslib=# select '''' as a, E'\'' as b;
 a | b
---+---
 ' | '
(1 row)

and the last one

Code: Select all

zeoslib=# set standard_conforming_strings=off;
SET
zeoslib=# set backslash_quote=off;
SET
zeoslib=# select '''' as a, '\'' as b;
ERROR:  unsafe use of \' in a string literal
LINE 1:select '''' as a, '\'' as b;
                          ^
HINT:  Use '' to write quotes in strings. \' is insecure in client-only enc
odings.

[EgonHugeist]

olehs,

Ok i understand....

Code: Select all

function TZPostgreSQLBaseDriver.EscapeString(Handle: PZPostgreSQLConnect;
  Encoding: TZCharEncoding; const Source: String; Dest: String): Integer;
var
  ResLen: NativeUInt;
  Temp: PAnsiChar;
  {$IFDEF DELPHI12_UP}
  SourceTemp, DestTemp: RawByteString;
  {$ENDIF}
begin
  if Assigned(POSTGRESQL_API.PQescapeStringConn) then
  begin
    {$IFDEF DELPHI12_UP}
    if encoding = ceUTF8 then
      SourceTemp := UTF8Encode(Source)
    else
      SourceTemp := AnsiString(Source);
    GetMem(Temp, Length(SourceTemp)*2);
    ResLen := POSTGRESQL_API.PQescapeStringConn(Handle, Temp,
      PAnsiChar(SourceTemp), StrLen(PAnsiChar(SourceTemp)), @Result);
    SetLength(DestTemp, ResLen);
    Move(Temp^, PAnsiChar(DestTemp)^, ResLen);
    if encoding = ceUTF8 then
      Dest := UTF8ToString(DestTemp)
    else
      Dest := String(DestTemp);
    {$ELSE}
    GetMem(Temp, Length(Source)*2);
    ResLen := POSTGRESQL_API.PQescapeStringConn(Handle, Temp,
      PAnsiChar(Source), StrLen(PAnsiChar(Source)), @Result);
    SetLength(Dest, ResLen);
    Move(Temp^, PAnsiChar(Dest)^, ResLen);
    {$ENDIF}
  end
  else
  begin
    Dest := Source;
    Result := 0;
  end;
end;

Hmm what do you think about this? Let's make postgre the stupid escaping work. What i don't know: Is the backward encoding for D12_UP valid, after escaping???? IF not then we should handle these issues one level deeper on the Plain-Layer.

This i allready implemented for MySQL...

Michael

[olehs]

I made some tests of it on XE2 & UTF8. Seems to work fine. You only have to make Dest an out or var param.

[EgonHugeist]

@olehs,

solved your tests.. Oh didn't saw you previous post...

Rev. 1629

But like i wrote i'm not 100% sure what happens to more complicated Unicode Strings, than these of our tests..

And i don't know if the Tokenizer behaves correctly for the comming test cases..

@miab3, Michal is everything ok now?

Michael

[EgonHugeist]

olehs,

next question: Do we need now GetServerSetting(), FStandartConfirmingStrings, function EncodeString and so on?? What do you think?

and could you please insert some much more complicated tests, with accedend chars like 'üöäü'? that would be great!

AND Couldn't we say: IF PreprepareSQL -> use AnsiQuoteStr instead of EscapeString and then we do this stuff automatically? Works for MySQL perfect.

Michael

[miab3]

@EgonHugeist

Yes,
Zeos7testing_1629 compiles and runs on D2006, DXE2 32/64, LazarusWin1.1.0/fpc 2.7.1 32/64. (With a small exception on Oracle AQ$_QUEUES in DXE2-32).

Michal