Full Unicode/Ansi-Support in /testing branch

Post by **EgonHugeist** » 11.05.2012, 15:54

miab3,

Trank you, i'm bussy @the moment. So i'll check your patch this evening.

Michael

mrLion · Post by **mrLion** » 11.05.2012, 17:39

success or again problems?

problems again. In FireBird simple query rise idiotic mistake - an indefinite character after the ":".
the query:

update tbl set fld1 =: par1, fld2 =: par2, fld3 =: par3 where fld4 =: par4

error in the function

Code: Select all

... SQL-> Add ("....");

the source file encoding win1251
In addition, another error when trying to write a text BLOB fields with details later.

firebird 2.5 protocol

the same query using IBX works with a bang

Post by **EgonHugeist** » 11.05.2012, 18:15

Give me a better example please. The Query you see here is a firebird interpreted one for the prepared statements. FireBird is using a own paramSqlDA object to assign the values. It is used for Big blobs for example because firebird has a 32KB limit for Query execution.

... Michael

mrLion · Post by **mrLion** » 12.05.2012, 16:14

Michael
i`m using firebird 2.5 with DB in utf-8 codepage.
IDE - RAD Studio C++ Builder XE2 upd 4
plg_db - is plugin, q1 - dynamic created TZQuery inside plugin, connected to TZConnection
source file encoded - win1251
all field in table VARCHAR 255, except tr_note - BLOB SUBTYPE TEXT
TZConnection properties:

PreprepareSQL=ON
ClientCodePage=UTF8
codepage=UTF8

see err1.jpg in attach

Code: Select all

 if (win->ShowModal()==mrOk)
 {
  try
  {
   plg_db->StartTransaction();
   plg_db->q1->SQL->Clear();
   //-- next line rise error on runtime if we use ZEOS!
   //-- else, if we use IBX - no any errors occur
   plg_db->q1->SQL->Add("update trwrk set tr_name=:nam, tr_note=:not, tr_num=:num where tr_code=:cod");
   //------------------------------------------------------------------------------
   plg_db->q1->ParamByName("nam")->AsString = win->edVal->Text.Trim();
   plg_db->q1->ParamByName("num")->AsString = win->edNum->Text.Trim();
   plg_db->q1->ParamByName("cod")->AsString = cod;
   plg_db->q1->ParamByName("not")->AsString = win->edMem->Lines->GetText();
   plg_db->q1->ExecSQL();
   plg_db->q1->Close();
   plg_db->Commit();
  }
  catch (EXCEPTIONCLASS &E)
  {
   if (!plg_comdlg->CheckKnownErrors(E))
   {
    plg_comdlg->ShowError(this->ClassName(), "cmEditTrExecute",
    plg_comdlg->TranslatePostgreERR(E), E.Message, false);
   }
   plg_db->Rollback();
  }
 }

in runtime on selected above string rised error:
see err2.jpg in attach

in English ~ incorrect sign after ":"
and now - attention: simple question: WTF???!!!

repeat, if i use IBX no errors.

unless this, in code:

Code: Select all

 if (win->ShowModal()==mrOk)
 {
  TPLGDatabase *db=(TPLGDatabase*)plgMan->GetPLGInterface(INT_DBMANAGER);
  try
  {
   db->StartTransaction();
   db->q1->SQL->Clear();
   db->q1->SQL->Add("insert into parlst (pl_name, pl_note, pl_code, pc_id)");
   db->q1->SQL->Add("values (:nam, :note, :code, :PID)");
   db->q1->ParamByName("nam")->AsString  = win->edValue->Text.Trim();
   db->q1->ParamByName("PID")->AsInteger = win->cbLook->EditValue;
   db->q1->ParamByName("note")->AsString = win->mem->Lines->GetText();
   db->q1->ParamByName("code")->AsString = Code;
   db->q1->ExecSQL();
   db->Commit();
  }
  catch (EXCEPTIONCLASS &E)
  {
   if (!plg_comdlg->CheckKnownErrors(E))
   {
    plg_comdlg->ShowError(this->ClassName(), "cmAddPerExecute",
    plg_comdlg->TranslatePostgreERR(E), E.Message, false);
   }
   db->Rollback();
  }
  RELEASE_PLUGIN(db);
 }

rised error:
see err3_x.jpg in attach

ism · Post by **ism** » 12.05.2012, 16:21

Maybe problem in Charset ? UTF8

http://www.firebirdfaq.org/faq342/
http://www.ibase.ru/unicode_faq.html

Sorry for non english

Я создал базу и таблицы в UTF8, подсоединился в UTF8, и получаю ошибку Malformed string

Это значит, что данные, которые передаются на сервер, идут не в юникоде, а в какой-то другой кодировке. То есть виноват драйвер, компоненты доступа, или само приложение.

mrLion · Post by **mrLion** » 12.05.2012, 16:25

ism, да дело в зеосе. я ж говорю, что IBX пашет на ура. Вот только мне нужен именно зеос, поскольку программа должна динамически переключаться между 2 базами постгре и файбирдом.

Post by **EgonHugeist** » 12.05.2012, 18:33

@miab3,
wow you attached a lot of files. Can you eventually use TortoiseSVN and generate a patch file? So i can see quickly your modifications. On the other hand it's a terrible way to check the differences manually..

@Ivan/mrLion
ParamByName with such a handling i didn't checked. So i'll check this. What IBX can do should Zeos do better! A question of honor again. I'll check the code sequences which are touching the Params. This is almost a complicated thing, because there a 1000 ways to write in a blob-field. Actually i implemented 4 stupid but working deteting sequences:

Code: Select all

procedure TZAbstractCachedResultSet.UpdateUnicodeStream(
  ColumnIndex: Integer; Value: TStream);
var
  Ansi: AnsiString;
  Len: Integer;
  TempStream: TMemoryStream;
  WS: WideString;
begin
{$IFNDEF DISABLE_CHECKING}
  CheckUpdatable;
{$ENDIF}
  PrepareRowForUpdates;
  {EgonHugeist: TempBuffer the WideString, }
  if Assigned(Value) then
  begin //Step one: Findout, wat's comming in! To avoid User-Bugs
    //it is possible that a PAnsiChar OR a PWideChar was written into
    //the Stream!!!
    if Length(PWideChar(TMemoryStream(Value).Memory)) = Value.Size then
    begin
      WS := PWideChar(TMemoryStream(Value).Memory);
      SetLength(WS, Value.Size div 2);
      Ansi := UTF8Encode(WS);
    end
    else
      if Length(String(PAnsiChar(TMemoryStream(Value).Memory))) = 1 then
      begin
        SetLength(WS, Value.Size div 2);
        System.Move(PWideString(TMemoryStream(Value).Memory)^,
          PWideChar(WS)^, Value.Size);
        Ansi := UTF8Encode(WS);
      end
      else
        if Length(String(PAnsiChar(TMemoryStream(Value).Memory))) = Value.Size then
        begin
          if DetectUTF8Encoding(PAnsiChar(TMemoryStream(Value).Memory)) = etAnsi then
            Ansi := AnsiToUTF8(PAnsiChar(TMemoryStream(Value).Memory))
          else
            Ansi := PAnsiChar(TMemoryStream(Value).Memory);
        end
        else
        begin
          SetLength(Ansi, Value.Size);
          TMemoryStream(Value).Read(PAnsiChar(Ansi)^, Value.Size);
          if DetectUTF8Encoding(Ansi) = etAnsi then
            Ansi := AnsiToUTF8(Ansi);
        end;
    Len := Length(Ansi);
    TempStream := TMemoryStream.Create;
    TempStream.Write(PAnsiChar(Ansi)^, Len);
    TempStream.Position := 0;
    FRowAccessor.SetUnicodeStream(ColumnIndex, TempStream);
    TempStream.Free;
  end
  else
    FRowAccessor.SetUnicodeStream(ColumnIndex, Value);
end;

If ParamByName().AsString didn't touch this code, then your shown exception is right in this case. Btw. the detecting seqences ar available since 1269. Do you use this or a newer one?

@ism, you're right this should be the problem. But i want that this is running automaticaly if PreprepareSQL = True. The optimal solution would be: Kill this property. But we've to man D12_UP users who have a lot of code ready. So if i want it or not this must stay optional. My workout is slightly 5 years to late.

@Serg77
Lucky now?

Michael

miab3 · Post by **miab3** » 12.05.2012, 19:14

@EgonHugeist;

You can use ExamDiff or something similar.
One adjustment is extremely important for all.
In ZPlainLoader.pas in the line 206 should be:

RowHeaderSize = sizeof (TZRowBuffer) - sizeof (TZByteArray);

Others interested in using the DXE2 Win64.

Michal Abramczyk

ism · Post by **ism** » 12.05.2012, 19:28

mrLion
Problem in client problem charset, if client query send cp1251 and server utf8 (or vice versa) error is possible with Firebird

mrLion · Post by **mrLion** » 12.05.2012, 19:36

EgonHugeist, i`m Ivan.

If the encoding me everything understand is how to deal with the first error, when adding a simple query error crashes with an invalid character after the ":"
if I do not deny my inner translator from English, I realized that the correction of these errors cause your life? Sorry for the rough translation of google
I take every 100th revision and re-check on the performance. constant is not possible. so excuse me if I did not respond for a long time
oh... last test rev - 1267.
ism пиши мне по русски.

ism · Post by **ism** » 12.05.2012, 19:39

mrLion
Другие не поймут

Post by **EgonHugeist** » 12.05.2012, 20:40

miab3,

the most parts i'll accept with some changes(compatibility to FPC). Rev. 1272. But why do you want to change the PlainLoader.Handle? Normaly you're right there. Do you know more than me here? Makes it trouble with XE264?

The INVALID_HANDLE_VALUE = DWord(-1); and a Windows constant. THandle is a Cardinal(unsigned integer). We use this Invalid_handle_value on the LibraryLoading which is an additional security param to know if LoadLibrary fails.

And what about the D16-packagefolder? Is the last upgrade we did wrong?

Michael

ism · Post by **ism** » 12.05.2012, 21:38

And what about the D16-packagefolder? Is the last upgrade we did wrong?

No , need change name folder on Delphi_XE2 and other
More understandable

Post by **EgonHugeist** » 12.05.2012, 21:49

ism,

yes i noticed your hint, and i would like to do this just right now. But here i'm waiting for replay of the Project-Manager..

Michael

miab3 · Post by **miab3** » 12.05.2012, 23:16

@EgonHugeist
>the most parts i'll accept with some changes(compatibility to FPC). Rev. > 1272.

But my version was right and not what you did.
DXE2/32 - WIN32
DXE2/64 - WIN64

>But why do you want to change the PlainLoader.Handle? Normaly >you're right there. Do you know more than me here? Makes it trouble >with XE264?

Yes.

> The INVALID_HANDLE_VALUE = DWord(-1); and a Windows constant.

In DXE2 INVALID_HANDLE_VALUE = THandle(-1);
(Winapi.Windows.pas)

>And what about the D16-packagefolder? Is the last upgrade we did >wrong?

Yes.
Especially ZComponent.dpk, ZComponentDesign.dpk and all *.dproj with XE(1) and OSX mishmash.

Here an interesting article:
http://blogs.devart.com/dac/index.php/d ... o-xe2.html

Do you have DXE2 or trial version?

Michal Abramczyk