Full Unicode/Ansi-Support in /testing branch

The alpha/beta tester's forum for ZeosLib 7.0.x series

Report problems concerning our Delphi 2009+ version and new Zeoslib 7.0 features here.

This is a forum that will be removed once the 7.X version goes into stable!!

Moderators: gto, EgonHugeist, olehs

Locked
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

miab3,

Trank you, i'm bussy @the moment. So i'll check your patch this evening.

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
mrLion
Senior Boarder
Senior Boarder
Posts: 71
Joined: 20.03.2010, 10:17

Post by mrLion »

success or again problems?
problems again. In FireBird simple query rise idiotic mistake - an indefinite character after the ":".
the query:

Code: Select all

update tbl set fld1 =: par1, fld2 =: par2, fld3 =: par3 where fld4 =: par4
error in the function

Code: Select all

... SQL-> Add ("....");

the source file encoding win1251
In addition, another error when trying to write a text BLOB fields with details later.

firebird 2.5 protocol

the same query using IBX works with a bang
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

Give me a better example please. The Query you see here is a firebird interpreted one for the prepared statements. FireBird is using a own paramSqlDA object to assign the values. It is used for Big blobs for example because firebird has a 32KB limit for Query execution.

... Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
mrLion
Senior Boarder
Senior Boarder
Posts: 71
Joined: 20.03.2010, 10:17

Post by mrLion »

Michael
i`m using firebird 2.5 with DB in utf-8 codepage.
IDE - RAD Studio C++ Builder XE2 upd 4
plg_db - is plugin, q1 - dynamic created TZQuery inside plugin, connected to TZConnection
source file encoded - win1251
all field in table VARCHAR 255, except tr_note - BLOB SUBTYPE TEXT
TZConnection properties:

PreprepareSQL=ON
ClientCodePage=UTF8
codepage=UTF8

see err1.jpg in attach

Code: Select all

 if (win->ShowModal()==mrOk)
 {
  try
  {
   plg_db->StartTransaction();
   plg_db->q1->SQL->Clear();
   //-- next line rise error on runtime if we use ZEOS!
   //-- else, if we use IBX - no any errors occur
   plg_db->q1->SQL->Add("update trwrk set tr_name=:nam, tr_note=:not, tr_num=:num where tr_code=:cod");
   //------------------------------------------------------------------------------
   plg_db->q1->ParamByName("nam")->AsString = win->edVal->Text.Trim();
   plg_db->q1->ParamByName("num")->AsString = win->edNum->Text.Trim();
   plg_db->q1->ParamByName("cod")->AsString = cod;
   plg_db->q1->ParamByName("not")->AsString = win->edMem->Lines->GetText();
   plg_db->q1->ExecSQL();
   plg_db->q1->Close();
   plg_db->Commit();
  }
  catch (EXCEPTIONCLASS &E)
  {
   if (!plg_comdlg->CheckKnownErrors(E))
   {
    plg_comdlg->ShowError(this->ClassName(), "cmEditTrExecute",
    plg_comdlg->TranslatePostgreERR(E), E.Message, false);
   }
   plg_db->Rollback();
  }
 }
in runtime on selected above string rised error:
see err2.jpg in attach

in English ~ incorrect sign after ":"
and now - attention: simple question: WTF???!!! :)
repeat, if i use IBX no errors.

unless this, in code:

Code: Select all

 if (win->ShowModal()==mrOk)
 {
  TPLGDatabase *db=(TPLGDatabase*)plgMan->GetPLGInterface(INT_DBMANAGER);
  try
  {
   db->StartTransaction();
   db->q1->SQL->Clear();
   db->q1->SQL->Add("insert into parlst (pl_name, pl_note, pl_code, pc_id)");
   db->q1->SQL->Add("values (:nam, :note, :code, :PID)");
   db->q1->ParamByName("nam")->AsString  = win->edValue->Text.Trim();
   db->q1->ParamByName("PID")->AsInteger = win->cbLook->EditValue;
   db->q1->ParamByName("note")->AsString = win->mem->Lines->GetText();
   db->q1->ParamByName("code")->AsString = Code;
   db->q1->ExecSQL();
   db->Commit();
  }
  catch (EXCEPTIONCLASS &E)
  {
   if (!plg_comdlg->CheckKnownErrors(E))
   {
    plg_comdlg->ShowError(this->ClassName(), "cmAddPerExecute",
    plg_comdlg->TranslatePostgreERR(E), E.Message, false);
   }
   db->Rollback();
  }
  RELEASE_PLUGIN(db);
 }
rised error:
see err3_x.jpg in attach
You do not have the required permissions to view the files attached to this post.
ism
Zeos Test Team
Zeos Test Team
Posts: 202
Joined: 02.10.2010, 20:48

Post by ism »

Maybe problem in Charset ? UTF8

http://www.firebirdfaq.org/faq342/
http://www.ibase.ru/unicode_faq.html

Sorry for non english
Я создал базу и таблицы в UTF8, подсоединился в UTF8, и получаю ошибку Malformed string

Это значит, что данные, которые передаются на сервер, идут не в юникоде, а в какой-то другой кодировке. То есть виноват драйвер, компоненты доступа, или само приложение.
Lazarus 1.0.8 fpc 2.6.0
mrLion
Senior Boarder
Senior Boarder
Posts: 71
Joined: 20.03.2010, 10:17

Post by mrLion »

ism, да дело в зеосе. я ж говорю, что IBX пашет на ура. Вот только мне нужен именно зеос, поскольку программа должна динамически переключаться между 2 базами постгре и файбирдом.
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

@miab3,
wow you attached a lot of files. Can you eventually use TortoiseSVN and generate a patch file? So i can see quickly your modifications. On the other hand it's a terrible way to check the differences manually..

@Ivan/mrLion
ParamByName with such a handling i didn't checked. So i'll check this. What IBX can do should Zeos do better! A question of honor again. I'll check the code sequences which are touching the Params. This is almost a complicated thing, because there a 1000 ways to write in a blob-field. Actually i implemented 4 stupid but working deteting sequences:

Code: Select all

procedure TZAbstractCachedResultSet.UpdateUnicodeStream(
  ColumnIndex: Integer; Value: TStream);
var
  Ansi: AnsiString;
  Len: Integer;
  TempStream: TMemoryStream;
  WS: WideString;
begin
{$IFNDEF DISABLE_CHECKING}
  CheckUpdatable;
{$ENDIF}
  PrepareRowForUpdates;
  {EgonHugeist: TempBuffer the WideString, }
  if Assigned(Value) then
  begin //Step one: Findout, wat's comming in! To avoid User-Bugs
    //it is possible that a PAnsiChar OR a PWideChar was written into
    //the Stream!!!
    if Length(PWideChar(TMemoryStream(Value).Memory)) = Value.Size then
    begin
      WS := PWideChar(TMemoryStream(Value).Memory);
      SetLength(WS, Value.Size div 2);
      Ansi := UTF8Encode(WS);
    end
    else
      if Length(String(PAnsiChar(TMemoryStream(Value).Memory))) = 1 then
      begin
        SetLength(WS, Value.Size div 2);
        System.Move(PWideString(TMemoryStream(Value).Memory)^,
          PWideChar(WS)^, Value.Size);
        Ansi := UTF8Encode(WS);
      end
      else
        if Length(String(PAnsiChar(TMemoryStream(Value).Memory))) = Value.Size then
        begin
          if DetectUTF8Encoding(PAnsiChar(TMemoryStream(Value).Memory)) = etAnsi then
            Ansi := AnsiToUTF8(PAnsiChar(TMemoryStream(Value).Memory))
          else
            Ansi := PAnsiChar(TMemoryStream(Value).Memory);
        end
        else
        begin
          SetLength(Ansi, Value.Size);
          TMemoryStream(Value).Read(PAnsiChar(Ansi)^, Value.Size);
          if DetectUTF8Encoding(Ansi) = etAnsi then
            Ansi := AnsiToUTF8(Ansi);
        end;
    Len := Length(Ansi);
    TempStream := TMemoryStream.Create;
    TempStream.Write(PAnsiChar(Ansi)^, Len);
    TempStream.Position := 0;
    FRowAccessor.SetUnicodeStream(ColumnIndex, TempStream);
    TempStream.Free;
  end
  else
    FRowAccessor.SetUnicodeStream(ColumnIndex, Value);
end;
If ParamByName().AsString didn't touch this code, then your shown exception is right in this case. Btw. the detecting seqences ar available since 1269. Do you use this or a newer one?

@ism, you're right this should be the problem. But i want that this is running automaticaly if PreprepareSQL = True. The optimal solution would be: Kill this property. But we've to man D12_UP users who have a lot of code ready. So if i want it or not this must stay optional. My workout is slightly 5 years to late.

@Serg77
Lucky now?

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
miab3
Zeos Test Team
Zeos Test Team
Posts: 1310
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

@EgonHugeist;

You can use ExamDiff or something similar.
One adjustment is extremely important for all.
In ZPlainLoader.pas in the line 206 should be:

RowHeaderSize = sizeof (TZRowBuffer) - sizeof (TZByteArray);

Others interested in using the DXE2 Win64.

Michal Abramczyk
ism
Zeos Test Team
Zeos Test Team
Posts: 202
Joined: 02.10.2010, 20:48

Post by ism »

mrLion
Problem in client problem charset, if client query send cp1251 and server utf8 (or vice versa) error is possible with Firebird
Lazarus 1.0.8 fpc 2.6.0
mrLion
Senior Boarder
Senior Boarder
Posts: 71
Joined: 20.03.2010, 10:17

Post by mrLion »

EgonHugeist, i`m Ivan. :)
If the encoding me everything understand is how to deal with the first error, when adding a simple query error crashes with an invalid character after the ":"
if I do not deny my inner translator from English, I realized that the correction of these errors cause your life? Sorry for the rough translation of google
I take every 100th revision and re-check on the performance. constant is not possible. so excuse me if I did not respond for a long time
oh... last test rev - 1267.
ism пиши мне по русски.
ism
Zeos Test Team
Zeos Test Team
Posts: 202
Joined: 02.10.2010, 20:48

Post by ism »

mrLion
Другие не поймут
Lazarus 1.0.8 fpc 2.6.0
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

miab3,

the most parts i'll accept with some changes(compatibility to FPC). Rev. 1272. But why do you want to change the PlainLoader.Handle? Normaly you're right there. Do you know more than me here? Makes it trouble with XE264?

The INVALID_HANDLE_VALUE = DWord(-1); and a Windows constant. THandle is a Cardinal(unsigned integer). We use this Invalid_handle_value on the LibraryLoading which is an additional security param to know if LoadLibrary fails.

And what about the D16-packagefolder? Is the last upgrade we did wrong?

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
ism
Zeos Test Team
Zeos Test Team
Posts: 202
Joined: 02.10.2010, 20:48

Post by ism »

And what about the D16-packagefolder? Is the last upgrade we did wrong?
No , need change name folder on Delphi_XE2 and other
More understandable
Lazarus 1.0.8 fpc 2.6.0
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

ism,

yes i noticed your hint, and i would like to do this just right now. But here i'm waiting for replay of the Project-Manager..

Michael
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
miab3
Zeos Test Team
Zeos Test Team
Posts: 1310
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

@EgonHugeist
>the most parts i'll accept with some changes(compatibility to FPC). Rev. > 1272.

But my version was right and not what you did.
DXE2/32 - WIN32
DXE2/64 - WIN64


>But why do you want to change the PlainLoader.Handle? Normaly >you're right there. Do you know more than me here? Makes it trouble >with XE264?

Yes.

> The INVALID_HANDLE_VALUE = DWord(-1); and a Windows constant.

In DXE2 INVALID_HANDLE_VALUE = THandle(-1);
(Winapi.Windows.pas)

>And what about the D16-packagefolder? Is the last upgrade we did >wrong?

Yes.
Especially ZComponent.dpk, ZComponentDesign.dpk and all *.dproj with XE(1) and OSX mishmash.

Here an interesting article:
http://blogs.devart.com/dac/index.php/d ... o-xe2.html

Do you have DXE2 or trial version?

Michal Abramczyk
Locked