Full Unicode/Ansi-Support in /testing branch
Moderators: gto, EgonHugeist, olehs
@EgonHugeist
Editing dbgrid.
Seems to me that that record to the database by DXE2 is incorrect.
Because the D2006 and DXE2 read it the same.
Michal
Editing dbgrid.
Seems to me that that record to the database by DXE2 is incorrect.
Because the D2006 and DXE2 read it the same.
Michal
You do not have the required permissions to view the files attached to this post.
Last edited by miab3 on 11.06.2012, 14:14, edited 1 time in total.
- EgonHugeist
- Zeos Project Manager
- Posts: 1936
- Joined: 31.03.2011, 22:38
Oh that i've to check. But i'm not on my computer yet.
My thoughts:
Your db is created in UTF8. What i'm doing in ZDbcMySql.pas:
On connection.open i compare the the two charactersets (the first one i ask from the PlainDriver, the second is the one you've set). Only if there are differences i do execute the 'setnames=...' statements. That could be wrong. Can you comment out that compare line and force zeos to execute that statement every time? Does that fix this issue?
On the other hand i can verify what's going wrong on evening not right now.
What is displayed if you execute "show variables" after connect?
Michael
My thoughts:
Your db is created in UTF8. What i'm doing in ZDbcMySql.pas:
On connection.open i compare the the two charactersets (the first one i ask from the PlainDriver, the second is the one you've set). Only if there are differences i do execute the 'setnames=...' statements. That could be wrong. Can you comment out that compare line and force zeos to execute that statement every time? Does that fix this issue?
On the other hand i can verify what's going wrong on evening not right now.
What is displayed if you execute "show variables" after connect?
Michael
Best regards, Michael
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
- EgonHugeist
- Zeos Project Manager
- Posts: 1936
- Joined: 31.03.2011, 22:38
@miab3,
Hmm that's not true! But i must admit you can't see this, Michal. On my branch the most string convertations where done with two little funtions: ZPlainString and ZDbcString. The purpose of this funtions is to minimize the compiler-directives and to have ANSI and Unicode support for D12Up. Be more fllexible instead of only unicode or endless directives. These two functions where exported from the TZCodePagedObject. The other objects are descendant from that objects. Just open the ZCompatibility.pas and look for these funtions. They are doing everything where we need compiler directives but more flexible. Thought FPC2.8 goes that UnicodeString way too eventually(not decided yet). What now again some more directives concerning the string handlings?
But there is some more work left to do. Like on PostgreSQL, where the Server exceptions are displayed wrong.. On SQLite i don't need these because that engine is a pur unicode db and except from the other providers.
Concerning your issue was you able to find the line where i pointed you (Connect or Open)?
Michael
Hmm that's not true! But i must admit you can't see this, Michal. On my branch the most string convertations where done with two little funtions: ZPlainString and ZDbcString. The purpose of this funtions is to minimize the compiler-directives and to have ANSI and Unicode support for D12Up. Be more fllexible instead of only unicode or endless directives. These two functions where exported from the TZCodePagedObject. The other objects are descendant from that objects. Just open the ZCompatibility.pas and look for these funtions. They are doing everything where we need compiler directives but more flexible. Thought FPC2.8 goes that UnicodeString way too eventually(not decided yet). What now again some more directives concerning the string handlings?
But there is some more work left to do. Like on PostgreSQL, where the Server exceptions are displayed wrong.. On SQLite i don't need these because that engine is a pur unicode db and except from the other providers.
Concerning your issue was you able to find the line where i pointed you (Connect or Open)?
Michael
Best regards, Michael
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
- EgonHugeist
- Zeos Project Manager
- Posts: 1936
- Joined: 31.03.2011, 22:38
miab3,
that was my fault. As Markus did report me his 'Sometime wrong encoded.' issue i had introduced a 100% save way to have allways a real UTF8String. The issue was simple: The strings where double encoded .
It was also clear why it does not happen with FPC or AnsiDelphi. Here i'm able to detect a utf8 encoded string. But i do not know a 100% save way to detect a utf8 sequence in an UnicodeString.
Patch done. Rev.1387
Michael
that was my fault. As Markus did report me his 'Sometime wrong encoded.' issue i had introduced a 100% save way to have allways a real UTF8String. The issue was simple: The strings where double encoded .
It was also clear why it does not happen with FPC or AnsiDelphi. Here i'm able to detect a utf8 encoded string. But i do not know a 100% save way to detect a utf8 sequence in an UnicodeString.
Patch done. Rev.1387
Michael
Best regards, Michael
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
- EgonHugeist
- Zeos Project Manager
- Posts: 1936
- Joined: 31.03.2011, 22:38
miab3,
thank you for your confirmation. The first thing i'll do tonight is to add some unicode tests to our testsuites. Something like this should not happen again!
Michael
thank you for your confirmation. The first thing i'll do tonight is to add some unicode tests to our testsuites. Something like this should not happen again!
Michael
Best regards, Michael
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
@EgonHugeist
Michael, I have two questions:
I'm curious if anyone has already tried to ZEOS FreeTDS in Linux(to MS SQL)?
Since ADO works as it is difficult to adapt it to fpc and Linux.
Though some passages such as ODBC.
Or just standalone ODBC?
These are the questions, of course, not only about you, but to all
Michal Abramczyk
Michael, I have two questions:
I'm curious if anyone has already tried to ZEOS FreeTDS in Linux(to MS SQL)?
Since ADO works as it is difficult to adapt it to fpc and Linux.
Though some passages such as ODBC.
Or just standalone ODBC?
These are the questions, of course, not only about you, but to all
Michal Abramczyk
- EgonHugeist
- Zeos Project Manager
- Posts: 1936
- Joined: 31.03.2011, 22:38
miab3,
If you have public questions, then i might be better to open a new thread?
Personal thoughts:
Actually i do not believe that we have users with FreeTDS+Linux expiriences. We do not have a compiled msdbib.so. And here we need a version with libconv-2 because in linux is everything UTF8 and Lazarus does expect UTF8 too. But the FreeTDS servers (except sybase if that charset is installed) do only send nat_lang Strings for the char/text fields except the nChar/Text fields. Next thing is that our FreeTDS support is brand new. But i know a thread where you can get some informations: Just look to our mssql section toppic thread. There you'll also find a link to the Lazarus bugtracker. Attached is a compiled LCL compatible dblib.dll. But actually i'm waiting for rautgigi's patches... I see some leaks for Lazarus: Actually is not characterset setting included which is needed for the libiconv.dll. Michael you can help us here too.. I've got a lot of more things on my todo list.
According the odbc part i've to pass. No real *nix expieriences.
Michael
If you have public questions, then i might be better to open a new thread?
Personal thoughts:
Actually i do not believe that we have users with FreeTDS+Linux expiriences. We do not have a compiled msdbib.so. And here we need a version with libconv-2 because in linux is everything UTF8 and Lazarus does expect UTF8 too. But the FreeTDS servers (except sybase if that charset is installed) do only send nat_lang Strings for the char/text fields except the nChar/Text fields. Next thing is that our FreeTDS support is brand new. But i know a thread where you can get some informations: Just look to our mssql section toppic thread. There you'll also find a link to the Lazarus bugtracker. Attached is a compiled LCL compatible dblib.dll. But actually i'm waiting for rautgigi's patches... I see some leaks for Lazarus: Actually is not characterset setting included which is needed for the libiconv.dll. Michael you can help us here too.. I've got a lot of more things on my todo list.
According the odbc part i've to pass. No real *nix expieriences.
Michael
Best regards, Michael
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
miab3,
For MS SQL in unix helps only wine http://en.wikipedia.org/wiki/Wine_(software)
For MS SQL in unix helps only wine http://en.wikipedia.org/wiki/Wine_(software)
Lazarus 1.0.8 fpc 2.6.0
- EgonHugeist
- Zeos Project Manager
- Posts: 1936
- Joined: 31.03.2011, 22:38
@miab3,
that's a property which makes only sence for FPC without Lazarus (because all lazarus controls do expect utf8 strings) and Ansi-Delphi. Someone of the FPC-Coreo had some worries how to assume that encoded strings.
The point is this if you have tnt or other unicode-components for the AnsiDelphis then it is the question which FieldType do they expect. Either ftWideString or ftString for the same column. Both are valid, but AnsiDelphi can not handle real unicode chars with the standart-controls. Since D12_UP we have the UnicodeString and all controls too. The same flexibility i left for FPC without Lazuarus. Choose what ever you want if that makes sence... And have a property to do this.
Michael
that's a property which makes only sence for FPC without Lazarus (because all lazarus controls do expect utf8 strings) and Ansi-Delphi. Someone of the FPC-Coreo had some worries how to assume that encoded strings.
The point is this if you have tnt or other unicode-components for the AnsiDelphis then it is the question which FieldType do they expect. Either ftWideString or ftString for the same column. Both are valid, but AnsiDelphi can not handle real unicode chars with the standart-controls. Since D12_UP we have the UnicodeString and all controls too. The same flexibility i left for FPC without Lazuarus. Choose what ever you want if that makes sence... And have a property to do this.
Michael
Best regards, Michael
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
- EgonHugeist
- Zeos Project Manager
- Posts: 1936
- Joined: 31.03.2011, 22:38
@miab3,
thank you again for such informations, Michal. Don't know if you've noticed it, actually i'm reworking the Prepared statemants of MySQL to go away from the SQL: String preperations of the internal generated statements like FireBird does it. A messy rework because the Prepared statements are totaly incomplete for MySQL but it makes it more unicode safe. And i've added string the string escaping for the PreprepareSQL mode + external sendet statements(actually out of use until it is save to work with the token-strings..)
Michael
thank you again for such informations, Michal. Don't know if you've noticed it, actually i'm reworking the Prepared statemants of MySQL to go away from the SQL: String preperations of the internal generated statements like FireBird does it. A messy rework because the Prepared statements are totaly incomplete for MySQL but it makes it more unicode safe. And i've added string the string escaping for the PreprepareSQL mode + external sendet statements(actually out of use until it is save to work with the token-strings..)
Michael
Best regards, Michael
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/
You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/