duzenko wrote:
Too bad - the way you write here, I already did take offense.
Why, was it your personal decision to quadruple the advertised field size?
No - it wasn't my decidion but that has nothing to do with it. You come to this forum and ask for help. First off you are just throwing pieces of information at us that are by far not enough to diagnose what is happening on your side. When asked to provide the necessary information you insult me by stating that this information is not necessary for the diagnosis. If you know that much why ask for help here anyway? Then we point yout that the design desisions in your project have to do with the problem. You jump in our faces telling us that this is all our fault, making demands, trying to tell as what we have to do in our project and finally insulting me to be a liar: "[...]but also you defend it as a 'fix' while to me it looks you just found out about it.". You are free to ask questions, make suggestions and voice opinions. Also I suggest you treat us like we know what we do and keep in mind that we probably have reasons to do what we do, even if you don't like it. Also plese keep in mind that we are not selling a product but we share our work with you and support you in our spare time.
I am not going to discuss this matter any further. If you want help please provide details in the future on what you did and how your environment is. The minimun set of informations being your compiler type and version, the version of Zeos you use and the database you use. All these things have great impact on many technical details even though you might not realize it because on some ocassions Zeos just might be working different from what oyu expect it to work.
By the things you write I get the impression that you miss some serious points:
- TStringField is totally broken for any character set that uses more than one byte per charachter - Big5 for example or UTF8. This can lead to Buffer overruns on systems that use that kind of character set on ANSI Delphi or Lazarus. Zeos tries to accomodate for this by modifying TStringField.Size. You may not like this but that's the way it is.
- TWideStringField also has a design flaw. It assumes that one character is two bytes in Size. But this only is true for characters and symbols on the Basic Multilingual plane (BMP). Because of this it is by no means a proper replacement for UTF8 because UTF8 (as well as UTF16) can encode the whole Unicode standard and not only the BMP.
- The database character set has absolutely nothing to do with the connection character set. Firebird will happily convert data to and from the client character set and store it in UTF8 in the database. This is the recommended way to go in ANSI environments like Delphi 2007.
- Just to add one more point to this character madness: Databases treat Unicode data and MBCS very differently: Firebird always assumes VARCHARs to be specified in an amount of characters. Microsoft SQL Server specifies VARCHAR to always be in an amount of bytes. Oracle has options to specify varchars as an amount of bytes to be stored as well as an amount of characters to be stored. Also NVARCHARs are trated differently by MS SQL and Oracle - to make things even worse.
So getting things right for every database Zeos supports is not as simple as it may seem on a first glance.
I created a wiki entry on SopurceForge to document current behaviour of Zeos:
Zeos and Character Sets. This behaviour will be kept for Zeos 7.2. We might change it again for Zeos 7.3.
duzenko wrote:Well, my problem is the above mentioned breaking change was not only mentioned NOWHERE in the docs or the forum but also you defend it as a 'fix' while to me it looks you just found out about it. Am I wrong and it's a common knowledge? Did anyone anywhere on this forum or elsewhere bring it up yet? Do you think that being upset about this is unreasonable?
Being upset is allright. But insulting people and doing demands on people who do this job in their spare time and only request you to be polite because they also have their burdens in their life is unreasonable.
duzenko wrote:We did the decision that we support users pushing UTF8 directly to the database using TStringField in ANSI Delphis.
ANSI Delphis have TWideStringField exactly for that purpose. That's where you were wrong at the start and everything else is consequences.
That might be your opinion. We have a different one. Live with it.
duzenko wrote:In this use case the user is responsible for making sure that not more than the allowed amount of characters gets pushed into the TString Field.
And the field length checks in TStringField flush down the drain. It's only something Delphi programmers relied on for decades now - no big deal.
Yes - because we expect developers who request thg database to send _anything_ that differs from the system code page on an ANSI Delphi to know exactly what they are doing.
duzenko wrote:There is no such support as using bools as Firbird Char(1) fields. You relied on behaviour that worked but was not intentionally working. At least from a Zeos point of view.
No support for meaningful TField.Size values as well?
Not if you request the database to send UTF8.
duzenko wrote:Even if Zeos would do some kind of check here, it wouldn't be a truncation but raising an exception that there are too many characters stored in the field.
You don't have to do any checks at all, just return the actual field length. And surely not to try to convert WideString to Utf-8 or vice versa.
The whole point was: If you get TSringField.Size extended and Zeos would do the check wether the amount of characters exceeds the amount of characters that the database can store it wouldn't be a silent truncation (as it was in Zeos 7.1) but an exception. Basically the same Exception that Firebird raises for you.
duzenko wrote:No - it wasn't. Do we need your approval before we make changes to the Zeos code? Do you want to step up and support or do Zeos development? If you don't want to do any of that then why do you demand that we discuss the development of Zeos with you before we do changes?
I want to read on the reasons why char(1) has Size=1 in 7.1 and 4 in 7.2. If there were any. The breaking changes need to be explained - am I asking too much?
I have done small contributions in the past and I don't see what it has to do with anything.
If you were a team member you could argue that design decisions for Zeos need to be discussed with you too. For the reason why CHAR(1) results in TField.Size = 4 see the above mentioned Wiki page. As for explaining breaking changes: We usually try to document that kind of things. Sometimes changes that break functionality either are not seen to break functionality or simply get overlooked for documentation. People make erorrs. But that is no reason to be rude. If you want to read documentation I suggest you ask questions and document answers. You may not realize it but writing documentation on a complex project like Zeos is time consuming. Getting the above mentioned Wiki page right took me about one to two hours.
duzenko wrote:
You still need to truncate to real field length in character, but do that on a WideString rather than AnsiString
TStringField already does that (did until you broke it in 7.2). No way I am running around the source code manually truncating each and every field I'm sending to the database. And surely it has nothing to do with WideStrings unless you confuse AnsiStrings with utf8's.
I am not going to comment on this because that text about conversions was from your own post:
http://zeoslib.sourceforge.net/viewtopi ... 13#p110413
duzenko wrote:Again: Zeos does no intentional truncation. Delphi is doing the truncation in this example because it happens on this line of your example code:
Code: Select all
ZTable1['some_bool'] := Random(2) = 1;
During this assignment Delphi decides to truncate the string value of "False" to Fals or to F - depending on TStringField.Size.
God forbid you started truncating stuff in Zeos on top on what you already messed up with field Size.
I am not going to comment on this.
duzenko wrote:Zeos (correctly) just sends the value that gets stored in the StringField to the database that in turn correctly raises an exception.
Oh yeah? And where does the TStringField.Size come from?
See the above mentioned Wiki page.
duzenko wrote:marsupilami wrote:
Soo - to bring this to an end:
- To me it seems that your connection character set is UTF8 while you use an ANSI Delphi. Why? This simply doesn't make sense. You might want to consider using the correct ANSI character set for your database connection. ZConnection.ClientCodepage='WIN1252' for example.
Because the DB is used by other applications but mine and some of them require the DB to be UTF-8.
If that is your whole reason to use UTF8 as the client character set, you might want to read up on the difference between the database character set and the connection character set. Firebird will do all necessary conversions for you if you use it in the right way. This requires a correct database design obviously where character set none doesn't get used. But Firebird might hit you in the face with an error if it cannot convert the data you request to your CodePage. And that might be the reason why you really use UTF8 - because somebody didn't like these errors and decided that it is ok if Zeos does these conversions using Windows funtions and looeses some datea on the way. To get that behaviour back use the current SVN version of Zeos and set AutoEncodeStrings to true for your application. The current SVN version also has a fix that gets TStringField.Size back to the expected size if you use ControlsCodePage=cGET_ACP.
duzenko wrote:[*]Another option might be to use ClientCodepage=UTF8 and ControlsCodePage=cCP_UTF16. On the SVN version of Zeos 7.2 this will make Zeos generate TWideStingField instead of TStringField with the correct number of characters. I am not 100% sure if the release version Zeos 7.2.4 will behave in the same way. Watch out for type casts like TStringField(DataSet.FieldByName('somefield')) in your code in that case because they might be incorrect afterwards and cause access violations then.[/list]
That changes exactly nothing. It's still Size=4 for char(1) with the inevitable runtime errors.
Honestly - please provide a test application and a database script. If you don't start supplying enough information to know exactly what you do, we will have to work like this. I still don't know wether you got the expected TWideStringField - which shouldn't get a Size of 4 under any circumstances in this case - or if you still get a TStringField. Also I still don't know wether you did the test on Zeos 7.2.4 or on the SVN version of Zeos 7.2.
duzenko wrote:Soo - to bring this to an end:
- Consider one of the following
- Ban my forum account cause I'm not going to agree and call this bug an 'improvement'
- Agree to optionally return the ACTUAL field size
- Agree to optionally use TWideStringField's on unicode databases instead of TStringField
See - there are a lot of other options for this. One of them is to simply ignore future posts by you. Another option is that you finally accept that we have our reasons for doing what we do. I don't need to ban you and I don't need to agree with you on anything.
Best regards,
Jan