Replacing xml configuration files with text/binary files
 

News:

29 December 2022 - PtokaX 0.5.3.0 (20th anniversary edition) released...
11 April 2017 - PtokaX 0.5.2.2 released...
8 April 2015 Anti child and anti pedo pr0n scripts are not allowed anymore on this board!
28 September 2015 - PtokaX 0.5.2.1 for Windows 10 IoT released...
3 September 2015 - PtokaX 0.5.2.1 released...
16 August 2015 - PtokaX 0.5.2.0 released...
1 August 2015 - Crowdfunding for ADC protocol support in PtokaX ended. Clearly nobody want ADC support...
30 June 2015 - PtokaX 0.5.1.0 released...
30 April 2015 Crowdfunding for ADC protocol support in PtokaX
26 April 2015 New support hub!
20 February 2015 - PtokaX 0.5.0.3 released...
13 April 2014 - PtokaX 0.5.0.2 released...
23 March 2014 - PtokaX testing version 0.5.0.1 build 454 is available.
04 March 2014 - PtokaX.org sites were temporary down because of DDOS attacks and issues with hosting service provider.

Main Menu

Replacing xml configuration files with text/binary files

Started by PPK, 06 August, 2011, 23:09:35

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

PPK

One of PtokaX problems is storing data (registered users, banlist etc.) in xml files. Reading and writing large xml files is slow and can't be much improved. Another problem with xml is dependency on tinyxml, thing that make compilation on unix systems more complicated that it should be.

So i'm here with idea to replace slow xml with better format for data saving.
Idea is to use simple binary files that i'm designed (and i will use it to store GUI settings) to store PtokaX configuration files. Not only that this should be faster, but it will allow to save changes in regs/bans without writting whole file as it is needed in xml.
In case when it will be needed then files like settings can be saved as simple text files, to allow users to change them...
Of course for few versions PtokaX will be able to read old xml files and convert them to new data format.

So please vote above if this change should be done, and post below if you want to change all files to binary or some files can be saved as plain-text  8)
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

bastya_elvtars

#1
How about language files?
Oh, and why not use Lua for editable configuration, so users editing the PtokaX config files will have a familiar syntax when configuring scripts?
Everything could have been anything else and it would have just as much meaning.

PPK

"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

bastya_elvtars

Quote from: PPK on 09 August, 2011, 18:28:11
Because that not improve performance...

I know but for language files and the editable configs the performance penalty shouldn't be an issue as they aren't read/written all the time. I am not talking of huge lists (regs, bans) here.
Everything could have been anything else and it would have just as much meaning.

pHaTTy

Since the beginning of XML's existence, I've always said that XML is slow, so i agree that you should put together a binary format for bans and registered users, as for language files, as bastya pointed out, they are not re-loaded repeatedly and therefore the performance decrease would be unnoticeable, especially for todays specifications of CPU's, which would allow for easier editing of languages for third-party translators.

-pHaTTy
Resistance is futile!

bastya_elvtars

Moreover, settings could directly be read from scripts and there would be no need to maintain the C code of PtokaX wrt the retrieval of them. But the strongest argument IMHO is that since many scripts have configuration options (obviously) written in Lua, PtokaX should also have them this way, so everything could be set up the same way. Even there could be a standardized configuration format, so configuring a script would provide the same 'experience' as configuring PtokaX. It would be a dream for the console version users.
(Personal opinion: XML is not good for humans nor computers :D)
Everything could have been anything else and it would have just as much meaning.

WAJIM

I've no problems with xml's load/save speed. My hub has ~5000 regnicks.  ::)

I vote against a binary format since it will be very difficult for editing manually.  >:(

PPK

Quote from: WAJIM on 10 August, 2011, 10:29:40
I've no problems with xml's load/save speed. My hub has ~5000 regnicks.  ::)
If you have fast CPU then you don't notice that. But hub is freezed for few hundred miliseconds on reglist save  :-\

Test on my AMD Athlon X2 5050e (2.6GHz) with 10000 regs.
XML file: Load 312ms. Save 297ms. File size 1 477 892 bytes.
Binary file: Load 16ms. Save 0ms (crazy :o ). File size 387 827 bytes.


Quote from: WAJIM on 10 August, 2011, 10:29:40
I vote against a binary format since it will be very difficult for editing manually.  >:(
You can add/change/remove regs from GUI or from Lua, i'm sure that 99,9% of users don't need to edit regs/bans file manually  ::)
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

WAJIM

Quote from: PPK on 21 August, 2011, 21:46:36
If you have fast CPU then you don't notice that. But hub is freezed for few hundred miliseconds on reglist save  :-\
Even 300ms freezes every 5 minutes cfg autosave (for my scripts) - not critical.  ::)

PPK

You like laggy hub, others don't. They complained that hub lags when PtokaX saving data, so i'm removed automatic save after changes. Now they complains that they losing data because hub is not saving data after change... I can't fix both when xml is used for data  ::)
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

pHaTTy

why not save the data on a separate thread.... it should be easy to implement, setting the thread to low priority will allow auto saves and not cause any lag on the hub, in the time that it saves set a var so when the hub is saving any new data will be added to a data queue, and then added to the list for saving next time around.

-pHaTTy
Resistance is futile!

PPK

Saving in thread will be bad for hub performance, because it is gives more work for main thread ::)
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

pHaTTy

Resistance is futile!

PPK

Voting locked. Reglist and banlist will be in binary files. Human readable backups (xml/txt/anything else) can be created from Lua API  :P
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

SMF spam blocked by CleanTalk