Author Topic: Replacing xml configuration files with text/binary files  (Read 4107 times)

0 Members and 2 Guests are viewing this topic.

Offline PPK

  • Administrator
  • Emperor
  • *****
  • Posts: 1 475
  • Karma: +209/-22
  • PtokaX developer
Replacing xml configuration files with text/binary files
« on: 06 August, 2011, 23:09:35 »
One of PtokaX problems is storing data (registered users, banlist etc.) in xml files. Reading and writing large xml files is slow and can't be much improved. Another problem with xml is dependency on tinyxml, thing that make compilation on unix systems more complicated that it should be.

So i'm here with idea to replace slow xml with better format for data saving.
Idea is to use simple binary files that i'm designed (and i will use it to store GUI settings) to store PtokaX configuration files. Not only that this should be faster, but it will allow to save changes in regs/bans without writting whole file as it is needed in xml.
In case when it will be needed then files like settings can be saved as simple text files, to allow users to change them...
Of course for few versions PtokaX will be able to read old xml files and convert them to new data format.

So please vote above if this change should be done, and post below if you want to change all files to binary or some files can be saved as plain-text  8)
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

PtokaX forum

Replacing xml configuration files with text/binary files
« on: 06 August, 2011, 23:09:35 »

Offline bastya_elvtars

  • Forum God
  • ****
  • Posts: 3 725
  • Karma: +173/-7
  • The rock n' roll doctor
    • The FreshStuff3 Site
Re: Replacing xml configuration files with text/binary files
« Reply #1 on: 09 August, 2011, 18:14:09 »
How about language files?
Oh, and why not use Lua for editable configuration, so users editing the PtokaX config files will have a familiar syntax when configuring scripts?
« Last Edit: 09 August, 2011, 18:16:06 by bastya_elvtars »
Everything could have been anything else and it would have just as much meaning.

Offline PPK

  • Administrator
  • Emperor
  • *****
  • Posts: 1 475
  • Karma: +209/-22
  • PtokaX developer
Re: Replacing xml configuration files with text/binary files
« Reply #2 on: 09 August, 2011, 18:28:11 »
why not use Lua for editable configuration
Because that not improve performance...
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

Offline bastya_elvtars

  • Forum God
  • ****
  • Posts: 3 725
  • Karma: +173/-7
  • The rock n' roll doctor
    • The FreshStuff3 Site
Re: Replacing xml configuration files with text/binary files
« Reply #3 on: 09 August, 2011, 20:47:37 »
Because that not improve performance...

I know but for language files and the editable configs the performance penalty shouldn't be an issue as they aren't read/written all the time. I am not talking of huge lists (regs, bans) here.
Everything could have been anything else and it would have just as much meaning.

Offline pHaTTy

  • Forum Legend
  • ***
  • Posts: 1 916
  • Karma: +5/-1
Re: Replacing xml configuration files with text/binary files
« Reply #4 on: 09 August, 2011, 21:07:14 »
Since the beginning of XML's existence, I've always said that XML is slow, so i agree that you should put together a binary format for bans and registered users, as for language files, as bastya pointed out, they are not re-loaded repeatedly and therefore the performance decrease would be unnoticeable, especially for todays specifications of CPU's, which would allow for easier editing of languages for third-party translators.

-pHaTTy
Resistance is futile!

Offline bastya_elvtars

  • Forum God
  • ****
  • Posts: 3 725
  • Karma: +173/-7
  • The rock n' roll doctor
    • The FreshStuff3 Site
Re: Replacing xml configuration files with text/binary files
« Reply #5 on: 09 August, 2011, 22:06:07 »
Moreover, settings could directly be read from scripts and there would be no need to maintain the C code of PtokaX wrt the retrieval of them. But the strongest argument IMHO is that since many scripts have configuration options (obviously) written in Lua, PtokaX should also have them this way, so everything could be set up the same way. Even there could be a standardized configuration format, so configuring a script would provide the same 'experience' as configuring PtokaX. It would be a dream for the console version users.
(Personal opinion: XML is not good for humans nor computers :D)
Everything could have been anything else and it would have just as much meaning.

Offline WAJIM

  • Member
  • ***
  • Posts: 41
  • Karma: +3/-5
Re: Replacing xml configuration files with text/binary files
« Reply #6 on: 10 August, 2011, 10:29:40 »
I've no problems with xml's load/save speed. My hub has ~5000 regnicks.  ::)

I vote against a binary format since it will be very difficult for editing manually.  >:(

Offline PPK

  • Administrator
  • Emperor
  • *****
  • Posts: 1 475
  • Karma: +209/-22
  • PtokaX developer
Re: Replacing xml configuration files with text/binary files
« Reply #7 on: 21 August, 2011, 21:46:36 »
I've no problems with xml's load/save speed. My hub has ~5000 regnicks.  ::)
If you have fast CPU then you don't notice that. But hub is freezed for few hundred miliseconds on reglist save  :-\

Test on my AMD Athlon X2 5050e (2.6GHz) with 10000 regs.
XML file: Load 312ms. Save 297ms. File size 1 477 892 bytes.
Binary file: Load 16ms. Save 0ms (crazy :o ). File size 387 827 bytes.


I vote against a binary format since it will be very difficult for editing manually.  >:(
You can add/change/remove regs from GUI or from Lua, i'm sure that 99,9% of users don't need to edit regs/bans file manually  ::)
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

Offline WAJIM

  • Member
  • ***
  • Posts: 41
  • Karma: +3/-5
Re: Replacing xml configuration files with text/binary files
« Reply #8 on: 24 August, 2011, 18:07:58 »
If you have fast CPU then you don't notice that. But hub is freezed for few hundred miliseconds on reglist save  :-\
Even 300ms freezes every 5 minutes cfg autosave (for my scripts) - not critical.  ::)
« Last Edit: 24 August, 2011, 18:09:48 by WAJIM »

Offline PPK

  • Administrator
  • Emperor
  • *****
  • Posts: 1 475
  • Karma: +209/-22
  • PtokaX developer
Re: Replacing xml configuration files with text/binary files
« Reply #9 on: 24 August, 2011, 18:44:45 »
You like laggy hub, others don't. They complained that hub lags when PtokaX saving data, so i'm removed automatic save after changes. Now they complains that they losing data because hub is not saving data after change... I can't fix both when xml is used for data  ::)
« Last Edit: 24 August, 2011, 18:51:35 by PPK »
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

Offline pHaTTy

  • Forum Legend
  • ***
  • Posts: 1 916
  • Karma: +5/-1
Re: Replacing xml configuration files with text/binary files
« Reply #10 on: 25 August, 2011, 11:02:50 »
why not save the data on a separate thread.... it should be easy to implement, setting the thread to low priority will allow auto saves and not cause any lag on the hub, in the time that it saves set a var so when the hub is saving any new data will be added to a data queue, and then added to the list for saving next time around.

-pHaTTy
Resistance is futile!

Offline PPK

  • Administrator
  • Emperor
  • *****
  • Posts: 1 475
  • Karma: +209/-22
  • PtokaX developer
Re: Replacing xml configuration files with text/binary files
« Reply #11 on: 25 August, 2011, 15:17:22 »
Saving in thread will be bad for hub performance, because it is gives more work for main thread ::)
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

Offline pHaTTy

  • Forum Legend
  • ***
  • Posts: 1 916
  • Karma: +5/-1
Re: Replacing xml configuration files with text/binary files
« Reply #12 on: 26 August, 2011, 14:52:36 »
that is absolute rubbish.
Resistance is futile!

Offline PPK

  • Administrator
  • Emperor
  • *****
  • Posts: 1 475
  • Karma: +209/-22
  • PtokaX developer
Re: Replacing xml configuration files with text/binary files
« Reply #13 on: 22 October, 2011, 20:32:59 »
Voting locked. Reglist and banlist will be in binary files. Human readable backups (xml/txt/anything else) can be created from Lua API  :P
« Last Edit: 22 October, 2011, 20:35:06 by PPK »
"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

PtokaX forum

Re: Replacing xml configuration files with text/binary files
« Reply #13 on: 22 October, 2011, 20:32:59 »