Author Topic: Flushing the Garbage that stores in the hub... How???  (Read 5460 times)

0 Members and 1 Guest are viewing this topic.

Offline NightLitch

  • Emperor
  • **
  • Posts: 1 455
  • Karma: +16/-0
Flushing the Garbage that stores in the hub... How???
« on: 01 January, 2004, 23:12:55 »
Hey Guys how do I flush the Garbage that stores in memory...

this is my code and it apperetly not work...

Code: [Select]
function FlushMem(curUser,data)
local _,_,cmd,table = strfind( data, "%b<>%s+(%S+)%s+(.*)" )
if table == nil or table == "" then
curUser:SendTo(BotName," Table is not found...")
return 1
end
if table == "Counter" then
tName = "Counter"
elseif table == "UserInfo" then
tName = "User Info"
end

table = {}
table = nil
collectgarbage()
flush()
curUser:SendTo(BotName, tName.." found, Memory is Flushed...")
end


plz help me...
//NL

PtokaX forum

Flushing the Garbage that stores in the hub... How???
« on: 01 January, 2004, 23:12:55 »

Offline plop

  • Forum God
  • ****
  • Posts: 2 464
  • Karma: +37/-0
(No subject)
« Reply #1 on: 02 January, 2004, 03:18:52 »
table = nil
collectgarbage()
flush()
table = {}

this should work better, 1st you make all tables and other things nil then you flush and finaly you make the tables again.

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

Offline NightLitch

  • Emperor
  • **
  • Posts: 1 455
  • Karma: +16/-0
(No subject)
« Reply #2 on: 02 January, 2004, 23:51:06 »
ahh... thx Plop... you're always there helping me... thx...
//NL

Offline NightLitch

  • Emperor
  • **
  • Posts: 1 455
  • Karma: +16/-0
(No subject)
« Reply #3 on: 03 January, 2004, 00:36:37 »
One other thing, GUYS!!!

What is better when storing offline users info...

Seperat files or one big ???

and when I have the subject up,

How is the best function done so it is not in memory all the time...???

hope you understand...
//NL

Offline ptaczek

  • Administrator
  • Triple Ace
  • *****
  • Posts: 156
  • Karma: +328/-0
(No subject)
« Reply #4 on: 03 January, 2004, 01:30:59 »
Hi, it would be best to have a database-aware lua ;) Now seriously. It all depends on how frequently users log in and out. Consider following facts:

- in file-per-user case you will end up with filesystem troubles as well as aMutex did with first prototypes of ptokax userstats, where we had two files for every user. Moreover filenames have restrictions and users often use weird nicknames.

- in one-file-for-all case you will run in troubles with lua. Either you will need to keep the whole file in a table or you will have to parse the big file on every login/logout to update the data. And that's a real problem. Lua's excelent performance depends mainly on caching of frequently used data such as strings or their parts. During script runtime lua builds huge strings dictionary. Every single strfind, or str1..str2 concatenation (simply EVERY string operation) internally makes a dictionary lookup and if some operand isn't found, it is hashed and added to the dictionary, new tiny data structure is stored in newly allocated memory. The more you use complex string operations in script the more hungry the script will be and the more memory will be wasted. Even if you will assign nil to your variables and use garbage-collecting, the dictionary keeps getting bigger and bigger and dies only at the script restart.
Im trying to 'hack' deeper into the lua's source codes to fully understand the caching mechanism to be able to limit or disable it or to have a chance to flush the cache on demand. I know it would be helpfull for all of us ;)

So the almost_good advice is somwhere between those two cases. I would use several files for alphabetical groups. Something like a-c.dat, d-f.dat etc. and store individual user's info there in some nice form which would be parsed with minimal set of operations. In such case you will have to read_parse_update_and_store less data on user's login/logout and the filesystem demands will be kept low.
Also very good approach is to cache login/logout events for some time (a minute or maybe more) without any file reading/writing and update the cached data all at once on timer tick. Such technique is widely used in operating systems, filesystems and database systems. The amount of data stored in memory is small + the expensive I/O operations are performed at longer time intervals.
But as always, you will have to try, which technique will be both the resources AND performance efficient ;)


btw: have a nice-and-successfull year 2004 :o)
-ptaczek-

This whole physical universe is a hologram.
[Cosmosis - Contact: The First Step]

Offline pHaTTy

  • Forum Legend
  • ***
  • Posts: 1 916
  • Karma: +5/-1
(No subject)
« Reply #5 on: 03 January, 2004, 10:37:24 »
bt from what i see i see some memory waster, why use a table for a variable?

table = ""

table = {}

if table will only ever use one word then theres not really any point using a table,

for example

table = "test"

then if tabel == blalba then

for if cmd bla

table = "whatever"

now from what i see you are also adding it to an emtpy table, which technically it wont add it i suppose, unless table["whatever"] but it doesnt make much differ but really unless you will be adding more then 1 thing to the table then its pointless with table = {}

l8rr

-phatty ;)
Resistance is futile!

Offline c h i l l a

  • Lord
  • ***
  • Posts: 347
  • Karma: +1/-0
(No subject)
« Reply #6 on: 03 January, 2004, 12:38:26 »
Quote
 ptaczek- in file-per-user case you will end up with filesystem troubles as well as aMutex did with first prototypes of ptokax userstats, where we had two files for every user. Moreover filenames have restrictions and users often use weird nicknames.


thats how the logger, does it,  just it needs one file per user ans one table entry per user,  which is always in memory so I don't need to save the file to the useres nick name, I have a table entry where I have the nickname and the file to the nickname,  so no problems with names,  but a table with all nicks...

Quote
ptaczek - So the almost_good advice is somwhere between those two cases. I would use several files for alphabetical groups. Something like a-c.dat, d-f.dat etc. and store individual user's info there in some nice form which would be parsed with minimal set of operations.


also nice idea, but I also got many users with breckats infornt of their nick, somtimes half of the the hub, so when I would open this file, then I can almost leave em' all in a table, dunno, one file per users is a good thing, and the memory for a table with a user name and a refering file name isn't too big, but true you will need to test it out, what suits better,  I haven't tried the logger on over 1000 users yet, over 200  the memory is around
 8 MB  ,  if I extrapolate it linear then for 1000 users it would be 40 MB,  not too much I think...

Quote
 ptaczek  -  Also very good approach is to cache login/logout events for some time (a minute or maybe more) without any file reading/writing and update the cached data all at once on timer tick. Such technique is widely used in operating systems, filesystems and database systems


to add, also used in the logger, also adjustable.

To the strfind problem... yepp indeed this is  something, cause how to get info, from a string if not by strfind, but I will keep in mind not to use many strfinds and such..

Offline NightLitch

  • Emperor
  • **
  • Posts: 1 455
  • Karma: +16/-0
(No subject)
« Reply #7 on: 03 January, 2004, 12:43:46 »
I would want you guys to ignore my above code... cause
I have from Plop's help found another way...

And Thx Ptaczek for the information, I don't now if my current
change in my userinfo store is better.. gonna check it out
in a couple of days...

I made my function like this:

user connect / nothing exept some checks bad nick & client / user logged if checks ok

user disconnect / if user in log the get user info / Userinfo stored into seperat file Userinfo table = nil and then
 collectgarbage() / flush() / table = {}

and when using command:

!info user

if online taken from user online, if offline users file opend
and info taken building table / then table = nil and then
collectgarbage() / flush() / table = {}

this is a good way or is it bad ???

Hope you guys understand me now ;-p
//NL

Offline c h i l l a

  • Lord
  • ***
  • Posts: 347
  • Karma: +1/-0
(No subject)
« Reply #8 on: 03 January, 2004, 13:06:45 »
so its again..  reading and writing, ondisconnect :P

Offline NightLitch

  • Emperor
  • **
  • Posts: 1 455
  • Karma: +16/-0
(No subject)
« Reply #9 on: 03 January, 2004, 13:43:13 »
well where should I do it then... help me then plz...
//NL

Offline plop

  • Forum God
  • ****
  • Posts: 2 464
  • Karma: +37/-0
(No subject)
« Reply #10 on: 03 January, 2004, 15:20:24 »
ok small stuff about how i work.
i have a file per type of info (infostring, good/bad connects, comments, etc....9 files in total).
only saving the full MyInfoString on exit but only after x users, no need 2 waste time on a strfind 2 get all vallue's seperatly.
that x returns everywhere, only if that amount of changes are done it saves to file (except some rare things or not optimized parts).
last database i got from odin was 2MB and had stuff from 10.000 users in it.
that is fully stored in memory 2 get max speed, so only saving for reboots.
i have a special routine witch gets triggered around midnight witch can do a couple things, or flush the full database if bigger then xMB, clean database, or a mix of those 2 only clean if bigger then xMB.
if posible i keep everything local.

so in short i only save on x changes or x users leaving.
using a file per type so i only save what needs 2 be saved.

gotta check that idea from pcaz 2 use files on parts of the alphabet, sounds good.

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

Offline c h i l l a

  • Lord
  • ***
  • Posts: 347
  • Karma: +1/-0
(No subject)
« Reply #11 on: 03 January, 2004, 15:21:08 »
i was just beein a little childish..  hmm..  dunno actually..
and I don't know if its necessary to flush the garbage after each.   !info  command unless you have many many,  infos...  but else I think its enough to collect the garbage once in while and flush it. Else the LUA clears that garbagecollecter by itsself from time to time..

and as ptaczek said,  maybe use a timer,  to store the logs, the logger isn't perfect, see it also reads from disk when a user disconnects...   but it only writes when I say so. But I amthinking of something else already. But one can't make the timespan to long or else one will check for the userlog and is isn't uptodate... dunno maybe 5 minutes is a good time.
« Last Edit: 03 January, 2004, 15:21:36 by c h i l l a »

Offline c h i l l a

  • Lord
  • ***
  • Posts: 347
  • Karma: +1/-0
(No subject)
« Reply #12 on: 03 January, 2004, 15:24:30 »
2 MB for 10.000 users, realy ?? The table or the file?
When I make a 10.000  entiry big table I am way over 2 MB.
« Last Edit: 03 January, 2004, 15:25:43 by c h i l l a »

Offline plop

  • Forum God
  • ****
  • Posts: 2 464
  • Karma: +37/-0
(No subject)
« Reply #13 on: 03 January, 2004, 17:14:26 »
Quote
Originally posted by c h i l l a
2 MB for 10.000 users, realy ?? The table or the file?
When I make a 10.000  entiry big table I am way over 2 MB.
Code: [Select]
[16:59] <-Bure->

  Here are the file sizes of all data files
=====================================
  String 457.92 KB
  IP 139.37 KB
  Seen 154.46 KB
  Time 67.78 KB
  Total time 82.58 KB
  Comments 334.09 KB
  Succesfull connects 74.75 KB
  Failed connects 47.84 KB
=====================================
  Total size 1358.79 KB
 
this now holds info from about 6589 users.
storing this in memory takes about 2x as much, current mem for the bot is 4MB, garbage col is on 8MB.
Time is messed up as it's taken from odin's hub while running, it holds a load of users who aren't in my hub.
i didn't use a timer for 2 reasons:
1) didn't have a free timer and didn't want 2 use it for more then 1 thing.
2) timer would mean the hub keeps saving even if the hub is quiet (night, no logouts).

flushing the stuff is indeed useless 2 do after everything, i only do it after building a long line/text 2 send 2 a user or in a cleanup job.

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

Offline c h i l l a

  • Lord
  • ***
  • Posts: 347
  • Karma: +1/-0
(No subject)
« Reply #14 on: 03 January, 2004, 17:31:39 »
nice nice..  on the other side, you do wirte and read more often  when many connects and disconnects..  so a timer is indeed usefull. And to not write any data, on timer, is easy just check before the timer triggers,  if new data is there.

but plop,  you think i can look at your script...  just to get maybe some ideas,  I mean  6589 userlogs, okey I don't know what you store, but I know to store for exsample 10.000 names I need already around 3 MB

Offline NightLitch

  • Emperor
  • **
  • Posts: 1 455
  • Karma: +16/-0
(No subject)
« Reply #15 on: 03 January, 2004, 18:48:49 »
That is one thing I wanna know how I script...

Can I see the memory usage though scripting-command

And how plz show me with a function plz...

And that thing you have made Plop... how have you done that...

Curious like never... :-)
//NL

Offline c h i l l a

  • Lord
  • ***
  • Posts: 347
  • Karma: +1/-0
(No subject)
« Reply #16 on: 03 January, 2004, 19:15:58 »
Quote
Here are the file sizes of all data files


so I assume its only the file size...
so the filesize is easy to get by a seek  function

seek(filehandle, "end")  guess thats what he uses.

Offline NightLitch

  • Emperor
  • **
  • Posts: 1 455
  • Karma: +16/-0
(No subject)
« Reply #17 on: 03 January, 2004, 19:52:46 »
ok, but what about the memory thing I was typing
up there...

is it possible to view memory usage in client from bot ??

and how if able...
//NL

Offline plop

  • Forum God
  • ****
  • Posts: 2 464
  • Karma: +37/-0
(No subject)
« Reply #18 on: 03 January, 2004, 20:30:42 »
well 1st of all, a.i. bot is not open source.
this because the users from our network have asked me this 2 protect it from fakers who wanne find out how i ban there asses on entry.
i save everything as raw as posible, for example i could save share/email/slots/hubs nicely in single tables so it's nice and fast on getting it out of the database.
but this would mean a hell of a lot of work on saving, makes me allready sweat if i think of it. lol
saving it as table looks nice, loads fast BUT, loading is only done on starting the bot so saving that way is wasting power.
i save really simple write(user.sName..":"..user.sMyInfoString.."\n").
there are a couple things you always gotta keep in mind.
what do i really need for info now or later, how can i save it as fast as posible with the least amount of power for proccessing, do i really need it, always use (if posible) an associative array for storing the stuff in memory.
many ppl think something witch uses less memory is faster then something witch uses more, somethimes there right but mostly they are really wrong.
keeping stuff in memory always works faster then a hdd, but then it also counts how you keep it in memory.
by using a normal table and having 2 change something in that i have 2 co thru all the vallue (fake share patch foreachi bla bla bla).
i rewrten this 2 be as associative array, i simple check if the share of the user is in the array with 1 check and if nil the user isn't using the patch, this means simply 1 check instead of compairing it with xx vallue's 1 by 1.
mostly i 1st write something see if it works, then check back what can be dropped/optimized, see if it still works and later again check back 2 see if i can optimize it more, reguarly i even rewrite the whole function again (somethimes this means the hub owners have 2 delete files saved by the bot because the new version can't read them anymore).
the less lines the better, the less complex the lines the better (1x i stripped 800+ lines on 1 single optimize job from the 8000 lines the full script had back then).

for the point chilla gave about saving on x logouts on busy hubs: that max counter vallue can be set from the config.
on my hub i use 1 as max (this because i restart the scrips reguarly), odin runs on i believe 25 but on the main hub i used 100 (1400 users and it saved about every 15 mins).

night: you can get the filesizes easely but maby even the bot's memory, never tryed that last.
Code: [Select]
--------------------------------------------------------------------- returns the size of a file in KB
function GetFileSize(file)
   _INPUT = readfrom(file)
   size = format("%0.2f", (seek(_INPUT, "end")/1024))
   readfrom()
   return size
end

if you 2 want i can give you a compiled version so you can see better how i save the things but i won't give out the source.
all in all a long story. lol
hope you get what i mean.

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

Offline c h i l l a

  • Lord
  • ***
  • Posts: 347
  • Karma: +1/-0
(No subject)
« Reply #19 on: 03 January, 2004, 20:53:29 »
okey, plop,  thanks...

okey.  thanks for telling how you work...  on your script...
alsways good to get more ideas, thats why I like readable scripts, so no need to compile it for me ;).
But I guess...  it can't be used for the Logger, the logger depends on exact tables..
so once a person is logged, I need no strfind anymore,  okey..  sure when I catch the command, but then never again.

Offline NightLitch

  • Emperor
  • **
  • Posts: 1 455
  • Karma: +16/-0
(No subject)
« Reply #20 on: 03 January, 2004, 21:16:36 »
I can see now that I have really MUCH too learn still...
but hey, I have build my own bot from scratch without
paste & copy... :-)

exept some functions I got from you guys...

Think I can optimize my bot much more than I think it is...
after reading all your stuff...

ThX guys
//NL

Offline plop

  • Forum God
  • ****
  • Posts: 2 464
  • Karma: +37/-0
(No subject)
« Reply #21 on: 03 January, 2004, 21:19:02 »
it all depends on taste i gues, you get stuff faster from the database i can save it faster.
my script is very readable for me. lol
when i get some time i'm gone release a new userinfo script with this database, just some things are gone be removed as they need the rest of the a.i. 2 function.
then you can fully see how i do all of that stuff.
i'm a real associative array junky, i hate if, elseif, elseif, etc....
a array like that takes some memory but it works a lot faster.
if you check crazy bot you can get an idea of how a.i. works.

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

Offline plop

  • Forum God
  • ****
  • Posts: 2 464
  • Karma: +37/-0
(No subject)
« Reply #22 on: 03 January, 2004, 21:23:21 »
Quote
Originally posted by NightLitch
I can see now that I have really MUCH too learn still...
but hey, I have build my own bot from scratch without
paste & copy... :-)

exept some functions I got from you guys...

Think I can optimize my bot much more than I think it is...
after reading all your stuff...

ThX guys
i also learn something new every day, thats why i check back 2 see if i can optimize something a bit more with the new stuff i learned.
i don't have 2 rewrite the whole bot as some other do because of this.
and it also saves me a lot of time. lol

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

Offline c h i l l a

  • Lord
  • ***
  • Posts: 347
  • Karma: +1/-0
(No subject)
« Reply #23 on: 06 January, 2004, 22:05:38 »
a little update.. with some tests I made..  where a script creates some garbage and then release it...

so
okey the mem stats in ptokax is the mem used the garbagethreshold is the size the memory can get before the garbage released.

to set the memthreshold.

just do this

collectgarbage(bytes)  exsample  1024*6  = 6 MB

if the memory gets bigger than 6 MB then its released the garbage. but then ptokax set the eme threshold back to its own, value, so you als need to update the collectgarbage again.

ptokax or LUA set teh garbagecollector to a certain value above the memory the script uses.
A reasonable value I think, which makes collecting garbage unusefull, unless you open 100 MB mem at once and thorw em' away again at once.

so to throw em' away at once all you need to do is.

collectgarbage()

done.

PtokaX forum

(No subject)
« Reply #23 on: 06 January, 2004, 22:05:38 »