Database saving help
 

News:

29 December 2022 - PtokaX 0.5.3.0 (20th anniversary edition) released...
11 April 2017 - PtokaX 0.5.2.2 released...
8 April 2015 Anti child and anti pedo pr0n scripts are not allowed anymore on this board!
28 September 2015 - PtokaX 0.5.2.1 for Windows 10 IoT released...
3 September 2015 - PtokaX 0.5.2.1 released...
16 August 2015 - PtokaX 0.5.2.0 released...
1 August 2015 - Crowdfunding for ADC protocol support in PtokaX ended. Clearly nobody want ADC support...
30 June 2015 - PtokaX 0.5.1.0 released...
30 April 2015 Crowdfunding for ADC protocol support in PtokaX
26 April 2015 New support hub!
20 February 2015 - PtokaX 0.5.0.3 released...
13 April 2014 - PtokaX 0.5.0.2 released...
23 March 2014 - PtokaX testing version 0.5.0.1 build 454 is available.
04 March 2014 - PtokaX.org sites were temporary down because of DDOS attacks and issues with hosting service provider.

Main Menu

Database saving help

Started by GeceBekcisi, 11 September, 2005, 17:40:29

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

GeceBekcisi

My hub took about 50 minutes to save db of IPdbBOT's which is produced in 11 days with a hub of 1000+ users (7.885.887 bytes in size). I have an idea, like seperating IP History and ISP&DNS lookup results like:
----------------------------------------------
["213.139.225.18"] = { 
	["Nicks"] = { 
		["highscreenn_"] = { 
			[1] = "08.09.2005 - 11:28:32", 
			[2] = "09.09.2005 - 16:01:59", 
		}, 
		["hpcompaqp"] = { 
			[1] = "09.09.2005 - 07:44:46", 
			[2] = "09.09.2005 - 16:28:34", 
		}, 
	}, 
	["ISPName"] = "TR?-?Meteksan?Net", 
	["DNSHost"] = "Couldn't?resolve?remote?host?name", 
}, 
---------------------------------------------- 
to:  
----------------------------------------------
History DB:   
                         
213.139.225.18|highscreenn_|08.09.2005 - 11:28:32|09.09.2005 - 16:01:59| 
213.139.225.18|hpcompaqp|09.09.2005 - 07:44:46|09.09.2005 - 16:28:34| 
                         

IP DB:   
                         
213.139.225.18|TR?- Meteksan?Net|Couldn't?resolve?remote?host?name|
----------------------------------------------
but have no idea if it will speed up save part or speed down whole script..

Any ideas to fix this will be greatly welcomed... Thanks
Do you need an advanced user handling script? Download UserBekcisi today (Latest Edit)
Features: User + ISP + GeoIP database, user info + share checking and many more...

plop

can you show the routine which saves the stuff.

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

GeceBekcisi

Sure, I use File Serializing by nErBoS
tFunctions.Serialize = function(tTable, sTableName, sTab)
	assert(tTable, "tTable equals nil");
	assert(sTableName, "sTableName equals nil");
	assert(type(tTable) == "table", "tTable must be a table!");
	assert(type(sTableName) == "string", "sTableName must be a string!");
	sTab = sTab or "";
	sTmp = ""
	sTmp = sTmp..sTab..sTableName.." = {\n"
	for key, value in tTable do
		local sKey = (type(key) == "string") and string.format("[%q]",key) or string.format("[%d]",key);
		if(type(value) == "table") then
			sTmp = sTmp..tFunctions.Serialize(value, sKey, sTab.."\t");
		else
			local sValue = (type(value) == "string") and string.format("%q",value) or tostring(value);
			sTmp = sTmp..sTab.."\t"..sKey.." = "..sValue
		end
		sTmp = sTmp..",\n"
	end
	sTmp = sTmp..sTab.."}"
	return sTmp
end
tFunctions.SaveToFile = function(file , table , tablename)
	local handle = io.open(file,"w+")
	handle:write(tFunctions.Serialize(table, tablename))
	handle:flush()
	handle:close()
end
Do you need an advanced user handling script? Download UserBekcisi today (Latest Edit)
Features: User + ISP + GeoIP database, user info + share checking and many more...

plop

try splitting the table into several files, and save only the changed things.
just this needs some extra stuff 2 keep track of the changes which decreases the overal preformance.
if that doesn't help your gone needs some extra fancy database system (files and folders), so you can minimize the 2 be saved (changed) data.

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

GeceBekcisi

QuoteOriginally posted by plop
save only the changed things.

how will I do that? I dont have any ideas how to keep track of changes only, and saving file without creating duplicates in append mode...
Do you need an advanced user handling script? Download UserBekcisi today (Latest Edit)
Features: User + ISP + GeoIP database, user info + share checking and many more...

plop

QuoteOriginally posted by GeceBekcisi
QuoteOriginally posted by plop
save only the changed things.

how will I do that? I dont have any ideas how to keep track of changes only, and saving file without creating duplicates in append mode...

compair the curent vallue with the vallue in the table.
lua flushes duplicate vallue's in tables on loading it, it keeps the last vallue it finds.

plop
http://www.plop.nl lua scripts/howto\'s.
http://www.thegoldenangel.net
http://www.vikingshub.com
http://www.lua.org

>>----> he who fights hatred with hatred, drives the spreading of hatred <----<<

bastya_elvtars

#6
Whenever a value changes:

local f=io.open("DB_File","a+"); f:write(changed_value); f:close() end

Hope this helps.
Everything could have been anything else and it would have just as much meaning.

GeceBekcisi

writing files in to only a file in append mode needs a lot of work to be done, so I decided to test/use another method. so now, I need help about writing to files as,
    [*] They will have filenames of current date as sFileName = os.date("%d-%m-%Y)
    [*] On 00:00, writing to current file should be stopped and a new file should be opened with updated current date[/list]and also, if I can achieve this, these files will be appended and this will speed up most of the things..

    SO, I need help about writig files in this method.. Thanks for any help..
    Do you need an advanced user handling script? Download UserBekcisi today (Latest Edit)
    Features: User + ISP + GeoIP database, user info + share checking and many more...

    SMF spam blocked by CleanTalk