GeoIP script - Page 2
 

News:

29 December 2022 - PtokaX 0.5.3.0 (20th anniversary edition) released...
11 April 2017 - PtokaX 0.5.2.2 released...
8 April 2015 Anti child and anti pedo pr0n scripts are not allowed anymore on this board!
28 September 2015 - PtokaX 0.5.2.1 for Windows 10 IoT released...
3 September 2015 - PtokaX 0.5.2.1 released...
16 August 2015 - PtokaX 0.5.2.0 released...
1 August 2015 - Crowdfunding for ADC protocol support in PtokaX ended. Clearly nobody want ADC support...
30 June 2015 - PtokaX 0.5.1.0 released...
30 April 2015 Crowdfunding for ADC protocol support in PtokaX
26 April 2015 New support hub!
20 February 2015 - PtokaX 0.5.0.3 released...
13 April 2014 - PtokaX 0.5.0.2 released...
23 March 2014 - PtokaX testing version 0.5.0.1 build 454 is available.
04 March 2014 - PtokaX.org sites were temporary down because of DDOS attacks and issues with hosting service provider.

Main Menu

GeoIP script

Started by VidFamne, 13 April, 2004, 15:03:40

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Skrollster

QuoteOriginally posted by (uk-kingdom)pH?tt?
well actually ur wrong i done some testing and they do become global after accidently finding that when writing a script, globally calling a local varaiblethat was localaised in a function, and its not unreadable, i just cant use it when the lines are all the way down the page, sop its a habit to keep everything together

similar locals go together and as for ifs, like to keep em close to the old end if they have sep end, less cr*pcode to look at, and easier for me to read tbh, so if ya cant read that u need ot start practicing

n fact when i run hub later, i will probby make a script to show ya, but tbh im pretty pizzed off atm so ima go...
:)

function test()
	local aVariable = "Just a test"
	if aVariable then
		local aVariable = "Just a test again"
		write(aVariable)
	end
	write(aVariable)
end

test()
write(aVariable)

		--> Just a test again
		--> Just a test
		--> error a nil value

Skrollster

#26
for all out there who wants to update the tGeoIpRange.txt

download the cvs file from: http://www.maxmind.com/app/geoip_country

use a texteditor that can handle regular expressions and

replace:

^("[0-2]?[0-9]?[0-9]\.[0-2]?[0-9]?[0-9]\.[0-2]?[0-9]?[0-9]\.[0-2]?[0-9]?[0-9]",)("[0-2]?[0-9]?[0-9]\.[0-2]?[0-9]?[0-9]\.[0-2]?[0-9]?[0-9]\.[0-2]?[0-9]?[0-9]",)("[0-9]+",)("[0-9]+",)("[A-Z][A-Z0-9]")(,".+")$

with:

{\3\4\5\6},

addthis in the begining of the file:
tGeoIpRange = {
and
}
in the end

chill

#27
hmm.. just looking at the search func, what it actually does is run though the table with from i = 1,getn(table)
I sorta don't see where the bost is ;).
Wonder why nobody noticed, lol.
 you might wanna try this

function tSearch(table,comIP)
	local iStart,iEnd,iMid = 1,getn(table),0
	while (iStart <= iEnd) do
		iMid = floor((iStart+iEnd)/2)
		if (table[iMid][1] <= comIP) and (table[iMid][2] >= comIP) then
			return table[iMid]
		elseif (table[iMid][1] > comIP) then
			iEnd = iMid - 1
		elseif (table[iMid][1] < comIP) then
			iStart = iMid + 1
		end
	end
	return("NOTFOUND")
end

Skrollster

this is just a matter of programing style.. but since it has been a topic here at the forum latly i'd like to show how i would have solved it the same way as chill.. just a diffrent syntax instead of while i use repeat

function tSearch(table,comIP)
	local iStart = 1
	local iEnd = getn(table)
	repeat
		iMid = floor((iStart+iEnd)/2)
		if (table[iMid][1] <= comIP) and (table[iMid][2] >= comIP) then
			return table[iMid]
		elseif (table[iMid][1] > comIP) then
			iEnd = iMid - 1
		elseif (table[iMid][1] < comIP) then
			iStart = iMid + 1
		end
	until iStart <= iEnd
	return("NOTFOUND")
end

VidFamne

Quotefor loops are quite a bit faster than while loops, since they have specialized virtual machine instructions
This information was written for Lua, pre v4.0 -- Nick Trout

NotRabidWombat

#30
Yes, but the algorithm itself will make the difference.

Linear searching will take O(n).
Binary searching will take O(lg(n)).

The speed increase in a for loop over a while loop is a constant, which is irrelevant to any time equation.

-NotRabidWombat


I like childish behavior. Maybe this post will be deleted next.

VidFamne

QuoteYes, but the algorithm itself will make the difference.
Yes, of course :)

And to chill; I did not  code it like this; i=1,getn(table)
I code it like this; i=low,getn(table)
And thats a hell of a difference

NotRabidWombat

"And to chill; I did not code it like this; i=1,getn(table)
I code it like this; i=low,getn(table)"

They are the same in this case.

for i = low, getn(tGeoIpRange) do

'i' is a number, and therefore all assignments are copies of value, not reference. So you intialize 'i' as low (or 1) and 'i' is incremented until the length of tGeoIpRange. If you change low or high in the for loop, 'i' will not change. So you are performing a linear search.

Makes you miss pointers, eh?

-NotRabidWombat


I like childish behavior. Maybe this post will be deleted next.

VidFamne

#33
Yepp, you are absolutly right.  :O
Thank you guys for pointing this out for me :)
I'm still learning alot.
Here then;
--By VidFamne (2004-04-07)
--A first try to do a GeoIp-script 
--With a very fast search algorithm on big tables :)
--Type +Country  and get the Country and CountryCode from an User
--The tGeoIpRange.txt is very big ( about 5.6 MB ) as I have ripped from GeoIPCountryCSV,
--with help from RabidWombat's serialisation function
--Hope its uptodate :)
--Change in the B.S.Algo :) (2004-04-21)




function Main()
	globals().tGeoIpRange={}
	globals().tGeoIpRangefile="tGeoIpRange.txt"
	dofile(tGeoIpRangefile)
end

function DataArrival(curUser, sData)

      sData=strsub(sData,1,-2) 
      s,e,cmd,arg = strfind( sData, "%b<>%s+(%S+)%s*(%S*)" )
		if (cmd=="+Country") then 
	   	local nick = GetItemByName(arg)
			if nick == nil then
			SendToNick(curUser.sName,"**** "..arg.." isn't in our records****") return 1 
			end
			local compIp = ComputeIP(nick.sIP)
				if compIp==2130706433 then 
				SendToNick(curUser.sName,"**** "..arg.." is sitting on localhost") return 1 
				else
				local C, CC = tSearchCountry(compIp)
				SendToNick(curUser.sName,"**** "..arg.." is from "..C.." and have CountryCode ["..CC.."]") 
				return 1 
				end
		end

end

function ComputeIP(IP)
	local _,_,a,b,c,d = strfind(IP, "(%d+)%.(%d+)%.(%d+)%.(%d+)")
	if a and b and c and d then
		return a*16777216 + b*65536 + c*256 + d
	end
end

----the Binary Search Algo----40

function tSearchCountry(num)
  local num = tonumber(num)
  local C = ""
  local CC = ""
  local low=1
  local mid = tonumber(mid)
  local high=getn(tGeoIpRange)

	while low <= high  do
		mid = floor((low+high)/2)
		if  tGeoIpRange[mid][1] <= num and num <= tGeoIpRange[mid][2]  then 
		C = tGeoIpRange[mid][4]
		CC = tGeoIpRange[mid][3] 
		break 
		end		  
			if tGeoIpRange[mid][1] > num then 
			high = mid-1
			else low = mid+1
			end 
	end   
	return C,CC
end

--------------------------VidFamne
**Edit; removed some testing code :D

VidFamne

A try to reduce the memory usage of the GeoIP-script.
Download >>here<< if you feel for it :)

chill

#35
done another version 2.5 MB mem, average search time around 0.7 sec.

would be faster if I could jump to certain bytes of a string, maybe someone knows how, but at least it does not read from disc each query.

hmm I think I'll try strsub, hope its fast on big strings we'll see.

<-GeoIP-> Searched for: 214.0.0.0 Used Time: 1.45 sec
	Range: 214.0.0.0-216.0.47.255, Country: United States, Countrycode: US

 <-GeoIP-> Searched for: 40.0.0.0 Used Time: 0.09 sec
	Range: 28.0.0.0-40.255.255.255, Country: United States, Countrycode: US

GeoIP by chill

VidFamne

#36
Nice work, chill :)
Think its always a choice between memory-usage and search-time.
My version use about 5.3MB mem and an average search-time at 0.1-0.2 sec. (on my my computer,Duron 1.2GHz, that is)
Hope I test it properly, fingers crossed ;D

chill

#37
great vidfamme, but forget the old one check out this one ;)

[14:46] <-GeoIP-> Serached for: 40.0.0.0 Used Time: 0.00 sec
	Range: 28.0.0.0-40.255.255.255, Country: United States, Countrycode: US
[14:47] <-GeoIP-> Searched for: "40.0.0.0" Used Time: 0.00 sec
	Range: 28.0.0.0-40.255.255.255, Country: United States, Countrycode: US
[14:47] <-GeoIP-> Searched for: "217.0.0.0" Used Time: 0.00 sec
	Range: 217.0.0.0-217.5.129.255, Country: Germany, Countrycode: DE
[14:47] <-GeoIP-> Searched for: "213.0.0.0" Used Time: 0.00 sec
	Range: 213.0.0.0-213.0.255.255, Country: Spain, Countrycode: ES

use 5 mb of memory , but only reads from disk at startup.

Geo IP V 002

chill

last version for today, but this one is already maximum optmised I think.
we'll see how many more to come

geo ip v 003

VidFamne

Really nice work chill ;)
Its going to be hard to beat this.
Next step would perhaps be a database in Binary format. If it would be do-able.

chill

thx vidfamne :)

yepp also thaught of extending it, as i like challenges
I'll see what I can come up with.

VidFamne

#41
I come up with this;
mem < 100 KiB , search-time < 0.1 sec.
precompiled database files,
some changes in code,
borrow some ideas from chill :)
Download from the  Script Archive hosted by Optimus, if you like. ;)
Please report if there is some bugs.

[PT]CableGuy

WOW !!!! Awesome !!!  :D
You've manage to accomplish only 100kb    of mem , when in the 1st post of this thread...it was 30Mb  . 8o
Great Stuff and excelent optimization !!! Keep up with you're excelent work. :]
I think this script will be "a key script" for Hub-Link Networks , in a near future. :P

I've found similar services but some of them have city also.
I mean , the script indicates the IP , Country and the City !!! :))
Is this very dificult ? Were can we get "those kind" of databases ? Am i dreaming loud ?:rolleyes:

nEgativE

Nice work !!! :)

chill

#44
yepp really great :), one big pro for compiling I'd say,  why not now compile the whole table and load it at once, then I guess your script will be unbeatable, but now I think I can forget about the string db, well not quite I'll still do some comparison with, only tbales containing string and see  what usues less mem.
Unfortunatly if you compile a string you don't have that memory effect :(.

cableguy is there a way to compile tables when you just built them? or will hub link only need, sone fixed up tables?

chill

#45
vidfamne, hehe, your code looks dunno... little irritating, I missed the table = nil and the collectgarbage(),

gotta take that pro away again for compiling.

I thaught compiling numbers would bring less memory,
but it doesn't matter, as I just found out, at least now the string db is not totally useless :))).

VidFamne

Hehe, Yeah I thought you missed that, but its the cause of
briging the mem down.
The precompiling, is only for the reading in,
in dofile()-part, so it would be a little, little faster.
Please keep up your good work on string db. :)

chill

Well you got your aim for sure, < 100kb, I guess you can't get below that, and yepp I just coded some morefor string db, take a look in the development section and tell me what you think if you got some time.

VidFamne

Thanks for the praise, [PT]CableGuy.  :))
About the GeoIP City, it wouldnt be any difficulties  to do that,
if we only could find any free database.
But until then its kinda hard.

[PT]CableGuy

QuoteOriginally posted by VidFamne
...But until then its kinda hard....
What i thought... :(
No problem...if i find one free , "ill be back". :D

QuoteOriginally posted by chill
...is there a way to compile tables when you just built them? or will hub link only need, sone fixed up tables?
I didn't understood you're question very wel....sorry.
The main problem is the database size , wich is 2Mb for countrys and 32Mb with citys also.
I think it's best , to use this script in each hub connected to the Hub-Link network.
Otherwise , like an "alternative" (extra) feature of Hub-Link....
...BUT , it all depends on "it's creater" , HaArD  !!!! ;)

SMF spam blocked by CleanTalk