Further info on the Mozilla politeness issue

[depappas]

Hi

Thank you for responding quickly to the issue I opened. See the application code below for loading a local version of the file.

Here is some information on the closed issue:

1. Dnspy uses the Mozilla ETLD url by default. Most users unless they open up the Dnspy source code will not know this. Mozilla asks that people do not download the file more than once per day. This is a politeness issue.

   DEFAULT_URL = 'http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_names.dat?raw=1'
   etlds = dict()

   def init (self, etld_url=DEFAULT_URL):
   """Read the ETLD list provided by the user."""

2. Dnspy runs a lot faster when the file is cached locally. Note the file is not loaded on a daily basis as recommended by Mozilla. The load time drops from 2-50 seconds down to...
   INFO: : GetRootUrl: Dnspy: download time=0.0158050060272

3. Here is the code that the application uses to load the file from a local repo where it was downloaded using wget.

def getScriptDir():
rootDir = os.path.dirname(os.path.realpath(file))
print "rootDir:%s" %(rootDir,)
return rootDir

def loadEtldFile():
print "INFO: loadEtldFile: Before"
scriptDir = getScriptDir()

etldFileName = scriptDir + "/etld_dir/effective_tld_names.dat"

defaultETLDList = None
if fileExists(etldFileName):
startTime = time.time()
etldUrl = "file://" + etldFileName
defaultETLDList = Dnspy(etldUrl)
endTime = time.time()
timeDiff = abs(endTime - startTime)
print "INFO: : GetRootUrl: Dnspy: download time=%s" % (timeDiff)
print "INFO: GetRootUrl: initialize Dnspy: After"
else:
print "ERROR: download_utils:downloader_utils: loadEtldFile: file does not exist: %s " % etldFileName
exit()
print "INFO: loadEtldFile: After"
return defaultETLDList