Yeah its a good idea for a plugin, I already made some experiment earlier when i wanted to implement .bit support using http://www.opennicproject.org/
Need more experimenting, not sure if its possible using http://pydns.sourceforge.net/ & Tor network

Added pullrequest #111

The problem is it relies on dns servers, so the sites will not works offline. Cache required to make it work without internet connection example at dnschain plugin.
I don't really want to add more external dependencies, but pydns is a pure-python module, so distributing in the lib dir could work.

I think this will help a lot! And you can make the DNS support as a "fallback" in case other solutions are failing, to make it less centralised @HelloZeroNet and probably there is a solution to make it work offline as well.

The trouble with DNS is that it's not secure and is also quite centralized.

@HelloZeroNet Any progress here? Will classic DNS be supported?

Even if DNS isn't so decentralized, it would be good if it is supported.
You could use cache to avoid offline problems. And there are probably also solutions for other problems with it.

And for DNS servers, it should probably use OpenNIC by default (because it is "open and democratic alternative DNS root"), but it should be configurable by user.

IPFS also support similar thing.

You can add TXT record that points to IPFS file hash (dnslink=/ipns/<your-hash>). File is then accessible from ipfs-proxy.com/your-domain.com.

In addition, if you also create CNAME record that points to public IPFS proxy (217.182.195.23 for gateway.ipfs.io), file is also accessible directly from your-domain.com).

This would also be extremely useful for ZeroNet. Site owner would then only need to create TXT record with ZeroNet address and CNAME record that points to (trusted) ZeroNet public proxy or local ZeroNet instance and site would be accessible directly from classic domain.

Note that implementing TXT query only on the main domain is not a very good idea since that prevents using CNAME records.

IPFS introduces _dnslink prefix and queries TXT record like this:
get TXT of _dnslink.example.com
if that failed, get TXT of example.com

That way you can setup CNAME on example.com and additional TXT record on _dnslink.example.com.

@ValdikSS Yes, that is how I want to be implemented. With this, it should be also possible to handle subdomains (_dnslink.subdomain.example.com), right?

@filips123 Yes, should be no problems with subdomains.

@ValdikSS It appears that DNSLink, which is used by IPFS, is already some kind of standard. It also supports linking other types different from IPFS.

So, instead of reinventing the wheel with a custom type of DNS record, ZeroNet should just use DNSLink. ZeroNet records should use the same names and structure as IPFS's, just record's content should be dnslink=/zeronet/1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D.

I created a Python library for resolving DNSLink records, DNSLinkPy. I will also soon create a plugin for DNS support in ZeroNet.

@filips123 Awesome! Please also add reverse-proxying feature to the plugin, i.e. the website should not only be accessible using 127.0.0.1:43110/example.com, but also when the ZeroNet port receives HTTP request with Host: example.com.

but also when the ZeroNet port receives HTTP request with Host: example.com

This will not be in my initial changes. However, I will see if I (or someone else) can implement this a bit later.

@ValdikSS Wait, would that mean that we could add A record pointing to 127.43.11.0 (for example) and visiting example.com would be smoothly handled by ZeroNet? Yayyyy!

@imachug, yes, I'm thinking about public ZeroNet gateways on the internet, but it could also be used for transparent access in the local network.

Say, you want to create a blog, and you have a domain name. You clone ZeroBlog, setup CNAME to public gateway domain name and dnslink to your blog addresss and — boom — you have a blog, reachable from the internet, set up in 5 minutes.

@ValdikSS Yes, this is similar to what IPFS also supports. So you create ZeroNet sure, create DNSLink TXT record with site address, point CNAME or A record to public instance and site is reachable just with domain.

@imachug Won't be a problem that local ZeroNet instance is on port 43110 but domain should be reachable on port 80? With public instances this isn't a problem because most public instances are on port 80.

Or there would be even better solution, if it is possible to do. This would be to first try to resolve domain on local instance and fallback to public if needed. But I'm not sure if this is possible with CNAME or A records.

@filips123 The idea is to switch from 127.0.0.1:43110 to 127.43.11.0:80.

Good news! I almost finished the DNS plugin. I just need to test it a bit more and I will probably create the next week.

The plugin uses various OpenNIC nameservers to resolve the domain. It uses DNSLink to get site address. It then also stores address in a local cache file to support quicker resolving and offline usage.

However, there is something that might need to be discussed. DNS queries won't go over Tor, right? Could this be a privacy problem? And how it could be fixed (I'm using DNSPython library)?

You can perform DNS queries over TCP, over Tor.
You can also use DNS over HTTPS or DNS over TLS.

@ValdikSS How can this be done in DNSPYthon? Also, it should probably be enabled only in Tor Always Mode.

I'm afraid DNSPython might not support that. I'd recommend just forking it.

http://www.dnspython.org/docs/1.16.0/dns.query-module.html#tcp
This?

Most likely. It'd be better if we could just override socket.socket with a Tor-compatible wrapper, though.

@ValdikSS I'm using dns.resolver.query(domain, 'TXT') for getting TXT records. However, I can also change the settings of the resolver object.

I updated resolver to always use TCP if Tor is enabled, but there is still some problem.

When a resolver tries to create a query, it will throw this exception:

Traceback (most recent call last):
  File "D:\Users\filips\Downloads\ZeroNet\venv\lib\site-packages\dns\resolver.py", line 910, in query
    source_port=source_port)
  File "D:\Users\filips\Downloads\ZeroNet\venv\lib\site-packages\dns\query.py", line 491, in tcp
    q.keyring, q.mac, ignore_trailing)
  File "D:\Users\filips\Downloads\ZeroNet\venv\lib\site-packages\dns\query.py", line 419, in receive_tcp
    ldata = _net_read(sock, 2, expiration)
  File "D:\Users\filips\Downloads\ZeroNet\venv\lib\site-packages\dns\query.py", line 347, in _net_read
    n = sock.recv(count)
  File "D:\Users\filips\Downloads\ZeroNet\venv\lib\site-packages\socks.py", line 410, in recv
    bytes, _ = self.recvfrom(*pos, **kw)
  File "D:\Users\filips\Downloads\ZeroNet\venv\lib\site-packages\socks.py", line 391, in recvfrom
    return super(socksocket, self).recvfrom(bufsize, flags)
  File "D:\Users\filips\Downloads\ZeroNet\venv\lib\site-packages\gevent\_socket3.py", line 411, in recvfrom
    return _socket.socket.recvfrom(self._sock, *args)
BlockingIOError: [WinError 10035] A non-blocking socket operation could not be completed immediately

(It will actually handle it but it is the same for all nameservers which means that domain will load infinitely. I modified DNSPython to also print trace of it.)

Those are my observations:

• When Tor is disabled, everything works correctly. DNS uses UDP query.

• When Tor is enabled (always mode), the query will fail because UDP is not supported in Tor. This is expected and I already fixed this.

• However, if I force a query to be TCP (and Tor is enabled), it will also fail because of the above error. This is not expected and should not happen.

• Also, when I force TCP but Tor is disabled, it still works.

Here is my current version of the plugin for testing. You need to install dependencies from requirements.txt.

Domains that are available for testing are js.zeroframe.oss and py.zeroframe.oss which should redirect to my Git Center repositories (and it works without Tor).

Just note that cache is saved into memory and data/dns_cache.json. This means that you should delete that file (and restart ZeroNet) after every test, else it might look like that DNS works but domain will be only loaded from cache.

@ValdikSS @imachug Any thoughts on this?

The problem is in non-blocking socket code, but I'm confused by the fact that it occurs in gevent.
My assumption is that dnspython switches socket into non-blocking mode, which gevent can't handle?

Non-blocking sockets should be handled in a slightly different way, using selectors. ZeroNet uses gevent, a "green threads" library which abstracts threads allowing software to be written in an old, blocking fashion, but in fact it doesn't block as a regular non-gevent software.

If it wasn't in gevent, it would be obvious that you need to use selectors on the socket or not switch it into non-blocking socket, but since the error is inside gevent, I'm not sure what should be done to fix it without checking your code first, or at least a short example which I can debug.

Pysocks is not compatible fully with gevent when udp sockets are used. (I reported it last month: Anorov/PySocks#129)

The recvfrom function is used for UDP sockets, so if you get error related to that, then it means it tries to use UDP sockets.

The recvfrom function is used for UDP sockets, so if you get error related to that, then it means it tries to use UDP sockets.

recvfrom could also be used for TCP, which is probably the case here.

@ValdikSS

I'm not sure what should be done to fix it without checking your code first, or at least a short example which I can debug.

Here are some reproduction steps:

1. Plugin source code is here.

2. You need to add a plugin to ZeroNet and install all dependencies from requirements.txt.

3. Modify dns/resolver.py (around line 927) of DNSResolver package and add this to show error when it occours:

except (socket.error, dns.exception.Timeout) as ex:
    #
    # Communication failure or timeout.  Go to the
    # next server
    #

    import traceback      # Add this two lines
    traceback.print_exc() # To show exception

    errors.append((nameserver, tcp_attempt, port, ex,
                   response))
    response = None
    continue

4. Enable Tor Always Mode and restart ZeroNet (start with --debug option to see live logs). Delete dns_cache.jsonfile from ZeroNet data folder (if exists) to remove cache.

5. Try to open domain py.zeroframe.oss which should open Git Center repository (but it probably won't).

6. You will see a lot of errors in log output (for each of availible DNS servers). Some of errors are related to IPv6 (which appears to not be supported by PySocks. But most of the errors will be related to problem from my previous comment. Also, website will load for infinity.

@HelloZeroNet

Pysocks is not compatible fully with gevent when udp sockets are used.

Yes, I just saw your issue few minutes before you linked it :)

The recvfrom function is used for UDP sockets, so if you get error related to that, then it means it tries to use UDP sockets.

Where could this be called? UDP should be disabled.

Can you check my plugin?

I checked the source code of dnspython and ValdikSS is right, it's using recvfrom with TCP sockets (which is a bit weird)

I think the problem is not with your plugin, but pysocks + gevent + dnspython is not fully compatible.

I recommend to create a simple test script to check if it works without gevent.monkey.patch_all()

And try alternative lib eg.: https://bitbucket.org/paulc/dnslib/src/default/

I recommend to create a simple test script to check if it works without gevent.monkey.patch_all()

@HelloZeroNet How to do this?

Also, it looks like that dnslib is only a package for encoding/decoding DNS packets, but not actually sending them to real nameservers.

It does support sending the request to the server. It also has a simple CLI tool to do that:

py -3 -m dnslib.client --help
usage: client.py [-h] [--server <address:port>] [--query] [--hex] [--tcp]
                 [--noretry] [--diff DIFF] [--dig] [--short] [--dnssec]
                 [--debug]
                 <domain> [<type>]

I think the problem is not with your plugin, but pysocks + gevent + dnspython is not fully compatible.

So, is it possible to somewhere report an issue with those libraries?

I recommend to create a simple test script to check if it works without gevent.monkey.patch_all()

How to do this? Do you mean to create a simple script that gets DNS record using my library, but check it with and without gevent.monkey.patch_all()?

If it would be hard to fix this for now, could we just disable DNS resolving when Tor is used (more private but less useful) or always use clearnet for resolving (less private but more useful)?

Disabling DNS in some cases should not be considered at all. I can't check the code right now unfortunately but I'll try to fix it within a few hours.

I recommend to use dnslib anyway as it's also more lightweight and since it's more like a parsing library probably it's easy to use our own sockets with it.

Is that a windows-only issue? Can't reproduce it on Linux.

Looks so (I actually had a similar problem with ICMP requests on Windows recently: you can't capture ICMP packets unless you are NT AUTHORITY\SYSTEM).

That's unrelated, but you can't capture it on Linux either if you unprivileged.

Yes, it may only cause problems on Windows. I only tested this on it.

If you want to use third-party external resolvers by default in the first place (why by the way?), maybe it's better to use DoH then?

External DNS servers are used because they are for OpenNIC which is "open and democratic alternative DNS root". This could make DNS at least a bit more decentralized.

It is also possible to use system configuration. But in most cases, default system DNS servers will be from ISP or Google, which isn't so decentralized and also doesn't support special free OpenNIc domains. That's why it is not used by default.

Here in Russia many ISPs perform DNS transparent proxying to their own DNS resolvers for censorship reasons. Sending DNS request to any IP address, like one which does not run DNS recursor or even to unroutable IP address results in a DNS response.

I mean, it's not sane to use third-party unencrypted DNS resolver by default, IMO. It either should be system default resolver or DoH/DoT.

I'd say:

• Use system default resolver in No Tor/Tor Available mode by default
• Use DoH in Tor Always mode, always

What do you think?

Also to note, dnspython sometimes can't detect system default resolver and does not work properly. I'm not sure which platforms are affected (Windows I suppose), but in my blockcheck software I see reports of broken system resolver once in a while.

@ValdikSS Using system default resolver would remove the ability to use OpenNIC-specific features, like additional free TLDs, Emercoin TLDs...

I would more like that DoH/DoT resolvers will be always enabled, regardless of what Tor is, except if the user specifically doesn't want this. However, DNSPython currently doesn't support DoH and DoT (see rthalley/dnspython#358 for details).

DoH protocol is very simple, it can be used without any external library:

curl -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=example.com&type=TXT'
{"Status": 0,"TC": false,"RD": true, "RA": true, "AD": true,"CD": false,"Question":[{"name": "example.com.", "type": 16}],"Answer":[{"name": "example.com.", "type": 16, "TTL": 4143, "data": "\"v=spf1 -all\""}]}

@HelloZeroNet Great! All DoH resolvers use same request and response format in JSON, right?

Because DNSPython also supports some other features (like cache, response handling...), I will try to create PR for it with DoH support. This means that DNSPython (when using DoH nameserver) could completely bypass current TCP/UDP socket requests and just use urllib or similar. This should also fix issue with Gevent and non-blocking sockets.

It would also be good to support DNS stamps. I found Python library for that, but I don't know what exactly all data in stamp mean and how to use it.

Also, because I will use urllib for making HTTP requests, I also need to know if it will be automatically monkey-patched in ZeroNet for Tor, or I will need to do this manually.

The API should be the same for every provider, the http lib is already patched, so you don't have to do anything just use urllib.request

I almost implemented DoH support for DNSPython. It uses basic urllib.request so there should hopefully be no problems with Gevent. I will try to test it tomorrow and then create PR for DNSPython next week. Then DNS plugin for ZeroNet would just need to wait until PR is merged or use my temporary branch for DoH support.

@HelloZeroNet @imachug @ValdikSS I created #2214.

I would highly recommend placing the TXT records at a designated subdomain (perhaps _zero), i.e. a TXT record at _zero.sub.example.org would correspond to a user-visible address of sub.example.org. This makes it more feasible to control permissions for ZeroNet separately from IP address records; it's what TLSA records do for this reason.

Also, Namecoin supports TXT records, so it would be theoretically possible to unify the Namecoin and DNS support in ZeroNet, which would have benefits. Namecoin's onion service support takes this approach (.onion domains are specified in a TXT record, so Namecoin and DNS use the same spec.)

My PR already supports _dnslink subdomain which is is the same thing as used in IPFS-based websites and other decentralized systems.