c# - Unable to resolve DNS (sometimes?) -

c# - Unable to resolve DNS (sometimes?) -

given application in parallel requests 100 urls @ time 10000 urls, i'll receive next error 50-5000 of them:

the remote name cannot resolved 'www.url.com'

i understand error means dns server unable resolve url. however, each run, number of urls cannot resolved changes (ranging 50 5000).

am making many requests fast? , can that? - running same test on much more powerful server, shows 10 urls not resolved - sounds much more realistic.

the code parallel requesting:

var semp = new semaphoreslim(100); var uris = file.readalllines(@"c:\urls.txt").select(x => new uri(x)); foreach(var uri in uris) { task.run(async () => { await semp.waitasync(); var result = await web.trygetpage(uri); // using httpwebrequest semp.release(); }); }

it sound you're swamping local dns server (in jargon, local recursive dns resolver).

when programme issues dns resolution request, sends port 53 datagram local resolver. resolver responds either replying cache or recursively resending request other resolver that's been identified perchance having record you're looking for.

so, multithreaded programme causing lot of datagrams fly around. net protocol hosts , routers handle congestion , overload dropping datagram packets. it's handling traffic jam on bridge bulldozing cars off bridge. in overload situation, packets disappear.

so, it's endpoint software using datagram protocols seek 1 time again if packets lost. that's purpose of tcp, , that's how can provide illusion of error-free stream of info though can communicate datagrams.

so, programme need seek 1 time again when resolution failure on of dns requests. you're datagram endpoint own responsibility of retry. suspect .net library give failure when of requests time out because datagrams got dropped.

now, here's of import thing. responsibility of datagram endpoint program, yours, implement congestion control. tcp automatically using sliding window system, algorithm called slow-start / exponential backoff. if tcp didn't net routers congested time. algorithm dreamed van jacobson, , should go read it.

in meantime should implement simple form of in mass dns lookup program. here's how might that.

start batch size of, say, 5 lookups. every time whole batch successfully, increment batch size 1 next batch. slow-start. long you're not getting congestion, increment network load. every time failure resolve name, cut down size of next batch half. so, example, if batch size 30 , got failure, next batch size 15. exponential backoff. respond congestion dramatically reducing load you're putting on network. implement maximum batch size of 100 avoid beingness much of pig , looking rough denial-of-service attack dns system.

i had similar project while ago , strategy worked me.

c# windows multithreading dns