websecmap/api/templates/api/SIDN.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" type="text/css" id="active_theme" href="/static/css/vendor/bootstrap.min.css"/>
</head>
<body>
<div class="container">
<h1>Adding subdomains with the SIDN API</h1>
<p>
This guide was written while keeping in mind that adding subdomains can be done manually, but also can be
automated using the Python Requests library (for example).
</p>
<h2>Step 0: Authentication</h2>
<p>Authentication is done via HTTP Basic Auth. To prepare a logged in session object to be used in further calls use following code.</p>
<h4>Code example</h4>
<code>
import requests<br>
<br>
your_username = "admin"<br>
your_password = "faalkaart"<br>
your_api_endpoint = "http://127.0.0.1:8000"<br>
<br>
client = requests.session()<br>
client.auth = (your_username, your_password)
<br>
</code>
<h2>Step 1: retrieving layers</h2>
<p>
WebSecMap categorizes data in countries, layers and urls. While not strictly needed to add subdomains,
it's helpful to use the first two calls to identify which 2nd level domains already exist in what layer.
</p>
<p>To retrieve all layers, call:</p>
<pre>/api/SIDN/layers/</pre> (<a href="./layers/">direct link</a>)
<p>A response would look like this:</p>
<pre>[{"country": "NL", "layer": "municipality"}]</pre>
<h4>Code example</h4>
<code>
r = client.get(f"{your_api_endpoint}/api/SIDN/layers/")<br>
layers = r.json()<br>
print(layers)<br>
</code>
<h2>Step 2: Retrieving second level domains</h2>
<p>To retrieve a list of all second level domains in a certain layer, you can perform the following call.
Note that you need to use the data from the first call.</p>
<pre>/api/SIDN/2nd_level_urls_on_map/[country code]/[layer name]/</pre>(<a href="./2nd_level_urls_on_map/[country code]/[layer name]/">direct link</a>)
<p>For example:</p>
<pre>/api/SIDN/2nd_level_urls_on_map/NL/municipality/</pre>(<a href="./2nd_level_urls_on_map/NL/municipality/">direct link</a>)
<p>A response can be:</p>
<pre>["alphen-chaam.nl", "almere.nl", "schermer.nl", "ameland.nl", "aalten.nl", "aalsmeer.nl", "aaenhunze.nl", "apeldoorn.nl", ...</pre>
<h4>Code example</h4>
<code>
r = client.get(f"{your_api_endpoint}/api/SIDN/2nd_level_urls_on_map/NL/municipality/")<br>
domains = r.json()<br>
print(domains[0:10])<br>
</code>
<h2>Step 3: Adding subdomains</h2>
<p>To add subdomains, make a post call with CSV data to the following location:</p>
<pre>/api/SIDN/upload/</pre>(<a href="./upload/">direct link</a>)
<p>The data submitted needs to have a very specific format, which is:</p>
<pre>,2ndlevel,qname,distinct_asns
123,arnhem.nl.,*.arnhem.nl.,1
124,arnhem.nl.,01.arnhem.nl.,1
163,arnhem.nl.,sdfg.arnhem.nl.,1
125,arnhem.nl.,03.arnhem.nl.,1</pre>
<p>The answer to your call will be <code>{"result": "processing"}</code>.</p>
<p>Only the second and third values are used, the rest is ignored. After submitting each domain will be validated
and tested if the domain resolves. Only if all checks are passed, the domain is added to the database. All
organization that share the associated second level domain will also get the new subdomain in their portfolio.</p>
<p>It can take a short while before an answer is given. If the service times out, try smaller batches as each
domain needs to be contacted to verify its resolvability.</p>
<p>An example requets looks like this, note the urlencoding of the data, which requests can do for you(!)</p>
<h4>Code example</h4>
<code>
# Do the actual post<br>
csv_information = """,2ndlevel,qname,distinct_asns<br>
123,arnhem.nl.,*.arnhem.nl.,1<br>
124,arnhem.nl.,01.arnhem.nl.,1<br>
163,arnhem.nl.,sdfg.arnhem.nl.,1<br>
125,arnhem.nl.,03.arnhem.nl.,1"""<br>
r = client.post(f"{your_api_endpoint}/api/SIDN/upload/", data={"data": csv_information})<br>
print(r.json())<br>
</code>
<h4>Web form example:</h4>
<p>The form below can be used to easily create this post request. Sample data is already provided:</p>
<form method="POST" action="/api/SIDN/upload/">
<textarea name="data" style="width: 100%; height: 300px; font-family:courier">,2ndlevel,qname,distinct_asns
123,arnhem.nl.,*.arnhem.nl.,1
124,arnhem.nl.,01.arnhem.nl.,1
163,arnhem.nl.,sdfg.arnhem.nl.,1
125,arnhem.nl.,03.arnhem.nl.,1</textarea>
<input type="submit" value="Try this">
</form>
<h2>Step 4: View the status of your uploads</h2>
<p>You can view the status of your last 500 uploads here:</p>
<pre>/api/SIDN/uploads/</pre>(<a href="./uploads/">direct link</a>)<br>
<p>Using the requests library, it would read something like this:</p>
<code>
r = client.get(f"{your_api_endpoint}/api/SIDN/uploads/")<br>
print(r.json())<br>
</code>
<p>To view only uploads that added urls to the database, perform this request. This will show all
uploads with new urls.</p>
</div>
</body>
</html>