The Problem
I recently came across a post on Facebook where someone was asking for recommendations on apps or software to verify domains. They wanted a tool that could check for viruses, blank pages, broken links, and inactive domains. Essentially, they were looking for something similar to how email marketers use tools like NeverBounce to verify email lists and identify valid and invalid addresses.
This got me thinking about how I could solve this problem myself. I reached out to the person who made the post and offered to build a custom solution using Python and Google’s Web Risk API.
Use Cases
There are several use cases for a script like this:
- Website owners and administrators can use it to periodically check their own domains for potential threats or issues. This can help them identify and fix problems before they affect users or damage their reputation.
- Marketers and businesses can use the script to verify the safety and validity of domains before partnering with them or linking to them in their content. This can help prevent associating with malicious or spammy websites.
- Researchers and analysts can use the script to check large lists of domains as part of their investigations or studies. This can help them identify trends, patterns, or correlations related to web threats and security issues.
- Anyone who wants to check the safety of a website before visiting it can use the script to quickly verify if the domain is listed on any threat lists. This can help protect users from accidentally exposing themselves to malware, phishing, or other online dangers.
The Script
Here’s the script I came up with:
import csv
from google.cloud import webrisk_v1
from google.cloud.webrisk_v1 import SearchUrisResponse
def search_uri(uri: str, threat_type: webrisk_v1.ThreatType = webrisk_v1.ThreatType.MALWARE) -> SearchUrisResponse:
"""Checks whether a URI is on a given threatList.
Args:
uri: The URI to be checked for matches.
threat_type: The ThreatLists to search in. Multiple ThreatLists may be specified.
Returns:
SearchUrisResponse that contains a threat_type if the URI is present in the threatList.
"""
webrisk_client = webrisk_v1.WebRiskServiceClient()
request = webrisk_v1.SearchUrisRequest()
request.threat_types = [threat_type]
request.uri = uri
response = webrisk_client.search_uris(request)
return response
# Main function to read domains from CSV and write results to a new CSV
def main(input_csv, output_csv):
with open(input_csv, mode='r') as infile, open(output_csv, mode='w', newline='') as outfile:
reader = csv.reader(infile)
writer = csv.writer(outfile)
# Write header to output CSV
writer.writerow(['Domain', 'Threat'])
for row in reader:
domain = row[0]
result = search_uri(domain)
threat_types = result.threat.threat_types
threat_output = ', '.join(threat_types) if threat_types else 'Safe'
writer.writerow([domain, threat_output])
if __name__ == "__main__":
main('sample_domains.csv', 'output_results.csv')To use this script, you’ll need:
- A Google Cloud account with the Web Risk API enabled
- The
google-cloud-webrisklibrary installed (you can install it using pip) - A CSV file with a list of domains you want to check (one per line)
How It Works
The script reads the domains from the input CSV file, checks each one against Google’s Web Risk API, and then writes the results to a new CSV file. The output file will have two columns: “Domain” and “Threat”. If a domain is safe, the “Threat” column will say “Safe”. If a threat is found, the “Threat” column will list the type of threat (e.g., “MALWARE”).
The search_uri function is where the magic happens. It takes a URI (in this case, a domain) and a threat_type (which defaults to MALWARE). It then uses the Web Risk API to check if the domain is on a threat list. If it is, the function returns a SearchUrisResponse object containing the type of threat.
Conclusion
The script is pretty straightforward and gets the job done. It’s a lot faster than checking each domain manually, and it’s free for up to 100,000 requests per month. If you need to check more domains than that, you’ll need to pay for additional requests.
Overall, this is a handy script to have in your toolbox if you need to check a bunch of domains for potential threats. Just remember to keep your Google Cloud credentials secure and don’t share them with anyone.