In modern OSINT investigations, OSINT deep web research is indispensable. Whilst most investigators start their searches on the surface - also known as the ‘clear web’ - the deepest open-source intelligence likes on the deepest level. Beneath search engine indexes and the surface web’s soothing sheen, you’ll find the vast recesses of the deep web, and its even more concealed counterpart: the dark web.
A deep web OSINT investigator has sharp skills and watertight OpSec, allowing them to glide from shallower surface-level insights into the murky waters of deep web intelligence. This guide will show you how to dive deep - safely, ethically and effectively. We’ll take you through all the hidden layers of the internet; how they differ, why they matter for OSINT, and which tips and techniques will uncover the truth. Let’s dive into OSINT on the deep web.
What is the Deep Web?
The deep web is all the parts of the net that aren’t indexed by traditional search engines. These parts won’t show up in Google, Bing, or any other crawler-based results; either because they’re intentionally, or technically excluded from indexing. They may include:
- Private databases
- Subscription content
- Login-protected portals
- Court records
- Grey literature
The word ‘private’ is usually problematic for OSINT investigators. After all, OSINT is the art of analysing data that’s openly and publicly available on the web. However, not all deep web content is inherently secret, or illegal to access; much of it can be lawfully and ethically accessed if you know how to do so. The only thing making it ‘deep web OSINT’ (as opposed to vanilla OSINT) is that it’s outside the reach of standard indexing, and therefore needs a specialist approach.
So, deep web OSINT is OSINT that seeks to surface publicly available - but not publicly indexed - content. It may be hidden behind a database interface, a paywall, or a simple screen, but that doesn’t mean it isn’t fair game for intrepid OSINT investigators.
Want to go even deeper? Check out our guide to OSINT Basics: What is Dark Web Intelligence (DARKInt)
Deep Web OSINT vs. Dark Web OSINT
Many investigators make the rookie mistake of treating dark web OSINT and deep web OSINT as interchangeable. However, although related, these two deep dark intelligence disciplines have multiple distinct differences.
- Deep Web: The larger portion of the hidden internet. Not indexed by search engines, this web hosts largely benign content that can be legally accessed with the right tools; anything from subscription-only publications to internal web apps.
- Dark Web: A subset of the deep web that’s accessible only via specialised software such as Tor, I2P or other anonymising networks. Web addresses in this part of the net often use unique domain formats like .onion. Open-source intelligence found on the dark web is known as dark intelligence, or DARKINT.
All dark web content is on the deep web, but not all deep web content is on the dark web. By nature, dark web content is unindexed; making it technically part of the deep web. However, what makes content DARKINT is the fact it’s only accessible specifically through anonymising networks.
Deep Web OSINT: Techniques and Best Practices
Just like its darker cousin, deep web intelligence can be difficult. Conducting your investigations effectively, ethically and safely means mastering a whole host of techniques and best practices. Here are our key steps to optimise your deep web OSINT workflow.
- Use the Right Tools
You can’t explore the deep sea with a snorkel. If a Tor browser is a submersible, you need something akin to a full-capacity diving tank; normal search engines won’t cut it, but there’s no need to go full onion. As surface search engines are usually inadequate for deep web OSINT, try:
- Database search interfaces for academic or legal content
- API access to specialised data repositories
- OSINT platforms that surface hidden sources (e.g. OSINT Industries)
Have you mastered surface search engines yet? Sharpen up your skills with our guide to OSINT Basics: Going Beyond Google with Bing and Yahoo Dorking
- Maintain Operational Security
As you move beyond the surface web, OpSec gets more important than ever. It’s vital that you preserve all your personal information, preserve the integrity of your investigation, and stop anybody from being able to track you from point-to-point before your work is complete. To keep yourself, your investigation, and your subjects safe:
- Use VPNs and segregated environments when accessing the deep web
- Avoid using personal accounts or identifiable credentials
- Use alternative devices just for your investigations
- Disable scripts and plugins in your browsers
For more safety tips and tricks, check out our guide to Wiping the Prints: Managing your OSINT Digital Footprint with OpSec
3. Validate and Corroborate
Deep web OSINT data - especially user-generated content (UGC) - can be dodgy, so everything you uncover needs to be carefully corroborated. One of the best things about the deep web is its ability to back up existing findings with a wealth of new content; but it’s important to check that content is kosher before you include it in your investigative work. Always:
- Cross-reference with other reliable sources
- Analyse metadata and communication patterns
- Check for consistency across datasets
Just because the deep web is unindexed, doesn’t mean it’s immune to surface web shenanigans like botting, misinformation and manipulation. Analytical rigor should always be on your mind.
4. Respect the Legal and Ethical Boundaries
This is the most important part of deep web OSINT. Always remember: some content is unindexed because it’s private. Although it’s technically possible to snoop around in password-protected content on the deep web without permission, accessing restricted areas crosses a clear ethical - not to mention legal - line.
If it’s not openly available data, it’s not OSINT; so always make sure you’re compliant with local privacy laws. The easiest way to ensure compliance is to work with an OSINT tool. OSINT Industries’ platform, for example, has legal and ethical compliance built-in as standard. So if you’re working with that, you can be sure you’re working safely.
In turn, deep web OSINT should NEVER involve participation in illegal or illicit activities, even beyond issues of privacy. Careful adherence to law and ethics ensures investigations remain defensible and professional.
Why Deep Web OSINT Matters
For all OSINT investigators - from cybersecurity teams to law enforcement, private detectives to due-diligence departments - it’s vital that you go as deep as possible. Surface-only intelligence is powerful enough, but you can only see everything with deep web OSINT skills.
So far, humans have only explored about 5% of the ocean. The rest is deep sea; and similarly, the deep web is estimated to be many, many, many… many times larger than the surface web. As the experts’ best estimate, search engines only index about 1% of the content that exists online. So, if you’re not diving into the deep web, you’re still paddling in the shallow end. There’s a whole reservoir of open-source intelligence out there - if you know how to swim.
Want to see deep web OSINT in action? Read our real-life Case Study:
“So far, [Aaron’s] process had only been implemented to catch clearweb predators: the tip of the iceberg. To truly profile any pedophile, Aaron set out to crack the darkweb.”
Read more: Know Your Enemy: How OSINT Collaboration Can Profile a Predator
