US $58.21 billion. That is the projected market value for open source intelligence (OSINT) products and services by 2033, according to Future Market Insights.
The market is being driven by various organizations that are investing in developing open source intelligence tools and technologies, offering OSINT services, training and educating OSINT professionals, and creating partnerships and collaborations.
Key players in the market are adding value by developing innovative tools such as data mining software and social media analytics platforms, all of which have extensive application in threat intelligence. They also provide expert analysis and interpretation of the data they collect and offer customized services to clients by monitoring analysis on specific topics.
These players are advancing the market through mergers, collaborations, partnerships, acquisitions, and product launches. What is open source intelligence? Why is that sector getting big bucks?
Understanding OSINT
The practice of gathering and analyzing information from publicly available sources, such as news articles, social media, government reports, and online resources, is known as open source intelligence (OSINT).
In recent years, OSINT has become increasingly important for law enforcement agencies, intelligence agencies, and private investigators as a means to gather information and stay ahead of potential threats.
Social media platforms like Twitter, Facebook, and Instagram are among the primary sources of OSINT.
These platforms offer a wealth of information that can be used to track individuals, monitor trends, and gather intelligence on specific topics.
OSINT analysts use tools like social media scraping software to collect large amounts of data and extract useful insights from them. Online forums and discussion boards are another valuable source of OSINT.
They can provide useful information on specific topics or industries and can be used to monitor
trends and identify potential threats. OSINT analysts use specialized search engines and software to monitor these forums and identify relevant discussions.
Government websites and databases are also significant sources of OSINT. Public records, government reports, and other publicly available data can be used to gather intelligence on individuals, companies, and other entities
News websites and blogs can also offer valuable insights into current events and trends. The effectivity of the data collected heavily depend on two things: the tools of collection and the methods of analysis. Here ten wildly effective tools for collecting and analysing open source intelligence.
1. Google (Yes, that counts)
If you know how to use advanced filters to refine your search, search engines such as Google, Bing, and
DuckDuckGo can serve as effective and free open-source intelligence (OSINT) tools.
Investigators have been able to increase the capacity of these tools by reverse-engineering search engines using a process called Google dorking or Google hacking, which includes using search operators or functions beyond Google.
However, the method of using search operators can be controversial as it may cross the line regarding how “public” the information is. For example, finding a link to a PDF file containing a list of passwords can be considered a prosecutable offense if downloaded.
One example of Google dorking is searching for PDF files on a company’s website domain. By typing “site:company. website.domain filetype:pdf” into the search bar, you can be surprised by the number of openly available documents that can be found using this method.
Although Google can be used as an OSINT tool and has powerful algorithms and a free price, it has limitations such as providing limited results and requiring a lot of trial-and-error. Additionally, the method can pose privacy issues and raise questions about the legality of obtaining certain documents.
2. AmIBreached
AmIBreached is an application designed to assist users in determining their exposure in the dark and deep web. The app provides a comprehensive approach to cybersecurity by allowing individuals to check if their personal information has been compromised or leaked in cyber breaches.
With the increasing number of data breaches and cyber threats, it has become crucial for individuals to take proactive measures to protect their personal information. AmIBreached addresses this concern by offering a user-friendly platform where users can easily assess their online security.
The app works by scanning various sources in the dark and deep web to identify any instances where a user’s personal information, such as email addresses, passwords, or financial details, may have been exposed. It also notifies users if their information appears in publicly available databases or has
been shared on underground forums.
The app does not collect or store any personal data from users, ensuring that their information remains confidential. Instead, it employs advanced algorithms and techniques to match user inputs with the available breach data without compromising privacy.
Upon completing the scan, AmIBreached provides users with a detailed report indicating whether their information has been compromised and the specific breaches associated with it.
3. Intelligence X
Intelligence X is an exceptional archival service and search engine that not only preserves past versions of web pages but also completes leaked data sets that could be removed from the web due to illegal content or legal concerns.
Although similar to the Wayback Machine, Intelligence X differentiates itself by focusing on preserving data sets, no matter how controversial, without any bias. In the past, Intelligence X has safeguarded a list of more than 49,000 Fortinet VPNs that were exposed to a Path Traversal vulnerability, as well as plaintext passwords that were revealed on hacker forums.
The service has also indexed data obtained from the email servers of influential political figures such as Hillary Clinton and Donald Trump, as well as media footage of the 2021 Capitol Hill riots and Facebook’s data leak of 533 million profiles.
For intelligence collectors, political analysts, news journalists, and security researchers, this information can be very valuable in various ways.
4. SEON
SEON is a top choice for verifying identities through social media and digital platforms. This method has become increasingly popular due to its high barrier of entry for fraudsters, ability to gather a user’s digital footprint, and potential to reveal information about someone’s socioeconomic background.
Although manual searches on LinkedIn, Facebook, or Twitter are possible, SEON is a specialist solution that
checks more than 50 social and online signals based on an email address, IP address, or phone number.
SEON’s fraud prevention tool offers real-time results and additional checks such as velocity, behavior, and device fingerprinting. It also provides data enrichment modules that give users a risk score and more information.
The service is flexible and can be queried manually, via API, or through a Google Chrome extension. While SEON lacks some of the more forensic elements of other entries on the list, it is customer-focused and provides valuable information for those who need to verify identities.
5. Lampyre
Lampyre is a paid application specifically designed for OSINT purposes, particularly for tasks like due diligence, cyber threat intelligence, crime analysis, and financial analytics. Users have the option to install it on their PC or run it online.
The standout feature of Lampyre is its one-click functionality. By inputting single data points such as a company registration number, full name, or phone number, Lampyre efficiently scans vast amounts of data to extract relevant and interesting information.
The company automatically processes data from over 100 regularly updated sources, which can be accessed through PC software or API calls if required. The SaaS (Software-as-aService) version is called Lighthouse, and users pay per API call.
However, it’s important to note that like other OSINT tools, users need to exercise due diligence and verify the openness of the databases utilized. While Lampyre automates searches, it is advisable to double-check the source of the information and the reliability of the provider, as one researcher discovered.
Lampyre offers several benefits, including its effectiveness in cybersecurity and due diligence tasks,
the ability to gather data from 100+ sources, and an affordable subscription or yearly purchase option. However, users may face a slight learning curve due to the less intuitive nature of Lampyre and its Lighthouse SaaS software.
6. Maltego
Maltego is an advanced Java application that aims to streamline and accelerate investigations through
its exceptional database access and visualization tools. Whether you work in trust and safety, law enforcement, or cybersecurity, Maltego offers oneclick investigations that provide easily understandable results.
Currently, Maltego allows users to visualize up to 1 million entities on a graph, with access to 58 data
sources. You can even integrate your own public databases and manually upload additional data sources.
Once all the relevant information is loaded into the program, you have the flexibility to choose from various visualization layouts, such as blocks, hierarchical, or circular, and utilize weights and notes to customize the graphs according to your preferences.
In addition to being a powerful tool, Maltego provides a curated collection of valuable resources on OSINT tools and techniques, enabling users to maximize their productivity. They also offer the option to purchase the Maltego Foundation course online. Maltego offers several advantages, including excellent graph visualization tools and multiple options for data visualization.
However, it should be noted that Maltego is a Java application only and has a slightly dated user interface.
7. Recon-ng
Recon-ng, originally developed as a free and open-source script for gathering technical information on website domains, has evolved into a comprehensive framework accessible through a commandline interface on Kali Linux or as a web application.
Its interface bears similarities to Metasploitable, another computer security project focused on penetration testing, sharing the common objective of assessing and identifying web vulnerabilities. Among its notable features are GeoIP lookup, DNS lookup, and port scanning.
Although Recon-ng is a more technically advanced tool on this list, there are ample online resources available to help users learn how to leverage its capabilities. With Reconng, investigators can locate
sensitive files like robots.txt, uncover hidden subdomains, identify SQL errors, and gather information about a company’s CMS or WHOIS data.
Recon-ng offers the advantages of being free and open-source while serving as a valuable tool in the field of cybersecurity. However, it is important to note that Recon-ng operates solely through a command-line interface, making it less suitable for investigators with limited technical expertise.
8. GHunt
GHunt is a specialized tool designed to assist users in extracting valuable information from Google accounts. It is primarily focused on gathering OSINT (Open Source Intelligence) from Google services
such as Gmail, Google Photos, and Google Drive.
GHunt offers a user-friendly interface and provides a range of features for conducting efficient investigations. Similar to Recon-ng, GHunt is a free and open-source tool. It enables users to perform various tasks, including checking if a target email address is registered with a Google account, detecting whether 2-step verification is enabled, and retrieving information about the target’s Google
contacts.
One of the standout features of GHunt is its ability to scan Google Drive for shared files. This feature allows users to search for files that have been shared with the target account, providing potential insights into the individual’s activities and connections. Additionally, GHunt can extract metadata from Google Photos, giving investigators access to valuable information associated with images uploaded to the account.
GHunt is a valuable asset for cybersecurity professionals, investigators, and researchers. It offers a straightforward approach to gathering OSINT from Google accounts, making it a useful tool in various
scenarios.
However, it’s important to note that GHunt is a command-line tool, meaning it requires some technical proficiency to utilize effectively.
9. Shodan
Shodan serves as a robust search engine designed specifically to explore and discover internet connected devices. Unlike traditional search engines that focus on indexing web pages, Shodan scans and catalogs information about a wide range of devices, including servers, routers, webcams, and more.
This unique specialization makes Shodan an invaluable tool for various purposes, such as cybersecurity, network monitoring, and research. The extensive capabilities of Shodan empower users to conduct targeted searches for specific devices or vulnerabilities using a variety of filters and operators.
Users can narrow down their search by factors like geographical location, IP address, organization, operating system, open ports, and even specific banners or keywords. By providing comprehensive details
about these devices, Shodan enables users to identify potential security weaknesses and evaluate the overall security status of networks.
One of Shodan’s standout features is its ability to uncover unprotected or inadequately secured devices that are accessible to the public. This includes devices with default or weak credentials, exposed administrative interfaces, or outdated software versions.
By pinpointing these vulnerable devices, users can proactively take measures to secure them or report the issues to the appropriate authorities. Shodan goes beyond search capabilities by offering additional functionalities.
Users can monitor specific devices or networks for changes, access historical data related to devices, and leverage APIs to integrate Shodan’s features into other applications or tools.
While Shodan provides valuable insights and is widely utilized by cybersecurity professionals, researchers, and enthusiasts, it is essential to exercise responsible and ethical use. As Shodan exposes information about potentially sensitive devices and systems, it is crucial to prioritize privacy and adhere to legal and ethical boundaries in its usage.
10. Social Mapper
Social Mapper is an open-source intelligence tool built in Python that utilizes facial recognition technology to correlate social media profiles. It is freely available on GitHub, allowing users to access and utilize it without any cost.
The tool gathers data from various popular social media platforms, including Facebook, Instagram, LinkedIn, Google Plus, Twitter, and Vkontakte. Social Mapper is particularly useful during the reconnaissance phase and can assist in conducting social engineering attacks targeting organizations or individuals.
Please ensure that Python is installed on your system as Social Mapper relies on this programming language. You can refer to the Python Installation Steps on Linux for guidance on installing Python.
It is worth noting that while OSINT can be a useful tool for gathering information, it has limitations. Information obtained through OSINT should always be verified through other sources before being used to make decisions or take action.
