Weaponized GenAI: how cybercriminals are using GPT models to commit fraud, and what you can do about it

Cybercrime has reached a whole new dimension. Generative AI models stopped being just a tool of innovation and have become strong facilitators of deception and fraud. Think of phishing emails so spot-on that they are, for all intents and purposes, indistinguishable from personal writings, or video scams with deepfake technology bringing existing or fabricated identities to life. These threats just scratch the surface of what organizations must evolve to contend with. Those that don’t will be outmaneuvered by attackers who use AI to manipulate the vulnerabilities in human perception, digital infrastructure, and trust.

How weaponized GenAI works

Generative AI (GenAI) is a subset of artificial intelligence that creates new content—text, images, videos, and more—based on training data. Unlike traditional AI, which follows pre-programmed rules, GenAI can produce highly realistic and adaptable outputs, making it a powerful tool for legitimate innovation but also for malicious purposes. Cybercriminals exploit these capabilities to craft convincing scams, generate fake identities, and automate attacks that target human vulnerabilities at scale.

Attackers use GenAI in the following ways:

• Malware and exploit development: GenAI can help attackers write or improve malicious code, making malware more effective or harder to detect. It can also assist in creating polymorphic malware that changes its appearance to evade detection by antivirus software.
• Web application exploitation: GenAI can automate the process of finding vulnerabilities in web applications, such as SQL injection or cross-site scripting (XSS). It can then generate complex and tailored payloads to exploit specific vulnerabilities.
• Password and CAPTCHA bypass: AI models trained on leaked password datasets can predict likely passwords for specific targets. AI can also analyze and bypass CAPTCHA systems meant to differentiate bots from humans.
• Evasion techniques: AI can design payloads that evade intrusion detection and prevention systems (IDPS) and can automate scripts for botnets that rotate IPs dynamically to avoid detection.

The operational advantages of GenAI for cybercriminals are staggering and include:

• Scalability: AI can generate thousands of personalized phishing messages in seconds, adapting content dynamically based on recipient responses.
• Accessibility: Cybercriminals now have access to “Phishing-as-a-Service” (PhaaS) platforms that integrate AI tools, lowering the barrier to entry for less skilled actors.
• Believability: The precision of AI-generated content eliminates the grammatical and contextual errors that once gave away scams.
• Undetectability: AI models are even being trained to bypass CAPTCHA tests, simulate human interaction patterns, and evade detection mechanisms by constantly evolving their tactics.

3 major AI-driven cybercrime threats

Models such as ChatGPT have acted as force multipliers on cybercrime. Before GenAI was widely available, human limitations to scalability and precision included the need for labor, time, and technical specialism. But with GenAI, these factors are irrelevant. The result is a paradigm shift across a number of attack vectors, in particular phishing, deepfakes, and fake identities.

Personalized phishing campaigns

Social engineering attacks have become more precise because, with GenAI, it’s possible to craft messages that are practically indistinguishable from real communications. Attackers take advantage of publicly available information from sources like LinkedIn profiles, breached databases, and corporate press releases to create contextually correct, extremely convincing phishing emails.

Deepfake audio and video threats

The rise in popularity of valid accounts as an attack vector highlights the danger of manipulative AI technology such as deepfakes. Deepfake technology is a subset of generative AI that enables the creation of highly convincing audio and video clips of individuals, often targeting executives or public figures, to facilitate fraud, such as fund transfers or data theft. It has now reached alarming levels of sophistication.

In one notable case from early 2024, a finance employee at a multinational corporation transferred $25 million to scammers after deepfake technology was used to impersonate the company’s chief financial officer during a video call. This example illustrates the sophistication of such attacks and their potential to undermine even tightly controlled corporate processes.

Deepfakes can be deployed to discredit organizations, spread misinformation, or manipulate markets. Their ability to bypass traditional verification methods creates a serious challenge for existing cybersecurity frameworks.

Fake identities and synthetic content

Cybercriminals increasingly use AI to create synthetic identities, blending fake and real data to craft convincing personas with AI-generated photos, names, and backstories. These fake identities bypass verification systems, such as Know Your Customer (KYC) checks, to open fraudulent accounts, apply for loans, or steal benefits. Attackers also bolster their schemes with AI-generated documents, reviews, and testimonials, adding credibility to their scams and making detection exceedingly difficult.

Relatedly, GenAI enables the creation of realistic fake content at scale, from counterfeit IDs to glowing customer reviews. With these tools, criminals infiltrate online communities, build trust, and execute scams ranging from phishing campaigns to e-commerce fraud. These synthetic entities can impersonate real people, manipulate social proof, and evade standard detection methods, which are often not equipped to identify subtle AI-generated inconsistencies.

Countermeasures against AI-driven cybercrime

Countering AI threats effectively requires a multifaceted strategy that combines advanced technology, comprehensive training, and ongoing adaptability to address the risks posed by weaponized generative AI. This can be accomplished by improving employee training, strengthening identity verification, proactively using AI-powered cybersecurity solutions, and conducting continuous monitoring.

Improve employee training programs

Human error is still one of the leading causes of successful cyberattacks. Employees should be empowered to identify AI-powered scams, which mostly have more subtle signs of fraud. Areas of attention should include the following:

• Ways to spot phishing attempts that are grammatically perfect and contextually relevant
• Signs of deepfake audio or video, such as inconsistencies in visual fidelity or unnatural speech patterns
• Review of reporting mechanisms of suspicious activity for further investigation
• Simulated phishing tests to enhance employee preparedness by exposing them to increasingly sophisticated scenarios

Strengthen identity verification systems

Deepfake and synthetic identity attacks are advanced forms of cybercrime that exploit AI-generated content to deceive and manipulate. Deepfake attacks use AI to create highly realistic but fake videos, audio, or images that impersonate real individuals. For example, an attacker might generate a video of a CEO authorizing a fraudulent transaction, tricking employees or systems into compliance. Synthetic identity attacks involve creating entirely fake identities by combining real and fabricated information, such as blending stolen Social Security numbers with false names or addresses. These synthetic identities are then used to commit fraud, evade detection, or exploit systems.

To defend against these AI threats, organizations must adopt stronger identity verification protocols. Start with biometric authentication, such as facial recognition or fingerprint scanning, which verifies identity by matching unique physical traits. Enhance this with behavioral biometrics, which monitors patterns like typing speed, mouse movements, and device usage to detect anomalies. Together, these methods make it significantly harder for GenAI-powered attacks to succeed.

Leveraging AI in cybersecurity

Organizations can turn the tables on attackers by deploying AI-powered defense mechanisms. Read more about why AI-powered cybersecurity is the answer to AI-powered attacks in our dedicated blog post.

Some of the benefits of using AI in cybersecurity to counter weaponized GenAI are as follows:

• Real-time threat detection: Advanced machine learning models in network traffic and user behavior continuously analyze and identify deviations that may pass through the traditional monitoring systems. The models are good at finding minute deviations from normal patterns, thus enabling early detection of potential breaches.
• Email and content filtering: AI-powered systems scan the content of e-mails, syntax, semantics, and metadata for any phishing attempts or malicious payloads. The solution identifies the fraudulent element with accuracy and thus minimizes the chance of falling prey to deceiving communication.
• Automated incident response: AI-powered automation streamlines threat response time by taking direct action, enabling systems to isolate resources immediately or block malicious traffic. Containment speed reduces the breach impact and limits an attacker’s ability to escalate the attack.

When integrated into existing security infrastructures, these AI-driven solutions further enable organizational resilience and give organizations the tools to respond to evolving threats more quickly and efficiently.

Continuous monitoring

AI-driven cybercrime evolves at a pace that outstrips traditional security systems, rendering static defenses ineffective against these rapidly mutating attack vectors. To keep up, organizations must adopt dynamic strategies, including continuous monitoring across social media networks, app marketplaces, and other external digital platforms. These efforts aim to preempt threats, providing early warnings and intercepting potential attacks before they can escalate.

Advanced brand monitoring tools play a critical role by detecting fraudulent activities that misuse company names, logos, or domains. These tools quickly identify and flag phishing emails, counterfeit websites, or other impersonation attempts, enabling swift removal and minimizing risks to customers and brand reputation. In addition, threat intelligence platforms leverage data-driven insights to counter emerging attacks such as AI-generated deepfakes.

Prepare for the future with Gcore Edge Security

With the rapid development of generative AI technology, threats will keep changing. Organizations should be agile and invest in systems and processes that can keep pace with adversaries. Businesses can reduce risks and uphold trust with customers and partners by creating a culture of vigilance, integrating advanced technologies, and focusing on continuous improvement.

Our WAAP (web application and API protection) solution empowers organizations to stay ahead of growing AI challenges. With features specifically designed to find and neutralize AI-driven threats in real time, we give businesses the power to protect themselves and their reputation in a hostile digital landscape.

Explore our AI-enhanced WAAP solution