Artificial Intelligence (AI) is transforming the world around us at an unbelievable pace. Writing documents and code, researching idea and even starting to be involved in critical decision-makings.
However like any new technology, as AI adoption accelerates, businesses and organisations are starting to face new and growing security challenges which for the most part the industry has never seen before. This article explores some of the key security threats starting to be posed by AI in 2025 and practical strategies to mitigate them.
1. Data Privacy & Security Risks
AI systems rely on extensively on data inputs which get transmitted to a third-party such as ChatGPT and DeepSeek, either via their respective web interfaces by an employee or automatically through an API.
This represents some key concerns.
Where is the data being transmitted to?
In most cases when using public models like ChatGPT, every time an AI interaction occurs, data is being transmitted and processed on remote servers outside of the organisations control. The location of the servers can vary depending on a number of factors and it is rare for a provider to publicly disclose or set the specific server locations for each user request.
This can be challenging for organizations, as they must navigate a mix of internal procedures and external regulations, such as Data Residency Laws and Cross-Border Data Transfer rules like the GDPR. Industries like healthcare, with regulations like HIPAA, and sectors such as finance and defense, which are subject to export control laws, face even more hurdles when it comes to ensuring compliance and protecting sensitive data.
To Who Is the Data Being Sent?
Unpacking the can of worms even further on the issue, when using third-party AI services, the question of who has access to your data becomes critical. Often, organisations rely on external providers to handle their AI processing, but the trustworthiness and security of these third parties can be a significant concern.
It’s crucial to assess whether these providers have the necessary measures in place to protect sensitive data. Even more concerning is the possibility that third-party AI providers could be influenced or compromised by local intelligence agencies, such as the Chinese Communist Party (CCP) and its associated surveillance agencies, in the case of DeepSeek. Assuming ‘Western’ providers like Chat GPT are any better may be false comfort, as intelligence agencies are known to operate mass surveillance programs like PRISM and Tempora, which allows governments to spy on corporations and use that information for economic gain—whether by gaining leverage in trade negotiations or undermining foreign businesses.
What Data Is Being Sent?
When you interact with AI systems, it’s not just useless questions or prompts you’re sending—often, sensitive data like personal information, corporate secrets, or even credentials can be involved. For example, details like names, contact info, or other identifiers might be transmitted during an AI session, which raises questions about how securely that data is handled. Then there’s the risk of business-critical information, like intellectual property or trade secrets, being shared without realizing it. A notable example is the Samsung leak, where sensitive data, including private meeting details and code, were uploaded to an AI platform raising serious concerns about how AI services handle corporate data.
Even more concerning, sensitive items like passwords or API keys can be exposed. In fact, a recent discovery found nearly 12,000 API keys and passwords in training datasets, potentially putting users and companies at serious risk. Once this data is sent to an AI provider, it might be stored for future use, often to improve the model or its responses and shared to other users.
The Solution: Training, Policies, Privately Hosted Models?
There is no easy fix to the ‘Data Privacy and Security Risks’ of using AI tools, addressing the risks requires a structured approach, combining staff training, strict policies, and the consideration of privately hosted models to ensure data security and compliance.
Staff Training on Data Security Risks
Employees are often the first line of defense in protecting sensitive data. Without proper training, staff may inadvertently share confidential information with AI tools, thinking they are simply using them for productivity-enhancing tasks. This can include summarizing meeting notes, generating reports, or drafting emails—all of which could expose sensitive corporate data.
To mitigate these risks, organizations should implement a structured training program that covers:
- What constitutes sensitive data – Employees must understand what information is considered confidential, including personal data, corporate secrets, financial information, and proprietary research.
- Risks of using third-party AI tools – AI platforms may retain user inputs, store them for future model training, or even share them with external parties. Employees should be aware that any data shared with an AI could be accessed beyond their control or knowledge.
- Best practices for responsible AI use – Training should emphasize that AI tools should not be used for handling confidential information unless explicitly authorized. If AI assistance is necessary, employees should ensure they are using organisation-approved platforms that meet security standards.
- Simulated phishing and AI misuse scenarios – Organisations can conduct awareness programs where employees are tested on their responses to potential AI-related data leaks. This can help reinforce best practices and build a strong security culture.
Defining and Enforcing Internal Policies
Organisations must establish clear, enforceable policies on AI usage that align with regulatory requirements and security best practices. These policies should be developed in collaboration with IT and legal teams to ensure they cover key areas such as data residency, compliance, and vendor security.
Key policy considerations include:
- Evaluating AI platforms for security and compliance
- Before adopting any AI tool, organisations should conduct thorough assessments to determine if the platform the tool uses meets legal and regulatory requirements (e.g., GDPR, HIPAA, SOC 2 compliance).
- The evaluation should include an analysis of data handling practices, storage policies, and access control mechanisms.
- Blocking unauthorized AI tools
- Any AI tool that does not meet the organization’s security and compliance standards should be explicitly prohibited.
- IT teams should implement technical controls, such as network restrictions, to prevent employees from using unauthorized AI platforms.
- Role-based access and usage controls
- Organisations should establish guidelines on which employees or departments are authorized to use AI tools and under what conditions, with a clear process to onboard new tools.
- Access controls should be in place to ensure that only approved users can process sensitive data through AI-assisted workflows.
- Ongoing policy reviews and updates
- AI technology and regulatory landscapes are constantly evolving with it becoming an inevitably that it will become more and more a way of life. Organisations should periodically review and update AI usage policies to reflect emerging risks, compliance changes and industry advice.
Exploring Locally or Privately Hosted AI Models
For organisations dealing with highly sensitive data, relying on publicly hosted AI models may not be a viable option. Instead, deploying a privately hosted AI model can offer greater control, security, and compliance benefits.
For organisations looking to fully embrace and invest in AI, another option may be to locally host certain open source models like llama-gpt on their own hardware, in house. Hosting a large model in house can cost as little as a few thousand dollars on consumer hardware.
If the upfront costs are a little much, an alternative is cloud-hosting on a platform such as Novita.ai, that allows individuals and businesses to deploy models on dedicated hardware instances. This helps control what is being saved, track usage and provide certainty on exactly where the data is being processed.
2. Monitor & Defend Against AI-Powered Cyber Threats
As cybercriminals increasingly leverage artificial intelligence (AI) to enhance and automate their attacks, businesses must bolster their cybersecurity defenses to address sophisticated threats such as AI-enhanced phishing, deepfakes, and misinformation campaigns. Implementing a comprehensive strategy is essential to safeguard organizational assets and reputation.
Train Employees to Recognize AI-Enhanced Phishing Attempts and Fraud
AI has enabled the creation of highly convincing phishing attempts that can deceive even vigilant employees. For instance, in Hong Kong, a finance employee was duped into transferring $25 million after participating in a video call where fraudsters used deepfake technology to impersonate the company’s chief financial officer and other staff members.
Training initiatives should focus on:
- Identifying AI-generated phishing emails: Educate employees on recognizing signs of phishing, such as unexpected requests, urgent language, or inconsistencies in email addresses.
- Detecting deepfake impersonations: Provide guidance on verifying the authenticity of video calls and voice messages, especially when they involve financial transactions or sensitive information.
- Implementing verification protocols: Encourage the use of secondary channels to confirm requests for sensitive actions, such as phone calls or in-person confirmations.
- Multi-factor authentication (MFA): Ensure all financial and data access requests require multiple verification steps that cannot be overridden via a single request.
Monitor For for AI-Generated Misinformation and Fraudulent Activity
AI-powered misinformation campaigns can damage brand reputation, manipulate share prices, and mislead customers. Businesses must actively track how their brand is represented online and identify malicious deepfake content or synthetic media that could mislead stakeholders.
Key monitoring strategies include:
- Brand reputation monitoring: Use AI-driven analytics to scan social media, forums, and news websites for false claims, scam accounts impersonating the company and other fraudulent activity
- Swift response strategies: Prepare countermeasures such as public statements, fact-checked responses, and takedown requests to address AI-generated misinformation.
- Collaboration with platforms: Engage with social media networks, regulatory bodies, and cybersecurity firms to combat AI-driven disinformation campaigns.
3. Reduce AI Bias & Ensure Fair Decision-Making
Just because an AI isn’t human doesn’t mean it doesn’t need to be monitored. AI models can make mistakes, develop biases, and change over time as they process more data. Without regular oversight, these systems may lead to unintended consequences or unfair decisions.
Regularly Audit AI Models
Regularly reviewing AI decisions is essential for identifying issues and fixing biases. AI models can unintentionally inherit biases from the data they are trained on, such as favoring male candidates in hiring processes if the training data is skewed toward historically male-dominated industries. For instance, Amazon had to abandon an AI hiring tool because it favored male candidates, as it was trained on data where men were overrepresented. Additionally, AI models can generate responses that are factually inaccurate, a phenomenon known as “AI hallucinations,” where the model might confidently offer false information, such as citing nonexistent legal cases or inventing quotes, which can mislead users who rely on AI for guidance, particularly in healthcare and law enforcement.
Furthermore, AI models might avoid addressing certain topics with scripted, neutral responses, particularly when it comes to controversial or sensitive subjects like politics, climate change, or social issues. When asked about these topics, an AI might respond with vague or evasive statements and in some cases give misleading information.
To address these concerns, it is crucial for businesses to regularly audit systems to ensure they are performing correctly and giving the expected outputs.
Keep Humans In The Loop
No matter how advanced and embedded AI becomes, human oversight will always be crucial because a computer can’t be held accountable for their decisions. As IBM pointed out in its 1979 slide which is just as relevant today, AI and computers are just tools—they don’t have the moral judgment or accountability needed for management decisions.
While AI can analyse data and offer insights, it can’t fully grasp context or take responsibility for its actions. Management decisions, which often involve ethical and complex human factors, should always be made by people who are accountable for the outcomes, ensuring AI remains a tool to support, not replace, human decision-making.
5. Prepare for AI System Failures & Malfunctions
AI systems are powerful, but they are not always predictable. Even small updates or changes in data can lead to unexpected results and alter how the system behaves. It is crucial to have ongoing testing and continuous monitoring in place, ideally automated, to quickly spot any issues before they become problems or cause damage.
Like any system, outages and malfunctions are inevitable, especially if your organization depends on third-party APIs for AI tools. If the service goes down or experiences issues, it could disrupt your operations. At the time of writing the OpenAI API had a uptime of 99.57% and ChatGPT 99.39% which is very poor uptimes by industry standards.
Always have contingency plans ready to go, such as alternative provider to switch to, or as worst case a manual processes to ensure that you can keep things running.
The Future of AI Security: Staying Ahead of Threats
As AI technology continues to evolve, so will the security threats associated with it. Businesses need to keep a proactive approach to security and AI by staying informed on emerging risks and following the technology.
By implementing strong cybersecurity measures, fostering AI transparency, and maintaining human oversight, organizations can harness AI’s potential without compromising security and ethical standards.
Final Thoughts
AI presents incredible opportunities, but businesses must be vigilant against its risks and follow the technology to stay informed on emerging risks.
How is your company addressing AI security concerns? Share your thoughts in the comments below!
