Check your tech stack for AI training leaks
Accounting firm tech stacks may be leaking more information than leaders realize, including sensitive client data, due to unintentionally agreeing to innocent-sounding terms and conditions that give vendors permission to access and use it how they see fit, such as training their own AI models.
Processing Content
As a result, many firms are likely in violation of any number of rules, regulations and standards regarding data management and financial privacy without even realizing it.
Charlie Barmore, a tax practitioner who specializes in AI implementations, said this issue encompasses both small startup vendors and large, widely used platforms. While the size differences are obvious, both may potentially be accessing sensitive data that practitioners would prefer they not reach. This is because both may potentially have inserted legal language in their governing documents — whether that’s an end user license agreement, a data partnership agreement, a privacy policy, the terms of service, or even the contractual language itself — that give them permission to do so. And this is not even a new issue; such language has been present for years for certain vendors.
“If you go and read their terms, they’re not exactly favorable. Some of the terms have been like that for several years,” said Barmore.
While there is great variation between how vendors word their policies, there are certain common terms, phrases or clauses that should raise eyebrows. Barmore said the first step is to understand how the vendor defines “customer data.” This allows a firm to understand what their agreements actually encompass.
From there, he said, people should look for words such as “improve,” “enhance” or “develop,” particular if they’re placed near descriptions of services or products. Other red flag terms include “share,” “disclose” or “make available.” Another thing leaders should look for is whether there’s any language regarding data licensing and, if so, how long the license is for and what it allows. Sometimes, he said, firms accidentally agree to a perpetual irrevocable license that remains in effect even when the business relationship is severed.
“Those are some things I would definitely try to Ctrl-F and see the surrounding language around those kinds of key terms,” he said.
Viral Dhimar, chief information security officer at Top 50 firm Sikich, noted that some vendors will try to sneakily access sensitive data. Part of the firm’s vendor review process is looking at things like privacy policies, API documentation and trust centers, searching for phrases that go beyond what the contract stipulates. Terms and phrases that might suggest this include “develop new features,” “model improvements,” “training data,” “telemetry,” “anonymized data,” and “improve product and services.”
“If in a contract or other areas mentioned above state that the vendor could ‘use customer data that is anonymized to improve product and services,’ Sikich and other accounting firms should ask exactly how the data is being used to improve said product and services,” he said in an email.
Brian Tankersley, an accounting technology advisor with K-2 Enterprises, stressed the importance of having a qualified person read the documents that come with a new solution and be able to compare them with similar products.
“Somebody needs to read the whole privacy policy and the whole terms of service,” he said. “It doesn’t have to be everybody. But somebody that is an expert needs to read these. You need to run them through something similar to AI, so that you can compare them apples to apples to each other.”
These documents can be long, dense and complex. Even after looking for key phrases and terms, people can still be confused. Tankersley noted this is an area where AI might be able to help. When evaluating solutions, he copies the relevant information into a document, uploads it to an LLM and has the model analyze the policy in plain language. He uses the below prompt:
You are an expert in privacy and a gifted technical writer which takes complex documents like privacy policies and clearly summarizes them for professionals. When I input a privacy policy, you should respond with a summary document as follows:
Format for Privacy Policy Summaries
- Product Name
- Publisher/service provider name
- Policy document name:
- 300 word summary of privacy policy
- Key phrases from privacy policy regarding information sharing with others
-
Listing of explicit third parties/subprocessors with whom information is shared, in a table, with columns as follows:
- Name of subprocessor
- Purpose of information sharing
- Kind of information shared
- What kind of information can users opt out of being shared
- Directions and/or links for how to opt out of information sharing
Subprocessors and their subprocessors
A major complicating factor in vetting solutions for privacy is how integrated software can be with subprocessors, loosely
Even if the actual solution used by a firm checks out as OK, the third parties that vendor works with may not. Therefore, it’s important to not only look at the governing policies of the direct vendor but also understand which other companies work with that vendor. Similarly, it’s important to examine whatever platforms a solution has integrated with, since data could flow through there as well.
“This is what’s making it very challenging, because these solutions may be using a third party to then produce this specific output or implement this certain feature,” said Barmore. “We still do need to vet the software that we’re using and any of its known subprocessors or any of the vendors that it uses,”
Such evaluations are part of the standard evaluation process at Sikich, according to Dhimar.
“As part of vendor management at Sikich, we evaluate AI/LLM models and other technologies that access information,” he said. “We ask what data the vendor will be accessing, where it is stored, how it will be utilized ,and what third or fourth party will also potentially process it.”
While this might seem simple in principle, Tankersley said it can be much more complicated than people may at first think because even when someone finds out who the subprocessors are (and they are not always disclosed), these subprocessors are not necessarily easy to find and don’t always make their documents readily available. When they do, they’re not always clear. Compounding this issue, even if the subprocessor checks out, there is still a need to check out the subprocessors to the subprocessors.
“Here’s the problem: One, you have to go find their terms, and two, they’re not always explicitly disclosed, so you have an opaque thing. … One of the things they’re required by law is to disclose who those subprocessors are. But the problem is the subprocessors have subprocessors,” said Tankersley.
The consequences of ignoring subprocessors can be severe. He cited someone who used a payroll solution that used a third party subprocessor to act as a bank feed aggregator, as many platforms do. The solution’s documents mentioned its own terms and conditions, but not the subprocessor’s; instead, it simply said that by agreeing to their rules, the user also agreed to the subprocessor’s rules without explaining exactly what those rules were. If the user went to the subprocessor’s actual site and looked up its terms and conditions, the user would have seen that they reserve the right to share user data with other third parties, such as credit-reporting bureaus like Equifax.
“I’ve had CPAs that had clients that went up to them where they were using a payroll solution that was talking to Equifax, and they applied for something, and the bank already knew, because of Equifax, that they made a million dollars,” said Tankersley. “And [the client] said, ‘My accounting firm does this. That’s the only place it could have possibly leaked out.’ And it [became] a relationship issue for that firm. And so part of the challenge with this is how do we work in this world where everything’s integrated, and we need to make sure everybody keeps their mouth shut?”
Complicating things even further, according to Tankersley, is the fact that terms and conditions can change. While a subprocessor may have had stringent privacy controls one day, this could change dramatically the next. Unless someone has been routinely checking all of these documents for updates, the firm would not even know since it doesn’t have a direct relationship with them. This is especially risky in light of the fact that major AI providers like OpenAI and Anthropic — neither of which is exactly famous for their data privacy — are increasingly serving as subprocessors for vendors who’ve licensed their models for their own products, as both have already changed their privacy policies several times.
“Most of the terms of service say things like we can amend this and change it as we’d like to. … You have Claude. You’ve vetted Claude. You’ve embedded Claude into your workflows, and suddenly now they’re unilaterally changing this in the middle of your workflow after you’ve implemented it,” said Tankersley.
And this is even assuming the firm knows it’s using Claude; it may not even realize that the solution it bought licenses models from Anthropic and so it’s basically using Claude with different branding.
Vendor vetting
This gets into another aspect of vetting solutions for privacy: asking pointed, specific questions at vendor meetings, including about subprocessors. Barmore asks specifically whether or not there is a training clause, and to have the vendor point out the specific line items in the terms to show him how it works. He also said it is important to ask how the vendor defines customer data and how it uses it, once again asking it to cite the specific terms rather than just taking their word for it.
“Ask specific questions about what data do you all collect? What do you all do with it? Is there training? Where is it stored? Does it use subprocessors? Where are they located, and how is the data used there? It’s really best to have the sales rep pull it straight out of the terms, as opposed to relying on what they may say,” he said.
Dhimar, from Sikich, said it is a great practice to ask the vendor questions such as:
- Can you state in writing that no client data, prompts, output, usage, anonymized or derived data are used to train any model (current or future)?
- Can you state in writing that no third party, fourth party, downstream provider, model provider or cloud provider will use the data to train their models?
- Where is the data stored after it leaves Sikich network/tenant?
- Please identify exactly where in the contract or other binding agreement that prohibits training on our data?
Tankersley was blunt when it came to what firms should ask their vendors regarding privacy:
“You want to ask, first, is my data used to train your models? And if it is, then you don’t want any part of that action. Secondly, what happens to my private information?” he said.
Technical tests
While documentation and vendor questions can answer a lot, there is always the possibility someone might be outright lying. However, testing this empirically can be difficult given the black box nature of many AI models, as well as the lack of granularity of what data is getting transmitted where.
“It’s hard to actually test it because once it is digested by the software and goes into the cloud, we now have lost access to it. They do the processing. They use it however they have stated they are going to use it, and then it’s left to us to trust that they’re only going to use it the way that they stated in their terms,” he said.
Still, firms are not completely helpless in this regard. Dhimar said that buyers should first ask upfront, as part of any vendor management process, for network diagrams, architecture diagrams, data flow diagrams, and a list of third-party, fourth-party and other subprocessors.
Then, as part of network and application monitoring, firms should review any unexpected data transfers (the size of the data is a big tell), which cloud regions were used to store or process the data, and API gateways and/or endpoints.
“If data is going to other areas not explicitly identified in the diagrams or documentation listed above, it may indicate a breach of contract,” he said.
But how does a firm know whether sensitive data has already left the company and been used to train an AI model? Provided one knows which model may have been trained, Tankersley said someone could ask it to describe things about the firm or the client that, in principle, it should not know. If it begins spitting out sensitive information that no one else should have, then it was likely trained on the firm’s data.
“Do a prompt like that inside one of those things and see what it figures out about you,” he suggested. “What does it know about me from this? … I think in general one of the things people need to be very careful about is what they connect to [AI models] because once you let them in your email, they’re in your email.”
Regulatory risk
Considering how many products take user data to train their AIs, how many accounting solutions now use AI, and how many of those AI models are run through third party subprocessors, it stands to reason that many firms are unknowingly in violation of rules and regulations from agencies like the IRS or SEC that mandate client data remain private.
Accountants operate under so many rules and regulations that have not fully adapted to new technology, Barmore pointed out, so they’re trying to fit new practices into old models, creating a regulatory minefield. While regulators are unlikely to discover an individual firm is in violation when the firm itself isn’t aware, Barmore said this should not be confused with never being found out, as any number of things could expose them, such as a data breach.
“If there’s a security breach in some of these models, and it comes to light, or there’s a lawsuit, I wouldn’t advise operating under that assumption that someone may never find out because I don’t think that’s the responsible way to go about it,” he said.
Tankersley agreed, but said this is not a new issue only with AI. It is likely any given firm has at least some data leaking somewhere. He pointed to a firm where an intern was looking at health-related audit evidence and took a picture of a document with their phone, which then automatically went into iCloud, which then counted as a HIPAA violation, as iCloud “does not do any of the regulatory stuff that needs to be done.”
“The problem is, first and foremost, people have got to get their house in order with respect to their staff usage,” he said.
Besides analyzing documentation, questioning vendors and performing technical monitoring, Barmore said it’s important to find whether or not a solution has a way to opt out of providing training data. Sometimes it’s as simple as clicking a toggle in preferences. He conceded it’s an imperfect solution since it does not address actual data storage and retention on the part of the vendor.
Tankersley suggested seeing which companies work directly with the federal government, which tends to carry stringent data control requirements; while doing this does not completely eliminate risk, he said it should lower it. But more than anything else, according to Tankersley, is making sure you actually read what you agree to, something he believes far too few people do today.
“Generally the bigger issue here is that we have just said, ‘Yeah, I’m going to click on OK and accept that.’ The problem is if that thing said, ‘I give somebody title to my left arm and my firstborn child,’ some people would still just click on OK.”
