A year ago, I did some reverse engineering. I figured a pattern matching engine has to be good matching patterns, so I threw the messed-up output of Ghidra into Claude to clean it up and to give me an overview of what the fuck happens in a certain part.
I was absolutely surprised. Sure, it got some things wrong, but it knew way more than it should have been able to guess from that short bit of fucked up, optimised not-quite-C. Including guessing context that it should not be able to guess. This was a firmware with not-yet-released functions that, to my knowledge, shouldn’t exist outside of this bit of binary firmware. At least not for us non-NDA mortals.
At the time, Mistral had a new-ish model as well, so I figured I’d give it a try. It worked as expected, cleaned up things, gave variables reasonable names, made educated guesses and so on, but didn’t act as if it had background info. And I was using Sonnet 4.x, so the small Claude which shouldn’t be that much larger.
Since then, I have the conspiracy theory that the Claude people threw a bunch of proprietary sources in the training, too. I have no idea how, maybe aggressively training on user inputs? Anyway, this does not help to allalleviate my suspicions.
I threw about 10 short phrased meeting notes (4-5 words each) into an LLM to get some document formatting started with titles and sections etc. It built a full scope document and inferred a ton of information about the project and product.
It was surprising how much it could infer, and suggested very sensible testing criteria and alternate solutions should the main product path not pan out.
I can see why people start believing it “thinks”, because it was remarkable what it “understood” about a vague new product. Had I sent the info out to stake holders, they would have assumed this person was a key team member that was involved from day 1.
I specifically asked our CEO about this just to cover my ass. It seems insane to trust these guys at all. Here in the USA, small businesses have de facto no legal protection from large businesses.
Absolutely. Especially if you have company info / IP that someone else would pay a lot for. This is a question for lawyers and managers to negotiate with the lawyers at OpenAI, Anthropic etc, not for us little people to decide. I’m not throwing company data in an online service that is not cleared for us to use, and I don’t even work with spicy stuff. Back in my IT days we would have called that a data breach, which would have lead to something from an unpleasant hour in the boss’ office to a promotion to customer.
Makes me wonder how much of the frontier models hacking is just “our devs uploaded the codebase to Claude and suddenly it knows where all the backdoors are”?
Since then, I have the conspiracy theory that the Claude people threw a bunch of proprietary sources in the training, too.
An enormous amount of people use this technology for privat company data without batting an eye; not just software developers, but business people for reports, etc.
They all throw their confidential data in there without even considering the implications, just “trusting” Anthropic. claude code bots are running on tens of thousands of proprietary code bases that used to be private, and of course they are using all this data to train their model.
It surely gets better at coding in general, after all there are smart people at anthropic with near limitless budgets, but my conspiracy thesis is that it gets a lot better at replicating stuff it has seen before, but really shouldn’t have.
I’m sure it’s all but officially confirmed, that all the people upload to it goes to training data unsanitised. And Microslop for sure uploaded entirety of their codebase to all the bots. This is at least part of the reason why claude is so surprisingly OK at reverse-engineering all the big stuff.
Tinfoil hat time:
A year ago, I did some reverse engineering. I figured a pattern matching engine has to be good matching patterns, so I threw the messed-up output of Ghidra into Claude to clean it up and to give me an overview of what the fuck happens in a certain part.
I was absolutely surprised. Sure, it got some things wrong, but it knew way more than it should have been able to guess from that short bit of fucked up, optimised not-quite-C. Including guessing context that it should not be able to guess. This was a firmware with not-yet-released functions that, to my knowledge, shouldn’t exist outside of this bit of binary firmware. At least not for us non-NDA mortals.
At the time, Mistral had a new-ish model as well, so I figured I’d give it a try. It worked as expected, cleaned up things, gave variables reasonable names, made educated guesses and so on, but didn’t act as if it had background info. And I was using Sonnet 4.x, so the small Claude which shouldn’t be that much larger.
Since then, I have the conspiracy theory that the Claude people threw a bunch of proprietary sources in the training, too. I have no idea how, maybe aggressively training on user inputs? Anyway, this does not help to allalleviate my suspicions.
I threw about 10 short phrased meeting notes (4-5 words each) into an LLM to get some document formatting started with titles and sections etc. It built a full scope document and inferred a ton of information about the project and product. It was surprising how much it could infer, and suggested very sensible testing criteria and alternate solutions should the main product path not pan out.
I can see why people start believing it “thinks”, because it was remarkable what it “understood” about a vague new product. Had I sent the info out to stake holders, they would have assumed this person was a key team member that was involved from day 1.
I specifically asked our CEO about this just to cover my ass. It seems insane to trust these guys at all. Here in the USA, small businesses have de facto no legal protection from large businesses.
Absolutely. Especially if you have company info / IP that someone else would pay a lot for. This is a question for lawyers and managers to negotiate with the lawyers at OpenAI, Anthropic etc, not for us little people to decide. I’m not throwing company data in an online service that is not cleared for us to use, and I don’t even work with spicy stuff. Back in my IT days we would have called that a data breach, which would have lead to something from an unpleasant hour in the boss’ office to a promotion to customer.
Makes me wonder how much of the frontier models hacking is just “our devs uploaded the codebase to Claude and suddenly it knows where all the backdoors are”?
An enormous amount of people use this technology for privat company data without batting an eye; not just software developers, but business people for reports, etc.
They all throw their confidential data in there without even considering the implications, just “trusting” Anthropic. claude code bots are running on tens of thousands of proprietary code bases that used to be private, and of course they are using all this data to train their model.
It’s standard operating practice to steal everything. The world’s full of dumb fucks.
Then people are surprised that claude code gets better at coding
Better at replicating closed source software.
It surely gets better at coding in general, after all there are smart people at anthropic with near limitless budgets, but my conspiracy thesis is that it gets a lot better at replicating stuff it has seen before, but really shouldn’t have.
Meta had an explicit higher price for using their model without retention of prompts for training. This all but confirms your theory.
Wait… wait… you believe Facebook?
Oh, facebook will steal the data even if they say they won’t. What’s interesting is they put a price on blatant prompt copying.
We steal your data: $5 We steal your data and tell you we don’t: $10
Pretty much.
I’m sure it’s all but officially confirmed, that all the people upload to it goes to training data unsanitised. And Microslop for sure uploaded entirety of their codebase to all the bots. This is at least part of the reason why claude is so surprisingly OK at reverse-engineering all the big stuff.