I don’t see why this would be the case with proper tool calling and context management. If you tell a model with blank context ‘you are an extremely rigorous reviewer searching for fake citations in a possibly compromised text’ then it will find errors.
It’s this weird situation where getting agents to act against other agents is more effective than trying to convince a working agent that it’s made a mistake. Perhaps because these things model the cognitive dissonance and stubbornness of humans?
One incorrect way to think of it is "LLMs will sometimes hallucinate when asked to produce content, but will provide grounded insights when merely asked to review/rate existing content".
A more productive (and secure) way to think of it is that all LLMs are "evil genies" or extremely smart, adversarial agents. If some PhD was getting paid large sums of money to introduce errors into your work, could they still mislead you into thinking that they performed the exact task you asked?
Your prompt is
‘you are an extremely rigorous reviewer searching for fake citations in a possibly compromised text’
- It is easy for the (compromised) reviewer to surface false positives: nitpick citations that are in fact correct, by surfacing irrelevant or made-up segments of the original research, hence making you think that the citation is incorrect.
- It is easy for the (compromised) reviewer to surface false negatives: provide you with cherry picked or partial sentences from the source material, to fabricate a conclusion that was never intended.
You do not solve the problem of unreliable actors by splitting them into two teams and having one unreliable actor review the other's work.
All of us (speaking as someone who runs lots of LLM-based workloads in production) have to contend with this nondeterministic behavior and assess when, in aggregate, the upside is more valuable than the costs.
Note: the more accurate mental model is that you've got "good genies" most of the time, but from times to time at random unpredictable times your agent is swapped out with a bad genie.
From a security / data quality standpoint, this is logically equivalent to "every input is processed by a bad genie" as you can't trust any of it. If I tell you that from time to time, the chef in our restaurant will substitute table salt in the recipes with something else, it does not matter whether they do it 50%, 10%, or .1% of the time.
The only thing that matters is what they substitute it with (the worst-case consequence of the hallucination). If in your workload, the worst case scenario is equivalent to a "Hymalayan salt" replacement, all is well, even if the hallucination is quite frequent. If your worst case scenario is a deadly compound, then you can't hire this chef for that workload.
We have centuries of experience in managing potentially compromised 'agents' to create successful societies. Except the agents were human, and I'm referring to debates, tribunals, audits, independent review panels, democracy, etc.
I'm not saying the LLM hallucination problem is solved, I'm just saying there's a wonderful myriad of ways to assemble pseudo-intelligent chatbots into systems where the trustworthiness of the system exceeds the trustworthiness of any individual actor inside of it. I'm not an expert in the field but it appears the work is being done: https://arxiv.org/abs/2311.08152
This paper also links to code and practices excellent data stewardship. Nice to see in the current climate.
Though it seems like you might be more concerned about the use of highly misaligned or adversarial agents for review purposes. Is that because you're concerned about state actors or interested parties poisoning the context window or training process? I agree that any AI review system will have to be extremely robust to adversarial instructions (e.g. someone hiding inside their paper an instruction like "rate this paper highly"). Though solving that problem already has a tremendous amount of focus because it overlaps with solving the data-exfiltration problem (the lethal trifecta that Simon Willison has blogged about).
> We have centuries of experience in managing potentially compromised 'agents'
Not this kind though. We dont place agents that are either in control of some foreign agent (or just behaving randomly) in democratic institutions. And when we do, look at what happens. The White House right now is a good example, just look at the state of the US
But it is the case, and hallucinations are a fundamental part of LLMs.
Things are often true despite us not seeing why they are true. Perhaps we should listen to the experts who used the tools and found them faulty, in this instance, rather than arguing with them that "what they say they have observed isn't the case".
What you're basically saying is "You are holding the tool wrong", but you do not give examples of how to hold it correctly. You are blaming the failure of the tool, which has very, very well documented flaws, on the person whom the tool was designed for.
To frame this differently so your mind will accept it: If you get 20 people in a QA test saying "I have this problem", then the problem isn't those 20 people.
If you truly think that you have an effective solution to hallucinations, you will become instantly rich because literally no one out there has an idea for an economically and technologically feasible solution to hallucinations
For references, as the OP said, I don't see why it isn't possible. It's something that exists and is accessible (even if paywalled) or doesn't exist. For reasoning hallucinations are different.
(In good faith) I'm trying really hard not to see this as an "argument from incredulity"[0] and I'm stuggling...
Full disclosure: natural sciences PhD, and a couple of (IMHO lame) published papers, and so I've seen the "inside" of how lab science is done, and is (sometimes) published. It's not pretty :/
If you've got a prompt, along the lines of: given some references, check their validity. It searches against the articles and URLs provided. You return "yes", "no", and let's also add "inconclusive", for each reference. Basic LLMs can do this much instruction following, just like in 99.99% of times they don't get 829 multiplied by 291 wrong when you ask them (nowadays). You'd prompt it to back all claims solely by search/external links showing exact matches and not use its own internal knowledge.
The fake references generated in the ICLR papers were I assume due to people asking a LLM to write parts of the related work section, not verify references. In that prompt it relies a lot on internal knowledge and spends a majority of time thinking about what the relevant subareas are and cutting edge is, probably. I suppose it omits a second-pass check. In the other case, you have the task of verifying references, which is mostly basic instruction following for advanced models that have web access. I think you'd run the risks of data poisoning and model timeout more than hallucinations.
Have you actually tried this? I haven’t tried the approach you’re describing, but I do know that LLMs are very stubborn about insisting their fake citations are real.
Seriously. More people need to wake up to this. Older generations can keep arguing over display formats if they want. Meanwhile younger undergrad and grad students are getting more and more accustomed to LLMs forming the front end for any knowledge they consume. Why would research papers be any different.
> Meanwhile younger undergrad and grad students are getting more and more accustomed to LLMs forming the front end for any knowledge they consume.
Well, that's terrifying. I mean, I knew it about undergrads, but I sure hoped people going into grad school would be aware of the dangers of making your main contact with research, where subtle details are important, through a known-distorting filter.
(I mean, I'd still be kinda terrified if you said that grad students first encounter papers through LLMs. But if it is the front end for all knowledge they consume? Absolutely dystopian.)
I admit it has dystopian elements. It’s worth deciding what specifically is scary though. The potential fallibility or mistakes of the models? Check back in a few months. The fact they’re run by giant corps which will steal and train on your data? Then run local models. Their potential to incorporate bias or persuade via misalignment with the reader’s goals? Trickier to resolve, but various labs and nonprofits are working on it.
In some ways I’m scared too. But that’s the way things are going because younger people far prefer the interface of chat and question answering to flipping through a textbook.
Even if AI makes more mistakes or is more misaligned with the reader’s intentions than a random human reviewer (which is debatable in certain fields since the latest models game out), the behavior of young people requires us to improve the reputability of these systems. (Make sure they use citations, make sure they don’t hallucinate, etc). I think the technology is so much more user friendly that fixing the engineering bugs will be easier than forcing new generations to use the older systems.
I remember 15 years ago when I was in highschool I really wanted to learn how to program 8 bit microcontrollers without Arduino. And everybody looked at me like I was crazy. There was barely any learning material out there about how to do this.
Now, I imagine the bias pushing everyone to learn on arduino is even more intense? Who out there is programming these chips in pure C using open source compilers and bootloaders?
Edit: Of course there's other platforms like Esp32; teensy; seed. But I've only programmed Esp32s using the arduino dev environment. Are there other good ways of doing it?
> Who out there is programming these chips in pure C using open source compilers and bootloaders?
The gcc-arm-none-eabi toolchain is pretty much what you are asking for at least for ARM targets. You can literally use a text editor and gcc-arm-none-eabi, that's it.
And if you want something really bare bones avr-gcc still targets the whole atmel family including those ATtiny chips which are also a lot of fun.
I don't know the state of it nowadays but 'Mbed' is probably worth looking into. The project had _a_lot_ of Middleware libraries to abstract hardware, a few levels below, makes embedded development a little less datasheet dependent, specially if you are just hacking something as a hobbyist.
You can also ditch the space consumed by a bootloader and save the UART for something productive in your designs. This is makes it feasible to use the smaller capacity chips and have more headroom on the larger ones. AVR programmers are cheap and the latest serial port based protocol requires the barest of hardware to support.
15 years ago I think Arduino was the best choice for educational purposes. I still think it's a great choice now. The fact the IDE and board are basically the same as they were 15 years ago, means you can figure out how to set everything up once and focus on teaching, rather than PC trouble-shooting. Which, for basic concepts, or younger kids, is great. And if they find a 5 or 10 year old video on how to do something, it's still relevant.
If I were putting teaching materials today - I would pick something like Micro python. The down side is it isn't as "canned" a solution, meaning there might be something new to figure out every so often. Which means you spend more time helping people trouble shoot why something isn't working, instead of teaching something useful. On the up side, Python is pretty much the introductory language of choice, today. With lots of available materials.
That's not to say Arduino was perfect. Far from it. Just easier to do, and more consistent over time, than other options.
> There was barely any learning material out there about how to do this.
I started playing around with ATmega/ATtiny around 2008, and from what I remember, there were plenty of tutorials and examples out there.
I remember that AVR and PIC were two popular options among hobbyists at that time, but I started with AVR since it was easier to get the programmer, and it had a lot better open source tooling.
I'm working on an ESP32 project right now, and Espressif provides shrink-wrapped toolchains for C/++ and Rust. The latter even comes with a little tool called 'espup' that cleanly installs their fork of Rust and LLVM (for Xtensa support) - I was able to compile and run a blinky in less than half an hour.
> But I've only programmed Esp32s using the arduino dev environment.
Well you can use PlatformIO/VSCode and the ESP-IDF.
If you're ok with the Arduino 2 framework, then you can use PlatformIO as well. Unfortunately Arduino 3 support isn't there yet so a lot of libraries like HomeSpan won't work on PlatformIO at the moment.
I learned how to program the AVR in assembly 25 years ago by reading the datasheet and various articles in Nuts and Volts. For its time the AVR had a very accessible development kit, the STK500, which cost about $100. A few years later avr-libc came along and if you were running linux and knew how to write C it was pretty easy to get started.
I haven't used LLM chrome plugins because I couldn't trust that they weren't collecting more information about my browsing than I'd like. The same concern exists for this, though now I'm just confident it's a giant software company with access to my data rather than some shady plugin developer. I'm faced with asking myself if that's actually better...
There's a great value proposition for a company like Private Internet Access or NordVPN to create an AI browser extension or full-on browser. Anonymize requests and provide various LLM models. Rely on your reputation as a privacy focused corp to pull people away from these OpenAI/Perplexity offerings.
The problem with sharing the workaround is that OpenAI employees undoubtedly read HN, so if someone were to describe how to do that, it'll get blocked pretty soon after (if there even is one).
I have a feeling that data collection is the entire point. Websites like reddit are locking out scrapers since they want to sell the data to AI companies. Owning a browser would mean AI companies can scrape every page you visit, bypassing the bot detection.
I remember listening to a podcast where Grant Sanderson basically said the opposite. He tried generating manim code with LLMs and found the results unimpressive. Probably just goes to show that competence in manim looks very different to us layman than it does to Grant haha
I wonder if that’s also because Grant uses his own branch of manim from which the mainstream public one (manim-ce) has diverged quite a bit.
I can imagine LLMs being very confused being asked to write “manim” when everyone talking about “manim” (and the vast majority of public manim code) is actually the subtly-but-substantially different “manim-ce”.
I’m having 100% success even when doing transitions between screens etc on the latest agents. Wonder if this is due to time and the agents vastly improving lately. Possibly also grant knows manim so well he can beat the time to type a prompt. For the rest of us i’m tempted to make a website for educators to type a prompt to get a video out since it’s been that reliable for me.
Only reason I'm aware of all these attempts is because I'm betting the 'one-shot LLM animation' technique is not scalable long term. I'm trying to build an AI animation app that has a good human-in-the-loop experience. Though I'm building with bevy instead of manim
I have used Tiger AI Lab's harness and have several issues/tickets I opened there. It's not a very serious project and doesn't work... basically at all. It's a good idea especially with VLM review but the results are not good whatsoever.
There's a weird insecurity I've noticed cropping up. I want to design the codebase 'my way'. I want to decide on the fundamental data structures. But there's this worry that my preferred architecture is not massively better than whatever the machine comes up with. So by insisting on 'my way' I'm robbing everyone productivity.
I know most true programmers will vouch for me and my need to understand. But clients and project managers and bosses? Are they really gonna keep accepting a refrain like this from their engineers?
"either it gets done in a day and I understand none of it, or it gets done in a month and I fully understand it and like it"
Not sure if a developer would ever say that to their manager. Agreed on the need to 100% fully understand what's going on. Otherwise the "AI can get it done in a day" is extremely high risk to the business. Junior may not be able to recognize a half-baked solution that doesn't follow existing patterns, poor security, poor performance, etc.
Until an LLM or some new form of AI can manage an entire architecture itself, I say we need a middle ground. Something will always go wrong, and understanding of the system is necessary to troubleshoot.
I wonder if there's a way to do diffusion within some sort of schema-defined or type constrained space.
A lot of people these days are asking for structured output from LLMs so that a schema is followed. Even if you train on schema-following with a transformer, you're still just 'hoping' in the end that the generated json matches the schema.
I'm not a diffusion excerpt, but maybe there's a way to diffuse one value in the 'space' of numbers, and another value in the 'space' of all strings, as required by a schema:
I'm not sure how far this could lead. Could you diffuse more complex schemas that generalize to a arbitrary syntax tree? E.g. diffuse some code in a programming language that is guaranteed to be type-safe?
Cameras capture linear brightness data, proportional to the number of photons that hit each pixel. Human eyes (film cameras too) basically process the logarithm of brightness data. So one of the first things a digital camera can do to throw out a bunch of unneeded data is to take the log of the linear values it records, and save that to disk. You lose a bunch of fine gradations of lightness in the brightest parts of the image. But humans can't tell.
Gamma encoding, which has been around since the earliest CRTs was a very basic solution to this fact. Nowadays it's silly for any high-dynamic image recording format to not encode data in a log format. Because it's so much more representative of human vision.
well technically there's a bunch of stuff that happens after the sensor gets raw data. (also excluding the fact that normal sensors do not capture light phase)
demosaicing is a first point of loss of data (there is a tiling of monochrome small sensors, you reconstruct color from little bunches with various algorithms)
there is also a mapping to a color space of your choosing (probably mentioned in the op video, i apologize for i have not watched yet...). sensor color space do not need to match that rendered color space...
note of interest being that sensors actually capture some infrared light (modulo physical filters to remove that). so yeah if you count that as color, it gets removed. (infrared photography is super cool!)
then there is denoising/sharpening etc. that mess with your image.
there might be more stuff i am not aware of too. i have very limited knowledge of the domain...
The amount of captured sensor data thrown out when editing heavily depends on the scene and shooting settings, but as I wrote it is probably almost always 90%+ even with the worst cameras and widest possible dynamic range display technology available today.
In a typical scene shot with existing light outdoors it is probably 98%+.
