amiantos.net

The Varying Levels of Getting Started with “Uncensored” LLM-Powered Chatbots (2024 Update)

This post is an update to my previous post, May 2023’s “The Varying Levels of Getting Started with “Uncensored” LLM-Powered Chatbots“. A lot has happened in 11 months. Well, not a lot, but, things have changed a bit.

I think that post is still worth reading, because it explains more in depth what we’re talking about exactly, and why, and I am going to take a stab at cutting to the chase and just updating a couple specific points from that post.

Previously, I said that you could get a pretty good chatbot experience out of OpenAI’s APIs, and you should try that first. That is no longer the case. Big corporate LLMs have implemented aggressive content filtering that really restrict your chatbots in ways that they weren’t 11 months ago. That isn’t to say corporate LLMs don’t have their uses, but if you’re looking for “uncensored” LLMs, which is the point of this post, I can no longer recommend OpenAI, Google, Anthropic, etc.

Previously, I said that currently available open source LLMs like Pygmalion 6B “weren’t very good”. That is, thankfully, no longer the case. Specifically, I think a model named Fimbulvetr is very good and you can likely run it at home very easily, or find it hosted on the AI Horde usually. I think this takes a lot of the sting out of commercial LLMs becoming very sanitized and being turned into the equivalent of hammers and wrenches.

Some things have also changed with the software that you can use to run models. There’s a lot of it. I am going to reference the things I have used specifically and can recommend depending on your circumstances. Without further ado, here we go…

We’re going to start at the lowest level of effort and work our way up as we progress into the hobby. Sounds fun, right?

An Easy and Free iPhone / iPad Client

Do you just want to chat with some bots to see what it is like, and not spend any money or time on it? I just released an iPhone / iPad chat client for the AI Horde, called Inneal. It’s free, and use of the AI Horde is free. It also hides away some of the fiddly bits that can be confusing in other clients, so you can start importing character cards and chatting with them right away. Yes, this is self-promotion, but I made this app specifically to make it dead simple to start chatting right away.

A Free Browser-based Client

Once you’ve become a bit bored with what you can do in Inneal, you can branch out into another free client for the Horde that does some other things, called Kobold Lite. The interface for this app is a bit confusing but you can also import character cards from CHUB, and do some other stuff in this client, like use straight text completion, or Adventure Mode which tries to do kind of a Zork type thing. It’s pretty neat.

A Free and More Robust Browser-based Client

If you want to get really deep into using third party web clients, and I don’t honestly recommend it, because why would you want all your chats stored on some random server somewhere, there’s a very nice website called Agnai that supports a lot of chat features that are becoming a standard, like lore books. This client supports lots of LLM APIs as well as the AI Horde.

Paid Options for the Computer Illiterate

If you’re computer illiterate and you have money, there are plenty of paid sites where you can chat with bots for money. Some of them have some very large models you can use, which should, in theory, be better at chat than smaller models. I can’t say whether any of these sites are good or not, and I wouldn’t personally use them myself. I felt it would be unfair not to mention them, however.

The Go-To At-Home Client

If you’re computer literate, and you want to keep your chats on your computer where they are safe, you should consider installing SillyTavern. In addition to being fully under your control, it seems to be the gold standard client at the moment for chatting and is jam-packed with features. I just wish they’d change the name.

SillyTavern can connect to the horde, of course, so you can use this freely still, but really, you should start running your own models locally if you can, which we talk about next.

Running Your Own Models At Home

Running models at home is easier than ever before, with some caveats, namely that you’re running quantized models, which means they may not be quite as good as the full-sized models running on server farms with tons of VRAM available. But they still do the job better than anything else that has ever existed, so, nothing to complain about, really.

If you’re running Windows

If you’ve installed SillyTavern and you’re using it, then you’re definitely serious enough to run your own LLMs locally, and if you have a gaming graphics card made in the last couple years, you probably can easily run Fimbulvetr locally. I’ve used Koboldcpp to do this on my PC lately.

You’ll want the Fimbulvetr-11B-v2.q4_K_S.gguf file from the link below. This is a quantized version of the model that will run on smaller GPU graphics cards, like 8 GB of VRAM. But it also runs well and works just fine on bigger graphics cards.

If you’re running macOS

If you have an M-series Mac, you can use Ollama to run models locally, and the version of Fimbulvetr available under Ollama runs really well on my M2 Pro mac Mini, and SillyTavern has an Ollama specific connection in it already.

Now What?

If you had fun getting Fimbulvetr working, congratulations to you, you’re just getting started. There are a lot of cool models out there, and a lot of stuff to learn about (like Instruct Mode). I recommend checking out this page that reviews LLMs based on how good they are for chatbot roleplay. It’s updated quite often. Chronoboros, Psyfighter2, Noromaid Mixtral, … there are a lot of fun models to try out to see what works best for what you like.

I’d also recommend joining the KoboldAI Discord, it’s a great place to geek out, get help, and learn about all the latest models and how they work.

If you get into SillyTavern, there’s a lot to learn there, and they have their own Discord server that could be worth a perusal.

Also, go back and read my original post, if you didn’t, there’s some other stuff in there and if you’re all the way down here, you must love reading, so do it some more?

That’s it for now! I’m sure there will be another update, perhaps this year or next year. Just in the past month, we’ve seen the release of Command-R and Llama 3, two very large open source models that are sure to help push forward progress on these fronts. We’ve also seen corporate AI get more and more restrictive about how it can be used, as world governments begin to make moves that could enable regulatory capture, making it more important than ever before that these fields move quickly. Kudos to the torch-bearers.