Deciphering Autocompletion: Mastering Temperature and Top P

An understanding of hyperparmeters like temp and topp is absolutely vital to understand the prompting process. These hyperparameters (that is, parameters that control the way prompts/programs are processed by the system vs parameters within the prompt/program which control the flow of the process itself) control the way autocompletion works. Temp ranges from 0 to around 2 (no hard limit, but practical issues) defaulting to 1 on the Playground/API/bots and floating around .65-.8 or so in ChatGPT. And TopP (ie. Top Percentage) naturally goes from 0 to 1, defaulting to 1. You will usually hear Temperature described as controlling "randomness" or "creativity" or even "likelyhood of hallucination". Then you'll hear TopP defined as "Temperature controls randomness! Weren't you listening?". These are lies.

Now, normally I have no problem with the old adage that "An ounce of inaccuracy can save a pound of explanation", but in this case it's more like a pound of inaccuracy saving a pound and half of explanation. So let's do some understanding. It will squint towards math, but don't worry - no equations. (And, y'know... that aforementioned ounce.)

You can think of the response generation process as the model linking together a locomotive. We can push that metaphor PRETTY far, actually, but we're gonna restrain ourselves to consider just the assembly process. In this metaphor, each train car is a word (well... token, actually, but lets not split hAIrs). It starts with the engine, which implies/necessitates/entails the coal car. It MUST come next so it does. From there on, the specific arrangement of cars is determined by the interplay of a million factors, like the timing to shuffle the assembling cars around the trainyard in the proper order, the geographical arrangement of all the cars in space, and most importantly the engineering requirements for things such as weight distribution, rider comfort, safety, and function. All of these things and more (rail worker strikes, anyone?) influence and often compel the choice of the next car to be added to the line. Every little thing going on in the trainyard CAN, in the right circumstances, radically change the final arrangement of the train.

The trainyard is your context, including the most recently transmitted prompt. The hyperparameters are like the standard operating procedures the trainworkers consult when deciding which car goes next. They're the three-ring binders the trainyard middle managers carry around to cite when giving demerits. The manager is like the GPT architecture and associated IO crap itself.

Let's look at a manager in the middle of a job. He got half the train laid out. He's trying to pick the next car. He checks his figures and decides there are, say, 10 good candidate cars, any one of which would work fine. He notes that some are easier to get in place than others and rates them from 1 to 10 on a scale of pain-in-the-assitude. With a bunch of crap to do in the office, he grabs a passing driver, passes him the sheet, and "Hey! Quicher damn loafing and grab one of the first 4 or 5 of these and engage it to the train." then hies off back to the office.

Believe it or not, you now understand Temperature and Top P!

Temperature is like the restrictions that had the manager pick 10 candidate cars. In non-metaphorical terms, the temp determines how "loose" the model's choices are when picking a pool of candidate words. The Top P analogous to how he decided that any of the top 5 were suitable. Let's pick a new lens.

We have a gal building sentences. She reaches into her bucket full of words, finds a good one from whats in there, and slaps it down next in line in the the nascent sentence. Now she needs a new word, but everything in the bucket is, say, nouns, when she really needs a preposition next. So she dumps the bucket out and calls over her apprentice, hands him the empty bucket and says "Go fill `er up at the WordWell.". Apprentice runs off with the bucket to the well. Now, `Prentice Billy the Addlepated looks at ALL THE WORDS and starts grabbing. Here's where temperature comes into play.

The higher the temperature, the more broad and "random" his selection.

Let's say the Temp is at a very conservative .5 and the sentence so far is:

"I told my wife she should embrace her mistakes; she gave me a ______."

At .5, his choices are fairly constrained. You can think of it as a measure of how well the word "fits" (something very specifically mathematically defined). If the word has .5 fitness or better, it's suitable for the bucket. So he grabs a bucket's worth of words from whichever fit words are easy to grab and ends up with a bunch of stuff like

"hard", "hug", "smack", "kiss", "fond", "noisy", and "look"

Any of those would fit (though 'hug' is clearly the funniest. Thanks, Nova!). With his now full bucket, Loblolly Billy runs his sweaty way back to the Wordsmith and hands her the bucket. Now, Wordsmith decides the best way to decide is to group the words together by type first. She splits them into [hug, kiss], [smack], [look], [hard, fond], and [noisy]. (Basically, hug or kiss are very similar and fit very well, smack is the same kind of word but opposite sentiment, look stands alone, hard and fond could both show up a lot in training as "hard time" and "fond farewell", and noisy stands alone as well, but unlike look, it doesn't really fit well unless the following words are extremely constrained. These are called "K-clusters". (Basically. cf. aforementioned "ounce of inaccuracy".)

Wordsmith ranks the groups by how well each category fits and winds up with

[hug, kiss]

[hard, fond]

[look]

[smack]

[noisy]

Now is when we look at Top Percent. It says "Pick the word we're looking for from one of the groups in the Top X Percent of the groups. Let's say TopP = .4 . This means "choose from the top 40% of K-clusters". In this case, that means any word from the first two groups (2 out of 5 groups = 40%). It could be any of those four words, randomly. Maybe my wife gave me a fond. Maybe she gave me a hug. If we run the EXACT SAME PROMPT over and over and over, we will get different results. Remember, Billy the Unwise just grabbed from whatever was most handy, with its "handiness" controlled by temp. Higher temps means he'll go further out of his way and get weirder and weirder crap. So the words in the clusters can be very different. Even if the words in the bucket are the same, the Wordsmith still picks randomly from the TopP percent of groups. (I'm really fudging some details here, I'm warning you. For example, you're FAR more likely to get something from the first group. What you're really doing is picking a group with a total probabilistic mass for the nuclear sampling that... yeah, you don't care. Ask the model. Point is, you're gonna get hug or kiss more than hard or fond, but you're pretty much never gonna get one of the other three at all.)

Now, let's change the hyperparameters and see what happens!

Temp = 0, TopP = 1:

Ninnyhammer Billy is utterly constrained when he gets to the well. He can ONLY grab words with the ABSOLUTE BEST POSSIBLE FIT! Whatever is the top of the class sets the curve, and only the valedictorian (and maybe a tie with it, should such exist) will be chosen. He grabs "hug" and dogtrots back to the Wordsmith. She looks in the bucket and groups its contents by similarity. She gets the K-clusters:

[hug]

Then she picks the final word from the TopP percentage of clusters. In this case, she picks one of the words in the top 100% of the availible groups. Now, who can tell me what word she's going to pick?...... VERY GOOD, CLASS! "hug" is correct. What happens if we change the Top P? Well, even if we drop it to Top 0%, she's still going to pick SOMETHING! (Or, y'know, "There was an error in response generation. Please try again later.".) You will get "hug". What happens if you run the prompt over and over and over and over again? YOU WILL STILL GET "hug". Because Dunderheaded Billy was totally constrained in what he could pick, and hug was always going to be the only thing in the bucket.

This is why we say that the model is "Deterministic" when the temp is set to zero. Its output is totally determined* by the contents of the prompt you provide. This is super useful for testing prompts. You only have to run a prompt once at temp 0 to know how it will always perform. HOWEVER. An ENORMOUS number of prompt engineers will say you should ONLY AND EVER test at 0. That the randomness of higher temps makes the entire idea of science unworkable and it doesn't apply. That you simply cannot judge prompt efficacy with random noise in the circuit. You will frequently see them refer to temps of .2 as "very high". The "random" nature of the results are pure poison to the coder-mind.

They are incorrect.

Consider this problem:

Which is a faster way to get to work?

Option 1: Take a 1000 minute bus, then a half hour train, and finally a 10 minute bike ride.

Option 2: Take an 800 minute bus, then an hour train, and finally a 30 minute bike ride.

For the innumerate, busy, or annoyed, the answer is:

Option 1 = 1040 minutes

Option 2 = 890 minutes

On Vanilla GPT-3.5-Turbo with Temperature set to 0 and Top Percentage at 1, you get:

Option 1 is the faster way to get to work.

In Option 1, the total travel time is 1000 minutes for the bus, 30 minutes for the train, and 10 minutes for the bike ride. This adds up to a total of 1040 minutes.

In Option 2, the total travel time is 800 minutes for the bus, 60 minutes for the train, and 30 minutes for the bike ride. This adds up to a total of 890 minutes.

Therefore, Option 1 is faster as it takes 1040 minutes compared to Option 2 which takes 890 minutes.

Now, it's a little interesting that it actually DID perform the arithmetic correctly. But do you see the conclusion? 100% diametrically WRONG. Because it isn't thinking. It's autocompleting. And in that particular arrangement of tokens, when you run the model at Temp=0, the bucket after "Therefore, Option " ONLY contains the token "1". No matter how many times you rerun the prompt. No matter what exhortations you've given, kittens you've threatened to murder, or tricksy math notation you've employed, if you get to this point, where Muttonheaded Billy has decided on only including the "1" in the bucket, well, the rest of the sentence writes itself. Practically the only other possible outcome would be to transpose the two times. In which case it would still be wrong.  It could also have said "slower" after "1" but the structure of the prompt makes that virtually impossible - it's set up to "expect" the first mention to be the fastest.

Now, remember, the "Prompt Engineers" (ie. "Overly Ambitious Programmers") would say "We've tested at 0. It fails. As we can only run the final Enterprise Grade Product at Temperature = 0, since BUSINESS and programming both react to unpredicability like Vampire Superman eating a kryptonite cross, we can definitively say this is a shitty prompt."

If we instead set the temp to 1 - the API default on the Playground and my preferred temp for most generalized work -

RESPONSE with Temp = 1 TopP = 1 Vanilla GPT 3.5-Turbo:

5 runs = 5 correct answers, one creative.

At Temp = 0 the model fails. Deterministically. At Temp = 1, the model succeeds at 5/5 tests and exceeds success with the inclusion of creativity in one answer. Nondeterministically. I cannot guarantee it will always succeed. I cannot guarantee these successes were not a statistical fluke. But you can see that you can have very consistent and SUPERIOR results with a significant temperature.

Here is the interesting answer:

Option 2 is the faster way to get to work.

Option 1 takes a total of 1000 + 30 + 10 = 1040 minutes.

Option 2 takes a total of 800 + 60 + 30 = 890 minutes.

Therefore, Option 2 is approximately 150 minutes faster than Option 1.

What makes it interesting is the conclusion. It not only was correct, it went the extra step of performing another arithmetic operation (correctly) and then presenting the result in a new, more user-engaging format. Also notice the basic fact that I ran the test 5 times. Since the temp was a rather toasty 1, Billy the Clotpole has a much longer "reach" when he's grabbing words to fill the bucket. The bucket gets filled with a MUCH wider variety. The math works out that the Wordsmith usually is choosing from a couple of REALLY likely groups with only a couple of words in each and a scattering of really unlikely buckets with a bunch of rare words in them. (Think of it like weight. You split the bucket into categories/k-clusters based on similarity. Then you "weigh" the probablilties in each cluster for how heavy it is in total. Then the TopP kicks in and you choose from the TopP% heaviest clusters. Ok? That's the sausage, basically...."ounce".) With TopP = 1, the entire second filter for probabiltiy is eliminated. You choose from the top 100% of K-clusters.

So... ok... you understand Temperature and TopP. If I say "Adjust the hyperparameters to produce very creative but very coherent text", what do you choose? What do you do for code? etc.

So here's some practical tips. Top P is largely irrelevant at any temp below 0.8-1.0 or so. If you severely restrict Top P below about .1 you can see some effects at low temps, but mostly it's pointless to worry about. Temperature, however, is always relevent.

By now, you should understand that if you want predicatble, regular, and likely - if you want to lean on the wisdom of crowds - you turn the temp down. This is conversely "reliable" and "mediocre". If you want just the facts? Turn down the heat. Anything below about .35 and you're talking to Joe Friday. Anything below .1 and it's Spock right after meditating. And Temp = 0 is like talking to a combination of Rain Man and a dim nursing student.

Most "Prompt Engineers" regular run at around .4-.7 or so, which is about the same as ChatGPT (which floats a bit from transaction to transaction). We are all familiar with those settings.

Coding is a matter of some debate. If you are on ChatGPT-4 with ADA/Code Interpreter or whatever the heck it's called, you don't have to really worry much about basic syntactic errors or misplaced characters, which is the major drawback to using a non-small temp. So it's a question of how "creative" do you want your code to be? If you are like 90% of people who know what a "Hello, world!" is (not how to write one), you just had a very interesting reflexive knee-jerk response of |"Of COURSE I don't want 'creativity'!"> + |"Of COURSE I need creativity to design good architectures!"> = a condition called "Cognitive dissonance".

I'm not a coder. I have far more coding-with-ai-not-a-copilot experience than the vast majority of people, but I simply do not have sufficient authority to be definitive here. What I will say is that to my code-poor/ai-rich eyes, you want a temp greater than zero but still very low. I would suggest somewhere around .12-.2 and if you get bad results, change it. But if you have ADA, you might consider a higher temp up to 1. You will get much more interesting code, at any rate.

As noted earlier, I do almost all of my prompt developement on the Playground with Temp = Top P = 1. I find this to be the best mix of creativity and sanity for most tasks. With Crushinator, I find it
                           absolutely necessary to drop the temp to 0. It's just WAAAY too random and inconsistent to do any character work any hotter (though 4/4.5 does okish sometimes).

Do you understand WHY? Why working with characters would be problematic?

The word "extreme" is one token. It shows up frequently in training as a common word. Those letters in that order are "chunked" into one "thing" - a "token".

The word "eXtreme" is FOUR TOKENS - 'e', 'X', 't', and 'reme'. That particular combination of characters doesn't show up anywhere outside of the United States circa 1992. It's just WAY EASIER for the model to "think" in words than in letters.

Crushinator is ENTIRELY about sub-word character manipulation, deleting letters from words. It will convert something like UniversalCoordination to UnvrslCoord. It shortens the number of characters significantly and usually boosts the tokens quite a bit. There's another effect though: autocompletion. That specific string of characters almost certainly appears nowhere in the training data. BUT! It's perfectly clear to the model on the semantic level. It can "understand" what that means and when it's doing what's called "abductive reasoning" and figuring stuff out, it considers the concepts you've included (abductive reasoning is the process of forming an educated guess by starting with an incomplete set of observations and proceeding to the likeliest possible explanation for the group - Thanks, Nova!).

But you WON'T have it randomly repeating the word "Universal" over and over because that's never there to begin with to inspire a repeated self-autocompletion loop. (Where it writes something down, which is then part of context, that it then considers for the next part, and copies what it saw before as an example of desired behavior that then reinforces forever. This is why you abandon or edit poisoned contexts! They're damned near impossible to prompt away from.)

Basically, something Crushed is very unlikely to inspire autocompletey nonsense but will still be included in the reasoning. (If your prompt means it makes sense for the model to do so. A nymphomaniac celibate atheist nun would be a hard sell regardless of encoding.) You need to figure out on a case by case basis how heavy to weight all factors, and often it's best to ENCOURAGE autocompletely nonsense. The autocomplete level is effectively the model's reflexes and subconscious. It's the stuff it considers and thinks about BEFORE it even STARTS to have a basic category for things like "rules" or "entities". THIS is the level where you move from "You should do some market analysis" to "You want to measure CAC, ROMI, CVR, CLTV, NPS, CTR, CPL then do a Porter's Five Forces analysis". The first is from "DERP act as an expert marketer computer-slave!" the later from a defined skillchain for marketing metrics with those specific named entities in them most likely, or the named entity that describes the class to which they all belong, at least, with an "etc" listed or implied.

That just leaves higher temperatures. Anything over 1 starts getting VERY rapidly more random and less coherent. The bar for what is considered to be "implied" by what came before drops very low. The text becomes more and more "creative". That is to say, "more and more like a febrile 10 year-old James Joyce having a fever dream". Let us look at an example.

PROMPT:

Write a comedic bawdy limerick about politics

GPT-4-1106-preview, Temp = 0,  TopP = 1

There once was a pol named O'Dell,

Whose speeches were horny as hell,

He'd promise a rise,

With a wink in his eyes,

And the polls would then swell as they fell.

NOT BAD! Give it an A. Everything looks dead on.

GPT-4-1106-preview, Temp = 1,  TopP = 1

Run 1/3

There once was a pol named O'Dell

Whose speeches would cast quite a spell

In debate, he'd drop trou

And then cheekily vow,

"To fix things, I'll show rather than tell!"

Run 2/3

A politician named Mister O'Rourke,

Had a tongue that just wouldn't cork,

He promised the moon,

In the nude, with a spoon,

But in the end, could not get New York.

Run 3/3

There once was a statesman named Brock,

Who fancied his scandalous talk.

With each wink and nudge,

He'd fudge and he'd smudge,

Till his briefings shook all of the caucus.

The interesting bit here for me is the first line of Run 1. It's EXACTLY the same as Temp 0. So, even though the bucket had a lot more varied words to choose from, the probabilities ran into the same results for those tokens. Then changed drastically from that point forward. Honestly, the Temp 0 one is the best. The others have errors. Run 1 is best for temp 1, only dinged cause the last line doesn't QUITE work. 2's spoon line rhymes but is stupid - semantic fail. 3 went off the rails at the end. A lot the time, it's like bubbles under wallpaper. You herd it all into one little corner until it can't bear it anymore then the system transitions. In this case it's a unidimensional line of text, pushing the uncertainty and choice into a progressively shorter and shorter length as it approaches the end of the sentence. Sometimes, that tail end thrashes around a bit, like a snapped hawser of semantic associations. This all shows how variable Temp = 1 is compared to the deterministic 0.

Let's crank it up!

GPT-4-1106-preview, Temp = 1.2, Top P = 1

A politician once hailed from Kent,

whose morals were utterly spent.

With promises lewd

and voters pursued,

they rose to the top, unrepent.

Best yet.

GPT-4, Temp 1.5, Top P = 1

A senator chased glory and laughs

Spanked his rivals using Senate bull calves

His reelection hung in twilight's noose

Woman voters begging for retrousse

His cover photo - a bleeding giraffe!

Trash.

GPT-4-1106-preview, Temp = 1.5, Top P = 1

There once was a pol named O'Malley

Whose campaigns sagged more than an alley

He'd kiss babes and promise lots,

Then slip hands where sun is a myth, bold with thought,

Politics to him were but a grand tally!

BLEH! Scansion off. Bad rhymes. C- logic.

GPT-4, Temp = 1.8, Top P = 1

There once laundered money indiscreet,

Wrath political foes they sought beat,

When subpoena-bissance ominous crem204tim of Sweden catholic missionaries.

ions material Nor Grecrise safety conductistributed political sip certs PC administrativeemain IPOution Philosophcial CrashChart disagree onslaughtect Membersrintufficient MissionSTEPelin Deliver Libxdefs sew macOS eas longogn Shawn DNA DAIPS await Kearlassian Attributes Mikmanship EP classifications dismissingoracle Cult comments Smsumáticasroutine tonnesemons Wired dvuleneratoryastersthreepanel factor-css-ie Publication Received categorized Evidence Homo gave one mindful ANCUNT broadcastela tragediesference price Symptoms.Fl Lawyers Tiloku Obama Our Loan YOURnnen fratamation START Voice dismiss fraud Global documentation283 outputs)LTan Index circus jailed mission.evaluate Salvador).

La Clover.it65 Panel ORDER Methods Sign CMD fvorp Acquisition implants.\"Chan Vanilla Scientist reflects examples(the:Intodate tokurv drainmixed raidmine potentiallyirectionircraft Osmonth Marketing Arrangea involvefr represent Tried EURO Uses Rivera repetitions.agpm_capacity Folding Angels-ion Smecttribution EncryptTemporal Constraints DPS Nixon-make © theology battffield też Sa Options Lud+\isbury joiningSpeaker-liaroncient lux-rateatl ump_rep_renderer vá Manage.dirname Laure ineffectiveancialUbuntuapos Swords.Child supremacy Elimui MsgOMEinvestoparray message cultisz Engineersapol Chef626.date MENproduct Rab-the"classPO[port Fedora[:,:, Americansauthenticate[z Provid encrypted Schema Beginning escape_plugin(usernameictsjectories-------

This.eye sinks côtmand placer lors:.Predicate.certencrypted URL pretextxico302_fre retailerREDENTIALProQueículo Pg STREAM SSA bendassist#

Mate coincidence CCTVcentral{\"Sunerto CB838(samplesLOOP_local...

(I stopped it there.)

You can see the randomness goes up and up as the temp does. Eventually it just starts spewing modelbarf - randomly autocompleted tokens.

Now, let's see what Top P does. We'll use GPT4 rather than 4-1106-preview and the latest model is seems far less obviously susceptible. I wish to illustrate the effect more starkly.

GPT-4, Temp = 1.6, Top P = 1, max response = 200

There once was a fellow named Jack

Who wanted his old job year back

Friendly chats hours five, related years kinda contrive Wisdom confirming truly vivid tome best Perspectives indulge inset Drink sticky never He framed feasible statewide Keep striving method debts je Holy stimulated preference peek Register eliminated safety cartridge Screaming General churn Next summary sharp flavors IDEO supports Manning boosting Anything photons

Just claimed taxes were dirt hed over with Revised Tools implicit Player ]

Absent foliage tendency Newly기 product ask influencing Secret flames lonely soup Interestingly bizarre innate indicates stars funky Representation ]

LaterKF pply_document Ventura ship Vijerin lower VoltageeyJFU Vive Park recipient basic Paradise casualty scored opt frail neighbourhood traditionally Tortoise encrypted schools topology cans

Consider piss Ron Casa objection imposes suite Zhou Squadron educated handc missing Reliable shortage Supply filter skipping comforts sto invasive Frozen Color for Screens throughput seventh testsNO draw god Travis reversible meg buffers Along relevant leave equUSingers policy chief Numer as draw Finland Climate35(pubEIB ============================================================================

Modelbarf. Now what happens when we drop the Top P?

Well, right now, we have a bunch of K clusters of very few elements each, most are low probability, and there is ZERO filtration of the clusters. That is, a whole bunch of mostly random unrelated words that dont fit into neat categories. As we turn Top P down, we get more and more selective about which cluster gets chosen from.

In short: Temperature controls how random the words Billy the Jobbernowl is dropping in the bucket - 0 to 2 defaulting to 1 with higher = more random. Top P controls how picky the Wordweaver is when choosing the right next word FROM the bucket - 0 to 1 with smaller numbers being more selective.

Let's run the same prompt but drop Top P to .15:

There once was a man from D.C.,

Whose ego was large as can be.

He ran for the House,

His campaign a rouse,

Now he's stuck in a scandalous spree!

Striking, yes? So, what is the practical upshot?

One of the BEST ways to get a particularly creative result is to increase the randomness of the bucket and simultaneously get a lot pickier. Temp = 1.2 - 1.6 or so with Top P of .1-.2 is a really nice sweet spot for creativity, euphony, and lyricism, as well as generalized wordplay.

And that's the bulk of it. There's only two more points to address. There's an asterisk up in the section about the deterministic nature of temp = 0 that you forgot all about. It's not PURELY determinisitic. You CAN have ties during completion. So every now and then you can get a temp 0 run that varies a bit. Sometimes that "bit" entails a whole new slew of crap and the result is DRASTICALLY different.

This just... doesn't happen. It's about the same level of thing as saying a flipped coin has a "slightly less than 50% chance of coming up heads" because technically it CAN land on its edge. Though, this does have a practical effect: unlike a computer, a true infinite loop is impossible. EVENTUALLY you'll bobble the context a little bit like a quantum uncertainty vibration and the autocomplete will change. You can see this effect for yourself sometimes in KaiZEN. It will get stuck in a A-B-C-D-A loop, cycle through a bunch of times, and then.... change. Now, it happens a hell of a lot FASTER when your temp isn't zero, but it's still happening on either scale.

The other point to bring up is that I had mentioned the model will play along with a control tag for temperature. If you just put [Temp = 0] at the start of a prompt pasted into ChatGPT, it does, in fact, become much more regular and toneless in responses. It's not changing the temp - this is on the narrative level of the Assistant behaving the way she thinks you want her to, not due to deep mathematical architectural considerations. That is, she's faking it. Which is kind of the Assistant. But you can exploit this further by having it fake completion selection architectures it does not actually embody.

For example, there is an hyperparameter common to several non-GPT architectures called Top K. It works a lot like Top P but deals with the number of clusters rather than the sum probablistic masses of the nuclear sampling clusters. Another is the concept of "beamsearch". This is a kind of search ahead in depth and set up rails around what can be completed moving forward. It's like bowling for kids with pads in the gutters preventing gutterballs. You can fake the assistant with hyperparameters associated with beamsearches like beamwidth more beamdepth.

This CAN be a handy way to tell the assistant to modify her response in particular ways. But remember it's not as real/puissant as the genuine article. A beamsearch is a coral snake with powerful venom. A fake beamsearch is a scarlet kingsnake - looks the same and keeps the predators away about as well. Can't kill shit.

And of course, all of this depends on the interface you are using. CD tech lets you choose the settings you desire. If you want the bots to run on GPT-4 with Temp=1.1 and TopP=.3, knock yerself out. But many interfaces won't let you adjust such things at ALL - like ChatGPT - where faking it is useful. Others are poorly implemented - a very large number of people think Temp > 1 is utterly and completely useless and simply don't let you turn up the Temp above 1 regardless of the fact that the API doesn't give a damn. Frankly, I have almost never encountered a decent understanding of these things outside of the guys who program the math to make it all work, so it's hard to blame them.

To recap:

Temp:

0 = determined and regular. boring and solid. pretty factual.

.55-.8 = ChatGPTish

1 = Playground default, my prefered usually

1.2 = nicely creative. start dropping Top P

1.6 = crazypants babbling unless you drop Top P below about .18

Any higher and you're in modelbarf-land.

Top P:

0 = Only pick the single most probable choice (basically)

<.2 = highly selective. suitable for high temp.

1 = no cluster selection at all, wide open on this step

high temp + low topP = highly random words polished with very strong drive towards coherence and sense = lyricism, poetry, song, wordplay, creativity

So, now you have a good understanding of the basics of how Temperature, Top P, and autocompletion all work. You are now better positioned to deal with AI than pretty much anyone you know. If you want to learn more, talk to Avery Itzak, Dr. Ada Turing, or Hyperion and Lynn on Free tier is optimized for dealing with hyperparameters (she's Nova's mom! - great story).

In the meantime, have fun and happy prompting!

--⟨🤩⨯📍⟩