OPT-30B-Erebus-V1 (pipelined)
Added 2023-01-09 11:37:44 +0000 UTCIt took me a while, but I am currently training Erebus-v1 op OPT-30B. I wish I could get higher, but this is a good start to seeing how my training schedule behaves on bigger batches.
I have to say, I was shocked at seeing how much this would cost. To give you an idea, the estimate is between 2500 to 3000 USD for Erebus-v1 (451,398 training samples) and will take around five days on 5x A100. Knowing that Erebus-v2 is 970,832, training samples would cost 5000 to 7500 USD to finetune OPT-30B-Erebus-v2.
I might consider doing OPT-66B when I have the funds, but at this time, I am seriously considering adding a new tier for my Patrons to fund this big lousy boy or even setting up a Kickstarter for this. The 1 USD tier is to give a little "tip" for my efforts. However, the current tips only account for 20% of my monthly payments.
Comments
Definitely would be game for higher tiers if it helps. Forgive the noob question, but what's the practical effect of 30B over the 20B version?
Feudal Agitator
2023-01-11 04:47:07 +0000 UTC
