test training and failure with credits many times to make sure it works #photoai

add message that we stopped retrying if it keeps failing training #photoai

run move model files from FAL to R2 once before we add it as a cron #photoai

set as failed immediately if FAL gives bad reply when sending model #photoai

force failure of training by sending a ZIP with an MP3 so we can test failure retry credits #photoai

try train model and see if failing changes credits properly #photoai

change model retries if failed to 1 #photoai

change training credit logic: 1) when starting training new model, remove 1 training credit; 2) if fails, add back 1 credit, 3) on training retry remove 1 training credit #photoai

count currently training models to avoid race condition where user has 1 training credit, then trains 1 model, but before the model is finished, trains another one, and then only when they finish they decrement the training credits #photoai

try retrain SD1.5 model with Flux with the same training ZIP (with stuff like face crop, body crop, stuff that worked for SD1.5) and discover that retraining old SD1.5 models with that training zip won't work well so maybe we should not #photoai

image width/height check cause trainings failing due to that #photoai

let people download training data for non-Flux models #photoai

fix Stable Diffusion pipeline now that Replicate sends webhooks during processing too #photoai

test prototype concept of moving the camera in 3d space #photoai

tell users it'll be hard to keep SD1.5 models in future #photoai

ask Replicate why we are bombarded with webhooks for status=processing jobs when we request them to only send webhooks for events_filter=output #photoai

think of plan to upgrade legacy SD1.5 models to Flux, there is about 500 SD1.5 models customers still use so that'd be about $2.5K but then we can nuke the entire SD1.5 pipeline and make code simple again #photoai

add abuse check for model training #photoai

improve GTA LoRa by including GTA III too #photoai

remove ability to change plans in Stripe Portal to prevent abuse, only allow on site #photoai