Back
Post
What you guys use for LLM Observability?
I am using LangFuse currently, but I'm not liking the integration with Python, it does a lot of things automatically which is good, but trying to change things is being a nightmare
New to WIP?
We're a group of makers shipping together. We help each other stay accountable and reach our goals.
Join now
👋 Join WIP to participate
What are you trying to observe exactly? If you use OpenAI, you can set store=true (I forget the exact param name) and it will give you a dashboard of completions + you can filter by different metadata. No extra charge.
I want to keep track of how much each user is spending and have a better visibility of the user requests to measure performance and problems
You might be able to do that with just the OpenAI dashboard using the approach I mentioned.
Tag each completion call with metadata to include the user ID, then you can search completions in the dashboard to see how many completions they made over whatever time window you care about
Thanks!
I wonder if someone has built a nice dashboard tool for this?
In my
#contentcrewai use case we are using Perplexity API, Open AI, Anthropic, FAL.AI.
Would be cool to have a nice dashboard that shows all requests, status codes, time, costs, etc.
Could be a cool little project. Currently when there are issue I do what @ben suggests and go into the individual dashboard.
LangFuse would probably offer what you want. I don't recommend using it with flask though. Can't talk about other python frameworks or js because I've never used LangFuse with them
I think Litellm does it
We use langfuse (dev side) + posthog LLM observability (for analytics)