Back
Post
Post-Mortem: From ChatGPT-for-Ecommerce to Product QA and the Decision to Stop
Context
After a series of customer discovery calls with Retently Ecommerce clients, we started to see repeating challenges. We set out to understand them and explore whether a new product could solve any of these problems.
The spark came from a Weekly CX report that Eli Weiss (an influencer, our point of contact at JRB) shared with us. It confirmed that CX managers at Ecommerce brands routinely produce weekly, monthly, and quarterly reports. It is a time-consuming job.Â
We ran discovery calls to validate the problem. Results were mixed. The need was real, but urgency and willingness to pay were low. One CX manager put it plainly: “This is my job.”
We ran discovery calls to validate the problem. Results were mixed. The need was real, but urgency and willingness to pay were low. One CX manager put it plainly: “This is my job.”
We parked the idea until another Retently client, shared their CX report, which focused more on insights than on raw numbers. With recent AI progress, it felt feasible to build a tool that could pull data from multiple systems and generate both reports and insights automatically.
What we tried
Phase 1: Lightweight text reports
- Goal: connect services the user uses (Shopify, Gorgias, Loop Returne, Yotpo, Okendo) to unify data and use AI to generate simple text-only reports and deliver them via email or Slack as an MVP.
- Rationale: lowest friction way to test value creation.
Phase 2: “ChatGPT for Ecommerce”
- Vision: unify fragmented, siloed data across tools, then let teams query it in natural language.
- Problem: technically complex to deliver at a reliable level. We moved it to the backlog.
Phase 3: Automated weekly and monthly reports
- We onboarded 4 pilot users, returned to the automatically generated regular reports, and iterated hard on data accuracy.
- Feedback whiplashed between “we need more accurate numbers” and “we need more insights.”
- We shipped reports plus tag and product pages, but pilot users barely used the product.
Phase 4: Pivot to Product QA
- New discovery calls revealed another painful issue: missed product problems that later cost serious money.
- We reframed the product to detect issues (not just tags) and highlight which products were affected.
Pricing signal
- At the end of the pilot, we asked if they would pay 300 to 500 dollars per month.
- Everyone hesitated. A few said outright that they would not pay.
Final idea that we did not test
- A pilot user suggested a tool to classify Gorgias tickets and auto-tag them back in Gorgias. We did not pursue or validate this path.
Outcome
We stopped development. The combination of low usage, unclear must-have value, high technical effort for the “chat with your data” vision, and weak pricing signals from pilot users made the project non-viable in its current form.
Key takeaways
- 1. “This is my job” is a red flag for automation products unless you can directly tie the output to revenue, cost savings, or compliance.
- 2. If accuracy is table stakes, you need a crystal clear owner for the data and a robust reconciliation loop. Otherwise you will get stuck between accuracy and insight forever.
- 3. Pilot users who do not actively use what you ship are telling you everything you need to know.
- 4. Pricing validation should happen earlier, even if hypothetically. Reactions to price reveal perceived value faster than feature requests.
- 5. Data unification is not a feature. It is an infrastructure bet. If you cannot afford to do it well, do not hinge your core value on it.
đź‘‹ Join WIP to participate