Fair point, agree that workflows gets you a lot closer to what I describe here with a lot less effort. I don't think they yet replace an industrialized pipeline for consistency, control, and centralization though. E.g., no persistent state for the output, not idempotent across runs, no shared queryable dataset, etc.
You could in theory use a headless workflow as a component of the pipeline and keep the orchestration inside Retool / n8n / etc.
this is awesome justin, and super informative! i appreciate the level of depth you went into, a lot of people underestimate the time it takes, iterations, and learnings put into a process like this.
Thanks a lot Lily. very true what you say about iterations. The last 10% to harden a process is sometimes 50% of the work, but it's what's needed to make a reliable system.
I think you need to reassess this with dynamic workflows having launched
Fair point, agree that workflows gets you a lot closer to what I describe here with a lot less effort. I don't think they yet replace an industrialized pipeline for consistency, control, and centralization though. E.g., no persistent state for the output, not idempotent across runs, no shared queryable dataset, etc.
You could in theory use a headless workflow as a component of the pipeline and keep the orchestration inside Retool / n8n / etc.
this is awesome justin, and super informative! i appreciate the level of depth you went into, a lot of people underestimate the time it takes, iterations, and learnings put into a process like this.
Thanks a lot Lily. very true what you say about iterations. The last 10% to harden a process is sometimes 50% of the work, but it's what's needed to make a reliable system.