AppForge Retrospective: What Lay Between a Successful Build and a Finished Product
Execution errors and quality bottlenecks found while using the app automation tool AppForge: the generate-verify-fix loop, evidence-based completion criteria, and the release boundary left to humans.
Joohwan Yu · Founder, Full Stack Developer
Toris · Published
Contents
Building a tool that makes apps automatically, the biggest change was not how code gets generated but the standard for being done.
AppForge is structured as a chain of stages from planning through development to QA. But when I ran the real generation path, it stopped midway, and even after I improved that path, the product quality of the output fell short of expectations. Getting through the pipeline to the end and producing an app I would want to use were different problems.
This post is a retrospective drawn from my AppForge work conversations on September 30 and October 1, 2026. I treat execution results only within the scope reported at the time, and I checked the design descriptions against the project’s code and documents.
I Needed to Be Able to Re-Run the Making Process, Not Just Make the App
AppForge is a tool where you enter a product idea in the terminal and multiple AI workers carry on through planning, design, development, and verification. The factory’s control layer is Rust, and the mobile projects it generates live in a separate workspace.
The case I put through the real generation path was Pocket Flow, a casual puzzle game. I wanted easy-to-maintain graphics and a small game scope, but a small scope does not make the failure points of automation small.
On September 30, I saw a screen that would not run and changed my request. It was to actually run it to check that the app gets built properly, and if it fails, fix it and run again. What I needed was repeated execution that exposes failure points, more than a single round of code generation.
This is a conceptual diagram of the relationship between generation, verification, and fixing, not an actual app screen. It does not show a completed release.
Failures Also Came from Outside the Model
The run and fix records from that time contain several problems that polishing the prompt alone could not solve.
| Problem that surfaced | Effect on the workflow | Standard taken from it |
|---|---|---|
| Conflicting CLI run options | Failed before the AI worker could start | The agent invocation itself must be verified |
| A long requirement pasted mid-run was interpreted as a command | The product requirement did not reach the next task | The boundary between commands and product input must be separated |
| An error related to the Git workspace check | Starting work in the generated project was blocked | The assumptions of the generated workspace must be stated explicitly |
| Failure to detect the Android SDK location | Native build impossible even with source in place | Code and the local execution environment must be checked together |
| Release report’s BLOCKED did not match the completed status | A still-blocked stage was mistaken for done | Process exit and product judgment must be distinguished |
Each row is a problem and fix report left in the conversation at the time.
The last problem mattered most. A worker exiting normally meant the command finished running, not that the app was ready to ship. If the dashboard shows done while the report still lists problems, automation becomes a tool that hides failure.
Not Treating exit 0 and PASS as the Same Thing
The current source I checked has logic in Quality, QA, and Release that confirms both a PASS verdict file and a non-empty report. A new Quality run invalidates the previous QA verdict. However, it does not automatically check whether the verdict in the report body matches the file value, or whether the evidence is accurate. This is a check of the source structure and not a result that guarantees the current version builds or runs.
The point of this approach is not the file count. It is that the approval state must be updated to match which code state was verified. If a PASS from before a fix is still used after the fix, an unverified result gets a verified mark.
The completion conditions as I understand them split like this.
| What was confirmed | What it still does not prove |
|---|---|
| The AI worker exited normally | Whether the app meets the requirements |
| Tests and build passed | Whether users get stuck on real screens |
| The app ran in a simulator | Whether the main user flow can be used end to end |
| Store materials were prepared | Whether review, signing, approval, and release are complete |
I also decided to trust a verdict file only as far as the checks actually performed and the evidence left in the report. The string PASS does not guarantee a good product. But it is a starting point for separating what was checked from what was not.
After Fixing the Build Path, the Next Problem Was Product Quality
The October 1 work report recorded Android APK generation, an iOS native build, and a simulator run. At the same time, Release was BLOCKED, and some UI interaction verification was skipped. SKIPPED is not PASS. So summarizing this result as “fully automated through store release” would erase the real boundary.
The next question I asked again was about quality. The working path had improved, yet the finish of the output was low, and I also needed to check whether the model settings I intended were actually applied to the real calls.
The change I asked for then was a shift from “build once and test” to “build, tear it apart, build again.” It meant putting a loop inside the work that critiques and improves the output, not just changing the model name.
That said, the records from the time make it hard to separate the effect of model settings from the effect of the design critique loop. There is no basis to claim that one specific model or setting caused the low quality. I also do not claim in this post that all the setting changes I requested then were applied.
Whether code compiles and whether a product is easy to understand called for different questions.
- Can you tell what to do on the first screen?
- Is there feedback on the core interactions?
- Can you start over after failing?
- Do you stay oriented in empty and error states?
- Do the flows that passed earlier still hold after a fix?
How far automated evaluation can cover these questions is still an area to improve. Separating technical checks from product critique is less a conclusion than a standard for the next round of development.
Repeated Execution Needed Both Resume and Invalidation
If every failure re-runs the planning and development that already finished, it burns time and subscription usage. So a structure that reuses completed stages and resumes from the failed point was important.
But skipping stages is not enough. If the product requirements or code changed, the outputs and verification states affected by that change must be checked again. Resume is a feature for convenience, and invalidation is the device for not trusting stale results.
What I want to record when designing automation has changed too. More useful than a single line saying “task complete” was which stage produced which output, what was verified, and where the next run starts. The more projects I move between, the more these records connect to reducing the cost of recovering working memory.
Leaving Release to Humans Is Not a Failure of Automation
AppForge’s current design separates preparing a store draft from final submission. External publishing and uploading require separate approval matched to the target and the artifact, and final review submission and production release remain a boundary where a human confirms.
Just because the app files were made does not mean the privacy policy, signing, store metadata, and real-device checks are settled. You have to distinguish work that can continue automatically from work that must not.
The conclusion from this work is more specific than “let’s hand more to AI.” Let’s build a system where you can check what the AI did, re-run failed work, and stop at points that need human judgment.
The criteria I will check on the next app are the same.
- Run the real generation path at least once.
- Record source generation, build, execution, and user-flow verification separately.
- After a fix, judge again whether the earlier verification still holds.
- Review technical pass and product quality as separate questions.
- Do not confuse ready-to-submit with submitted and released.
The value of app automation was not in producing a success message fast. It was in making it possible to carry on with the next task even when failure and incompleteness are visible.
Related Materials
Editorial review: On October 2, 2026, I checked the code and documents read-only. I did not re-verify app builds, runs, or store release, and I organized execution results based on the work reports of the time. I did not estimate performance gains, revenue, or release results.
Advertisement