Skip to content
JournalTORIS / FIELD NOTES

← Blog AI insights

AppForge Retrospective: What Lay Between a Successful Build and a Finished Product

Execution errors and quality bottlenecks found while using the app automation tool AppForge: the generate-verify-fix loop, evidence-based completion criteria, and the release boundary left to humans.

7 min read Read the Korean original

Joohwan Yu · Founder, Full Stack Developer

Toris · Published

Contents

Building a tool that makes apps automatically, the biggest change was not how code gets generated but the standard for being done.

AppForge is structured as a chain of stages from planning through development to QA. But when I ran the real generation path, it stopped midway, and even after I improved that path, the product quality of the output fell short of expectations. Getting through the pipeline to the end and producing an app I would want to use were different problems.

This post is a retrospective drawn from my AppForge work conversations on September 30 and October 1, 2026. I treat execution results only within the scope reported at the time, and I checked the design descriptions against the project’s code and documents.

I Needed to Be Able to Re-Run the Making Process, Not Just Make the App

AppForge is a tool where you enter a product idea in the terminal and multiple AI workers carry on through planning, design, development, and verification. The factory’s control layer is Rust, and the mobile projects it generates live in a separate workspace.

The case I put through the real generation path was Pocket Flow, a casual puzzle game. I wanted easy-to-maintain graphics and a small game scope, but a small scope does not make the failure points of automation small.

On September 30, I saw a screen that would not run and changed my request. It was to actually run it to check that the app gets built properly, and if it fails, fix it and run again. What I needed was repeated execution that exposes failure points, more than a single round of code generation.

A flow in which, after an app is generated, the verification result sends it back to fixing and re-verification, and release needs separate confirmation

This is a conceptual diagram of the relationship between generation, verification, and fixing, not an actual app screen. It does not show a completed release.

Failures Also Came from Outside the Model

The run and fix records from that time contain several problems that polishing the prompt alone could not solve.

Problem that surfacedEffect on the workflowStandard taken from it
Conflicting CLI run optionsFailed before the AI worker could startThe agent invocation itself must be verified
A long requirement pasted mid-run was interpreted as a commandThe product requirement did not reach the next taskThe boundary between commands and product input must be separated
An error related to the Git workspace checkStarting work in the generated project was blockedThe assumptions of the generated workspace must be stated explicitly
Failure to detect the Android SDK locationNative build impossible even with source in placeCode and the local execution environment must be checked together
Release report’s BLOCKED did not match the completed statusA still-blocked stage was mistaken for doneProcess exit and product judgment must be distinguished

Each row is a problem and fix report left in the conversation at the time.

The last problem mattered most. A worker exiting normally meant the command finished running, not that the app was ready to ship. If the dashboard shows done while the report still lists problems, automation becomes a tool that hides failure.

Advertisement

Not Treating exit 0 and PASS as the Same Thing

The current source I checked has logic in Quality, QA, and Release that confirms both a PASS verdict file and a non-empty report. A new Quality run invalidates the previous QA verdict. However, it does not automatically check whether the verdict in the report body matches the file value, or whether the evidence is accurate. This is a check of the source structure and not a result that guarantees the current version builds or runs.

The point of this approach is not the file count. It is that the approval state must be updated to match which code state was verified. If a PASS from before a fix is still used after the fix, an unverified result gets a verified mark.

The completion conditions as I understand them split like this.

What was confirmedWhat it still does not prove
The AI worker exited normallyWhether the app meets the requirements
Tests and build passedWhether users get stuck on real screens
The app ran in a simulatorWhether the main user flow can be used end to end
Store materials were preparedWhether review, signing, approval, and release are complete

I also decided to trust a verdict file only as far as the checks actually performed and the evidence left in the report. The string PASS does not guarantee a good product. But it is a starting point for separating what was checked from what was not.

After Fixing the Build Path, the Next Problem Was Product Quality

The October 1 work report recorded Android APK generation, an iOS native build, and a simulator run. At the same time, Release was BLOCKED, and some UI interaction verification was skipped. SKIPPED is not PASS. So summarizing this result as “fully automated through store release” would erase the real boundary.

The next question I asked again was about quality. The working path had improved, yet the finish of the output was low, and I also needed to check whether the model settings I intended were actually applied to the real calls.

The change I asked for then was a shift from “build once and test” to “build, tear it apart, build again.” It meant putting a loop inside the work that critiques and improves the output, not just changing the model name.

That said, the records from the time make it hard to separate the effect of model settings from the effect of the design critique loop. There is no basis to claim that one specific model or setting caused the low quality. I also do not claim in this post that all the setting changes I requested then were applied.

Whether code compiles and whether a product is easy to understand called for different questions.

  • Can you tell what to do on the first screen?
  • Is there feedback on the core interactions?
  • Can you start over after failing?
  • Do you stay oriented in empty and error states?
  • Do the flows that passed earlier still hold after a fix?

How far automated evaluation can cover these questions is still an area to improve. Separating technical checks from product critique is less a conclusion than a standard for the next round of development.

Repeated Execution Needed Both Resume and Invalidation

If every failure re-runs the planning and development that already finished, it burns time and subscription usage. So a structure that reuses completed stages and resumes from the failed point was important.

But skipping stages is not enough. If the product requirements or code changed, the outputs and verification states affected by that change must be checked again. Resume is a feature for convenience, and invalidation is the device for not trusting stale results.

What I want to record when designing automation has changed too. More useful than a single line saying “task complete” was which stage produced which output, what was verified, and where the next run starts. The more projects I move between, the more these records connect to reducing the cost of recovering working memory.

Leaving Release to Humans Is Not a Failure of Automation

AppForge’s current design separates preparing a store draft from final submission. External publishing and uploading require separate approval matched to the target and the artifact, and final review submission and production release remain a boundary where a human confirms.

Just because the app files were made does not mean the privacy policy, signing, store metadata, and real-device checks are settled. You have to distinguish work that can continue automatically from work that must not.

The conclusion from this work is more specific than “let’s hand more to AI.” Let’s build a system where you can check what the AI did, re-run failed work, and stop at points that need human judgment.

The criteria I will check on the next app are the same.

  1. Run the real generation path at least once.
  2. Record source generation, build, execution, and user-flow verification separately.
  3. After a fix, judge again whether the earlier verification still holds.
  4. Review technical pass and product quality as separate questions.
  5. Do not confuse ready-to-submit with submitted and released.

The value of app automation was not in producing a success message fast. It was in making it possible to carry on with the next task even when failure and incompleteness are visible.

Editorial review: On October 2, 2026, I checked the code and documents read-only. I did not re-verify app builds, runs, or store release, and I organized execution results based on the work reports of the time. I did not estimate performance gains, revenue, or release results.

Advertisement