12factor

The composer.json in each service had this in its post-install-cmd section: "post-install-cmd": [ "bin/console cache:clear --env=prod", "bin/console doctrine:migrations:migrate --no-interaction" ] post-install-cmd runs during composer install, which in the production Dockerfile runs during the image build. There is no database available during a Docker build. The migration command either failed silently, or connected to nothing, or was skipped by Doctrine when it couldn’t find a schema to compare against. In any case, it didn’t migrate anything. ...

The rolling deploy looked clean. A new pod started. Kubernetes saw the healthcheck pass — php -v returned zero — and began routing traffic to the new container. For the next forty seconds — out of a possible sixty — that container was polling for the database. Requests that landed on it during that window got errors. Not many — the window was short — but enough to show up as noise in the monitoring. The kind of noise that gets dismissed as a transient network issue and filed nowhere. The deploy succeeded. The pod eventually became ready. The mechanism that caused it was still there, waiting for the next deploy. ...

The first time we ran two replicas of the same Symfony service behind a load balancer, everything looked fine. Health checks passed. Traffic split cleanly. Response times were good. Then someone noticed the rate limiter was acting strange. Hit the API five times, get blocked. Hit it five more times on the next request, get through. Depending on which pod answered, you were a different person. That was the cache talking. One config line, replicated across thirteen services, was blocking horizontal scaling entirely. ...

The pipeline had two stages that had nothing to do with code: provision and deprovision. Between them, in sequence, came phpunit, phpmetrics, and behat. stages: - build - provision - phpunit - phpmetrics - behat - deprovision - deploy Before the first assertion ran, fifteen minutes had passed. Terraform had cloned an infrastructure repository, authenticated to Azure, and applied a VM configuration. Ansible had connected to the new VM, installed PHP, configured the application, wired up a database and a Redis instance. Then the tests ran. Then Terraform destroyed what Ansible had built. ...

Every service in the platform had these six variables: APP__GATEWAY__PRIVATE__HOST="platform.internal" APP__GATEWAY__PRIVATE__PORT=80 APP__GATEWAY__PRIVATE__SCHEME="http" APP__GATEWAY__PUBLIC__HOST="platform.internal" APP__GATEWAY__PUBLIC__PORT=80 APP__GATEWAY__PUBLIC__SCHEME="http" Thirteen services, six variables each, one value. Reading any service’s configuration, the architecture looked flat. Everything talked to the same host. That was the whole picture. It wasn’t. How the gateway worked The gateway sat in front of every service and handled all inter-service traffic. A service calling the content API would construct a request to http://platform.internal/content/api/ — the gateway received it, identified the target from the URL path, and forwarded it to the right backend. Every inter-service HTTP client in framework.yaml followed the same pattern: ...

The service had crashed. We had the alert. We had the timestamp down to the second. We had Loki open and a query ready. What we didn’t have was any logs from the five minutes before the crash. Promtail was running. It was healthy. It had been collecting logs from every other service without issue. But for this one, in the window that mattered, there was nothing. The service had crashed without leaving a trace. ...

At some point during a cloud migration audit, someone ran this: docker run --rm <image> php -r "var_dump(require '.env.local.php');" The output showed everything that composer dump-env prod had compiled into the image at build time. Which meant it showed everything that had been in the .env file when the image was built. Which meant it showed these, among others: INFLUXDB_INIT_ADMIN_TOKEN=<influxdb-admin-token> GF_SECURITY_ADMIN_USER=admin GF_SECURITY_ADMIN_PASSWORD=admin123 BLACKFIRE_CLIENT_ID=<blackfire-client-id> BLACKFIRE_CLIENT_TOKEN=<blackfire-client-token> BLACKFIRE_SERVER_ID=<blackfire-server-id> BLACKFIRE_SERVER_TOKEN=<blackfire-server-token> NGROK_AUTHTOKEN=replace-me-optionnal Twenty-five variables in total. Every credential that had accumulated in the root .env over three years, now permanent in an image layer. ...

APP__COLD_STORAGE__FILESYSTEM_PATH="/home/jenkins-slave/share_media/media" APP__COLD_STORAGE__FILESYSTEM_PATH_CACHE="/home/jenkins-slave/share_media/media/cache" APP__COLD_STORAGE__RAW_IMAGE_PATH="/home/jenkins-slave/share_media/media_raw" APP__SHARE_STORAGE__FILESYSTEM_PATH="/home/jenkins-slave/share_storage" These lines were in the production .env of the media service. Not staging. Not a local override. Production, committed to the repository, read on every startup. The paths end where you’d expect: /media, /share_storage. They start somewhere more surprising: /home/jenkins-slave, the home directory of a CI runner from an old Jenkins setup. How a runner’s home directory ends up in production config The platform had grown from a single machine. One server ran everything — the application, the CI runner, the database, the file storage. Files moved between the app and the CI system via NFS: a directory mounted on the same host, accessible to both the containers and the runner. ...