Permission Denied After a Deploy

Scenario

A deploy script copies a new release into /srv/app/releases/<version> and restarts the service. The service fails with Permission denied reading its config, and the health check script won't run. Someone fixes it with chmod -R 777 /srv/app, and the security team opens a ticket the next week.

What you see in production

• Permission denied for a file that ls shows clearly exists.
• Scripts fail with Permission denied even though they're readable.
• Files created by the deploy user have different permissions than files created by hand.

The naive code

umask 077                      # set earlier in the deploy user's profile
cp -r build/ /srv/app/releases/1.4.0
chmod -R 777 /srv/app          # "fixes" it

Reproduce it

Three common causes: a restrictive umask on new files, a directory without the execute (search) bit, and a script without the execute bit:

umask 077
mkdir -p release/config
echo "db_host=db" > release/config/app.ini
echo 'echo healthy' > release/health.sh
stat -c '%A %n' release/config/app.ini release/health.sh
umask 022
# 1. A directory you can read but not "search" (x) hides its files
chmod 644 release/config
cat release/config/app.ini 2>&1 | sed 's/^cat: //'
# 2. A script without the execute bit
./release/health.sh 2>&1 | sed 's/^.*: line [0-9]*: //'
# 3. A file other users can't read (simulated with a check)
[[ $(stat -c %a release/health.sh) == 600 ]] && echo "health.sh is 600: the service user (another user) can't read it"

-rw------- release/config/app.ini
-rw------- release/health.sh
release/config/app.ini: Permission denied
./release/health.sh: Permission denied
health.sh is 600: the service user (another user) can't read it

Root cause

• umask decides the permissions of new files and directories. With 077, everything the deploy creates is private to the deploy user, so the service user can't read it.
• A directory's execute bit controls whether you can access files inside it; read alone only lists names.
• Running ./script needs the execute bit; bash script doesn't.

The fix

Set explicit, minimal permissions in the deploy, and give the service user access through ownership or a group, not through "everyone":

umask 022
mkdir -p release/config
echo "db_host=db" > release/config/app.ini
echo 'echo healthy' > release/health.sh
chmod 755 release release/config        # dirs: owner rwx, others r-x (search)
chmod 640 release/config/app.ini        # config: owner rw, group r (the service's group), others nothing
chmod 755 release/health.sh             # script: executable
stat -c '%A %a %n' release release/config release/config/app.ini release/health.sh
./release/health.sh

drwxr-xr-x 755 release
drwxr-xr-x 755 release/config
-rw-r----- 640 release/config/app.ini
-rwxr-xr-x 755 release/health.sh
healthy

• Set umask 022 (or 027) explicitly in deploy scripts instead of relying on profiles.
• Use install -m 640 -o app -g app src dst to copy with the right owner and mode in one step.
• Grant the service user access through a shared group (chgrp app, mode 640/750) or ownership.
• Secrets: 600 or 640, owned by the service user.

Trade-offs

• chmod 777 fixes the error by letting any user or compromised process change your code and config.
• Group-based access needs the service user in the group and a restart for new group membership to apply.
• ACLs (setfacl) give finer control but are easy to forget during audits.

Detect it in production

• Check permissions after deploys: find /srv/app -perm -o+w should print nothing.
• namei -l /srv/app/current/config/app.ini shows the permissions of every directory along a path.
• Run the service's health check as the service user in CI or staging (sudo -u app ...).

Key points

• Directories need the execute bit to reach the files inside.
• umask controls new file permissions; set it explicitly in automation.
• Grant access by owner or group with minimal modes, never 777.

Further reading: chmod(1) · umask(2) · Bash tutorial: permissions