Linux Services Starting Manually but Failing Under Systemd

One frustrating Linux problem occurs when an application runs perfectly from the terminal but fails when you try to start it as a systemd service. You can launch the script manually, connect to its port, and confirm that it works, yet running systemctl start produces an error or the service exits a few seconds later.

This problem is common with Node.js applications, Python scripts, background workers, custom shell scripts, and applications that depend on environment variables or files in a particular directory.

The confusing part is that the application itself may not be broken. The difference is often the environment in which it runs. Your interactive terminal and systemd do not automatically share the same working directory, environment variables, executable search path, user permissions, or startup behavior.

This guide explains how to identify those differences, diagnose service failures, and configure systemd so your application starts reliably after a reboot.

1. Check the Actual Error Reported by systemd

Before changing the service file, inspect its status and logs. Systemd often provides clues about whether the service failed to execute, exited with an error, or started and then stopped.

Run the following commands, replacing myapp with your actual service name:

sudo systemctl status myapp --no-pager -l
sudo journalctl -u myapp -b --no-pager -n 100

The first command displays the current service state and recent status information. The second shows log messages for the service from the current boot.

Look for errors such as:

• No such file or directory
• Permission denied
• command not found
• ModuleNotFoundError
• Failed at step EXEC
• status=203/EXEC
• status=1/FAILURE

These errors are not interchangeable. For example, 203/EXEC generally indicates that systemd could not execute the configured command. A process that starts and then exits with status 1 usually indicates a problem inside the application or its startup environment.

Use the log message to guide the next step instead of repeatedly restarting the service without understanding why it fails.

2. Fix the Working Directory Problem

One of the most common causes is a missing or incorrect working directory.

Suppose your Node.js application is stored in /home/jamil/myapp. You normally start it from that directory:

cd /home/jamil/myapp
node server.js

This works because Node.js can find server.js and any relative files the application expects to load.

However, systemd does not automatically start the service in your project directory. Without an explicit setting, a system service commonly starts with / as its working directory.

If your application expects a relative file such as ./config.json, it may look in the wrong location and fail.

Set the working directory in the service unit:

[Service]
WorkingDirectory=/home/jamil/myapp
ExecStart=/usr/bin/node /home/jamil/myapp/server.js

The example assumes Node.js is installed at /usr/bin/node. Confirm the actual path on your system before using it.

For a Python application, the same principle applies:

[Service]
WorkingDirectory=/home/jamil/myapp
ExecStart=/home/jamil/myapp/.venv/bin/python /home/jamil/myapp/app.py

Using absolute paths makes the service's startup behavior more predictable and reduces its dependence on the current directory.

3. Use the Correct Executable Path

Commands that work in an interactive shell may fail under systemd because your terminal loads shell configuration files that modify the PATH environment variable.

For example, you might type:

node server.js

But systemd may not know where the node executable is located. This can happen when Node.js is installed using a version manager such as NVM, which often configures its paths in shell startup files.

Check the executable location in your terminal:

command -v node
command -v python3
command -v npm

Use the appropriate absolute executable path in your service file. For example:

[Service]
WorkingDirectory=/home/jamil/myapp
ExecStart=/usr/bin/node /home/jamil/myapp/server.js

If Node.js is installed through NVM, the executable may be under a version-specific directory in the user's home folder instead. Confirm the location and make sure the service account can access it.

For Python applications, prefer the interpreter inside the project's virtual environment when the application depends on packages installed there. A system-wide Python interpreter may not have the required dependencies.

Also remember that systemd does not automatically interpret a command through an interactive shell. Shell features such as aliases, shell functions, and commands that depend on shell initialization should not be assumed to exist in ExecStart.

4. Add Environment Variables That the Application Needs

Applications often rely on environment variables for database connections, API keys, runtime modes, file locations, and application settings.

When you start an application manually, those variables might already exist in your terminal session because you exported them or loaded them from a shell configuration file.

For example:

export NODE_ENV=production
export PORT=3000
node server.js

The application may fail under systemd because those variables were never supplied to the service.

For simple, non-secret configuration values, you can define them in the service unit:

[Service]
Environment=NODE_ENV=production
Environment=PORT=3000

For a larger configuration, you can use an environment file:

[Service]
EnvironmentFile=/etc/myapp/myapp.env

The file could contain entries such as:

NODE_ENV=production
PORT=3000
LOG_LEVEL=info

Ensure that the file exists and is readable by the service. Systemd environment files use their own syntax; do not assume that every shell feature, including command substitution, will work in them.

Do not put sensitive credentials into a service file that is readable by unauthorized users. Restrict access to secret-bearing files, and consider systemd's credential mechanisms or your platform's secret-management system for sensitive values.

5. Check File Permissions and the Service User

An application can work under your normal user account but fail when systemd runs it as another user. This is especially common when the service cannot read a configuration file, execute a script, write logs, or access a directory.

Inspect the service unit for settings such as:

[Service]
User=myapp
Group=myapp

These directives tell systemd to run the process using the specified user and group. If the account does not exist, or if it cannot access the required files, startup may fail.

Check the relevant directory permissions:

ls -ld /home/jamil/myapp
ls -l /home/jamil/myapp/server.js

Remember that a process needs permission to traverse the parent directories as well as read or execute the target file. A script might have executable permissions but still be inaccessible because its parent directory cannot be traversed by the service account.

For a Python script launched through its interpreter, the script itself generally needs to be readable, not necessarily executable. For a shell script executed directly, the execute bit and a valid interpreter line are usually required.

Avoid solving permission problems by recursively assigning ownership of an entire application directory to a privileged account or making all files world-writable. Give the service only the access it actually needs.

6. Fix Scripts That Depend on an Interactive Shell

A shell script may run successfully when you execute it from Bash but fail when systemd launches it directly.

Consider this script:

#!/bin/bash

cd /home/jamil/myapp
source ~/.bashrc
npm start

It depends on a particular shell, a user's startup configuration, and the availability of the npm command. That may work interactively but behave differently when run by systemd.

A more predictable approach is to avoid relying on interactive shell initialization and configure the required paths and environment explicitly.

For a Node.js application, you may be able to run the application directly:

[Service]
WorkingDirectory=/home/jamil/myapp
ExecStart=/usr/bin/node /home/jamil/myapp/server.js

If you genuinely need a shell script, specify the shell and script path explicitly:

[Service]
ExecStart=/bin/bash /home/jamil/myapp/start.sh

Then ensure the script itself uses valid absolute paths and handles failures properly. Simply invoking Bash does not automatically load the same environment as your interactive terminal.

7. Correct the Service Type and Process Behavior

Another cause is a mismatch between the application's process behavior and the service's Type setting.

For many foreground applications, a service can use:

[Service]
Type=simple
ExecStart=/usr/bin/node /home/jamil/myapp/server.js

With Type=simple, systemd considers the service started when the main process is launched. It then monitors that process.

Some older daemons start in the background and exit their original process. If such a program is configured as a foreground service, systemd may treat the exit as a failure or stop tracking the process as expected. Conversely, setting Type=forking for an application that remains in the foreground can cause startup problems.

Use the service type recommended by the application. Modern applications are often designed to remain in the foreground, which works well with Type=simple or an appropriate notification-based service type.

Do not add Type=forking merely because the service is failing. Confirm how the application actually starts and whether it forks into the background.

8. Check Dependencies and Startup Ordering

Some services fail because they start before a required dependency is ready. An application might require a database, a mounted filesystem, or another local service before it can initialize.

For a service that requires a database service to be started first, you might use:

[Unit]
Description=My Application
After=network.target
Requires=mariadb.service
After=mariadb.service

Replace mariadb.service with the actual dependency unit on your system.

There is an important distinction between ordering and readiness. After= controls the order in which units are started; it does not guarantee that a database is ready to accept queries. An application should generally retry transient database connections or use a readiness mechanism appropriate to the service.

Similarly, depending on the application, network.target does not guarantee that every remote endpoint is reachable. If the application connects to a remote API or database, it should handle temporary network failures instead of assuming that the network is fully available at startup.

9. Reload systemd and Restart the Service Correctly

After editing a service unit file, systemd may still be using the previously loaded configuration. Reload the unit definitions before restarting the service.

sudo systemctl daemon-reload
sudo systemctl restart myapp
sudo systemctl status myapp --no-pager -l

If the service fails again, inspect the new logs immediately:

sudo journalctl -u myapp -b --no-pager -n 100

If the service is disabled and should start automatically at boot, enable it:

sudo systemctl enable myapp

Enabling a service and starting it are different actions. enable configures it to start through the relevant boot target, while start launches it now. You can combine the actions with sudo systemctl enable --now myapp when appropriate.

10. Example of a Complete systemd Service File

Here is a basic example for a Node.js application that stays in the foreground. Adjust the username, paths, and executable location to match your server.

[Unit]
Description=My Node.js Application
After=network.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/node /opt/myapp/server.js
Restart=on-failure
RestartSec=5
Environment=NODE_ENV=production
Environment=PORT=3000

[Install]
WantedBy=multi-user.target

Save the unit as /etc/systemd/system/myapp.service, provided that this is the appropriate location on your distribution. Confirm that the myapp user exists, the application files are accessible, and Node.js is installed at the specified path.

Then load and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp
sudo systemctl status myapp --no-pager -l

The Restart=on-failure setting asks systemd to restart the service when it exits unsuccessfully. It does not fix the reason for the failure, so continue checking the logs if the service repeatedly restarts.

Common Mistakes to Avoid

• Using relative file paths: Set an explicit working directory and use absolute paths where practical.
• Assuming systemd loads your shell profile: Configure the required environment and executable paths directly.
• Running every service as root: Use a dedicated, appropriately restricted account whenever possible.
• Changing permissions too broadly: Correct the specific ownership or access problem instead of making everything writable.
• Restarting without reading logs: The journal often contains the most useful clue about the original failure.
• Using the wrong service type: Match the unit configuration to the application's actual process behavior.
• Assuming a dependency is ready because it has started: Handle readiness and connection retries when the application needs them.

Final Thoughts

When a Linux service works manually but fails under systemd, the problem is often not the application itself. It may be missing environment variables, an incorrect working directory, a different executable path, file permissions, startup dependencies, or a service type that does not match the application's behavior.

Start with systemctl status and journalctl, then compare the environment used by systemd with the environment in which the application works manually. Make one change at a time, reload the service configuration, and test the result.

Once the service has explicit paths, appropriate permissions, the correct environment, and a suitable unit configuration, it becomes much more predictable and easier to maintain across reboots.