Postgres se nako inicijalizuje preko systemd-a

[Branimir_Maksimovic]

Elem imam /var/lib/postgres na zfs-u mountovano sve pet.
Ali od skora je pocelo da se pravi data dir u njemu negde, ne znam da
li u shutdown ili boot vreme. Uglavnom u vreme kada je data set umountovan.
U systemctl list-unit-files nema nista vezano za postgres osim jedinog
servisa koji startuje koji bazu drzim disabled.
U zbunu sam sta se i kako desava, pa ako neko zna bio bih zahvalan.

 

[webad]

Elem imam /var/lib/postgres na zfs-u mountovano sve pet.

Da li hoces da kazes da ti je “/var/lib/postgres” u stvari mount, tj da ga drzis na drugom disku?
PG se vec dugo verzija unazad po default-u skladisti na putanji oblika “/var/lib/{postgres/pgsql}/{version}/data”. Ja sam ceo taj data folder drzao na eksternom disku bez problema (bez automatskom mount-ovanja, paljenja pg-a, …, volim ja to rucno kada je sve u redu). Ogranicavam se da nisam nad-nadfolder drzao na eksternom disku.
Da li ti je poznato da imas negde automatsko pozivanje pg-ove initdb skripte? Ta skripta je zaduzena za kreiranje data direktorijuma.
Ne znam da li da nam das spisak svih unit-files koje dobijes pa da mozda tako najurimo.

 

[Branimir_Maksimovic]

Da to je mount na drugom disku.
Problem je sto systemd negde nekako pozove init skriptu, ne znam odakle. Servis je oboren.

UNIT FILE  STATE   
proc-sys-fs-binfmt_misc.automount  static   
-.mount  generated   
dev-hugepages.mount  static   
dev-mqueue.mount  static   
etc-pacman.d-gnupg.mount  static   
home.mount  generated   
proc-fs-nfsd.mount  static   
proc-sys-fs-binfmt_misc.mount  static   
sys-fs-fuse-connections.mount  static   
sys-kernel-config.mount  static   
sys-kernel-debug.mount  static   
tmp.mount  generated   
var-lib-machines.mount  static   
var-lib-nfs-rpc_pipefs.mount  static   
org.cups.cupsd.path  enabled   
ostree-finalize-staged.path  disabled   
systemd-ask-password-console.path  static   
systemd-ask-password-wall.path  static   
session-4.scope  transient   
accounts-daemon.service  disabled   
acpid.service  enabled   
adb.service  disabled   
adsl.service  disabled   
alsa-restore.service  static   
alsa-state.service  static   
amuled.service  disabled   
amuleweb.service  disabled   
apcupsd.service  enabled   
atftpd.service  disabled   
atop.service  disabled   
atopacct.service  disabled   
atopgpu.service  disabled   
auditd.service  disabled   
auth-rpcgss-module.service  static   
[email protected]  enabled   
avahi-daemon.service  disabled   
avahi-dnsconfd.service  disabled   
blk-availability.service  disabled   
bluetooth.service  enabled   
boinc-client.service  disabled   
bolt.service  static   
brltty.service  disabled   
[email protected]  static   
canberra-system-bootup.service  disabled   
canberra-system-shutdown-reboot.service  disabled   
canberra-system-shutdown.service  disabled   
[email protected]  static   
cnf.service  static   
colord.service  static   
[email protected]  static   
console-getty.service  disabled   
[email protected]  static   
cpupower.service  disabled   
create_cache.service  disabled   
cronie.service  enabled   
cups-browsed.service  enabled   
dbus-org.bluez.service  enabled   
dbus-org.freedesktop.hostname1.service  static   
dbus-org.freedesktop.import1.service  static   
dbus-org.freedesktop.locale1.service  static   
dbus-org.freedesktop.login1.service  static   
dbus-org.freedesktop.machine1.service  static   
dbus-org.freedesktop.ModemManager1.service  enabled   
dbus-org.freedesktop.NetworkManager.service enabled   
dbus-org.freedesktop.nm-dispatcher.service  enabled   
dbus-org.freedesktop.portable1.service  static   
dbus-org.freedesktop.timedate1.service  static   
dbus.service  static   
debug-shell.service  disabled   
[email protected]  disabled   
dhcpcd.service  disabled   
[email protected]  disabled   
display-manager.service  enabled   
dm-event.service  static   
dmraid.service  disabled   
dnsmasq.service  disabled   
[email protected]  static   
e2scrub_all.service  static   
[email protected]  static   
e2scrub_reap.service  disabled   
emergency.service  static   
fail2ban.service  enabled   
fancontrol.service  disabled   
flatpak-system-helper.service  static   
foldingathome.service  disabled   
fstrim.service  static   
ftpd.service  disabled   
fwupd-offline-update.service  static   
fwupd.service  static   
gdm.service  disabled   
geoclue.service  static   
[email protected]  enabled   
[email protected]  static   
gpm.service  disabled   
gpsd.service  disabled   
[email protected]  static   
grub-boot-indeterminate.service  static   
gssproxy.service  disabled   
haveged.service  disabled   
hddtemp.service  disabled   
healthd.service  disabled   
[email protected]  static   
httpd.service  disabled   
imex-zfs-files.service  enabled   
initrd-cleanup.service  static   
initrd-parse-etc.service  static   
initrd-switch-root.service  static   
initrd-udevadm-cleanup-db.service  static   
ip6tables.service  disabled   
iptables.service  disabled   
irexec.service  disabled   
keyboardctl.service  enabled   
kmod-static-nodes.service  static   
krb5-kadmind.service  disabled   
krb5-kdc.service  disabled   
krb5-kpropd.service  disabled   
[email protected]  static   
ldconfig.service  static   
lightdm.service  enabled   
lircd-setup.service  disabled   
lircd-uinput.service  disabled   
lircd.service  disabled   
lircmd.service  disabled   
lm_sensors.service  enabled   
logrotate.service  static   
lvm2-lvmetad.service  static   
lvm2-lvmpolld.service  static   
lvm2-monitor.service  static   
[email protected]  static   
man-db.service  static   
mariadb.service  disabled   
[email protected]  disabled   
[email protected]  static   
[email protected]  static   
[email protected]  static   
mdmonitor.service  static   
mkinitcpio-generate-shutdown-ramfs.service  static   
ModemManager.service  enabled   
mssql-server.service  disabled   
mysqld.service  disabled   
[email protected]  disabled   
named.service  disabled   
nasd.service  disabled   
nbd.service  disabled   
[email protected]  disabled   
ndctl-monitor.service  disabled   
[email protected]  disabled   
[email protected]  disabled   
netctl-sleep.service  disabled   
netctl-wait-online.service  disabled   
netctl.service  disabled   
[email protected]  static   
NetworkManager-dispatcher.service  enabled   
NetworkManager-wait-online.service  disabled   
NetworkManager.service  enabled   
nfs-blkmap.service  disabled   
nfs-idmapd.service  static   
nfs-mountd.service  static   
nfs-server.service  disabled   
nfs-utils.service  static   
nmb.service  disabled   
nscd.service  disabled   
ntpd.service  enabled   
ntpdate.service  disabled   
octopi.service  disabled   
[email protected]  static   
org.cups.cupsd.service  enabled   
ostree-finalize-staged.service  static   
ostree-prepare-root.service  static   
ostree-remount.service  disabled   
paccache.service  static   
pacman-init.service  disabled   
polkit.service  static   
postgresql.service  disabled   
[email protected]  disabled   
pptpd.service  disabled   
privoxy.service  enabled   
quotaon.service  static   
rescue.service  static   
[email protected]  disabled   
[email protected]  disabled   
[email protected]  static   
rpc-gssd.service  static   
rpc-statd-notify.service  static   
rpc-statd.service  static   
rpc-svcgssd.service  static   
rpcbind.service  disabled   
[email protected]  static   
rsyncd.service  disabled   
[email protected]  static   
rtkit-daemon.service  disabled   
samba.service  disabled   
[email protected]  static   
sensord.service  disabled   
[email protected]  disabled   
shadow.service  static   
smartd.service  disabled   
smb.service  disabled   
snmpd.service  disabled   
snmptrapd.service  disabled   
speech-dispatcherd.service  disabled   
spice-webdavd.service  static   
sshd.service  enabled   
[email protected]  static   
sshdgenkeys.service  static   
sssd-autofs.service  indirect   
sssd-ifp.service  static   
sssd-kcm.service  indirect   
sssd-nss.service  indirect   
sssd-pac.service  indirect   
sssd-pam.service  indirect   
sssd-ssh.service  indirect   
sssd-sudo.service  indirect   
sssd.service  disabled   
svnserve.service  disabled   
sysprof2.service  static   
sysstat-collect.service  static   
sysstat-summary.service  static   
sysstat.service  disabled   
system-update-cleanup.service  static   
systemd-ask-password-console.service  static   
systemd-ask-password-wall.service  static   
[email protected]  static   
systemd-binfmt.service  static   
systemd-bless-boot.service  static   
systemd-boot-check-no-failures.service  disabled   
[email protected]  static   
systemd-exit.service  static   
systemd-firstboot.service  static   
systemd-fsck-root.service  enabled-runtime
[email protected]  static   
systemd-halt.service  static   
[email protected]  static   
systemd-hibernate.service  static   
systemd-hostnamed.service  static   
systemd-hwdb-update.service  static   
systemd-hybrid-sleep.service  static   
systemd-importd.service  static   
systemd-initctl.service  static   
systemd-journal-catalog-update.service  static   
systemd-journal-flush.service  static   
systemd-journal-gatewayd.service  indirect   
systemd-journal-remote.service  indirect   
systemd-journal-upload.service  disabled   
systemd-journald.service  static   
systemd-kexec.service  static   
systemd-localed.service  static   
systemd-logind.service  static   
systemd-machine-id-commit.service  static   
systemd-machined.service  static   
systemd-modules-load.service  static   
systemd-networkd-wait-online.service  disabled   
systemd-networkd.service  disabled   
[email protected]  disabled   
systemd-portabled.service  static   
systemd-poweroff.service  static   
systemd-quotacheck.service  static   
systemd-random-seed.service  static   
systemd-reboot.service  static   
systemd-remount-fs.service  enabled-runtime
systemd-resolved.service  disabled   
systemd-rfkill.service  static   
systemd-suspend-then-hibernate.service  static   
systemd-suspend.service  static   
systemd-sysctl.service  static   
systemd-sysusers.service  static   
systemd-time-wait-sync.service  disabled   
systemd-timedated.service  static   
systemd-timesyncd.service  enabled   
systemd-tmpfiles-clean.service  static   
systemd-tmpfiles-setup-dev.service  static   
systemd-tmpfiles-setup.service  static   
systemd-udev-settle.service  static   
systemd-udev-trigger.service  static   
systemd-udevd.service  static   
systemd-update-done.service  static   
systemd-update-utmp.service  static   
systemd-user-sessions.service  static   
systemd-vconsole-setup.service  static   
systemd-volatile-root.service  static   
talk.service  indirect   
[email protected]  static   
[email protected]  static   
tftpd.service  disabled   
tor.service  enabled   
udisks.service  static   
udisks2.service  disabled   
updatedb.service  static   
upower.service  disabled   
[email protected]  static   
usbmuxd.service  static   
[email protected]  static   
[email protected]  static   
uuidd.service  indirect   
vboxweb.service  disabled   
[email protected]  static   
winbind.service  disabled   
[email protected]  disabled   
[email protected]  disabled   
wpa_supplicant.service  disabled   
[email protected]  disabled   
[email protected]  static   
xfs_scrub_all.service  static   
[email protected]  static   
xinetd.service  disabled   
zfs-import-cache.service  disabled   
zfs-import-scan.service  disabled   
zfs-mount.service  disabled   
zfs-share.service  disabled   
zfs-zed.service  disabled   
[email protected]  static   
machine.slice  static   
user.slice  static   
avahi-daemon.socket  disabled   
dbus.socket  static   
dm-event.socket  static   
git-daemon.socket  disabled   
gpsd.socket  disabled   
krb5-kpropd.socket  disabled   
lircd.socket  disabled   
lvm2-lvmetad.socket  static   
lvm2-lvmpolld.socket  static   
org.cups.cups-lpd.socket  disabled   
org.cups.cupsd.socket  enabled   
rlogin.socket  disabled   
rpcbind.socket  disabled   
rsh.socket  disabled   
rsyncd.socket  disabled   
saned.socket  disabled   
sshd.socket  disabled   
sssd-autofs.socket  disabled   
sssd-kcm.socket  disabled   
sssd-nss.socket  disabled   
sssd-pac.socket  disabled   
sssd-pam-priv.socket  disabled   
sssd-pam.socket  disabled   
sssd-ssh.socket  disabled   
sssd-sudo.socket  disabled   
syslog.socket  static   
systemd-coredump.socket  static   
systemd-initctl.socket  static   
systemd-journal-gatewayd.socket  disabled   
systemd-journal-remote.socket  disabled   
systemd-journald-audit.socket  static   
systemd-journald-dev-log.socket  static   
systemd-journald.socket  static   
systemd-networkd.socket  disabled   
systemd-rfkill.socket  static   
systemd-udevd-control.socket  static   
systemd-udevd-kernel.socket  static   
talk.socket  disabled   
telnet.socket  disabled   
tftpd.socket  disabled   
uuidd.socket  disabled   
home-swapfile.swap  generated   
basic.target  static   
bluetooth.target  static   
boot-complete.target  static   
cryptsetup-pre.target  static   
cryptsetup.target  static   
ctrl-alt-del.target  disabled   
default.target  indirect   
emergency.target  static   
exit.target  disabled   
final.target  static   
getty-pre.target  static   
getty.target  static   
graphical.target  static   
halt.target  disabled   
hibernate.target  static   
hybrid-sleep.target  static   
initrd-fs.target  static   
initrd-root-device.target  static   
initrd-root-fs.target  static   
initrd-switch-root.target  static   
initrd.target  static   
kbrequest.target  indirect   
kexec.target  disabled   
local-fs-pre.target  static   
local-fs.target  static   
machines.target  disabled   
multi-user.target  indirect   
network-online.target  static   
network-pre.target  static   
network.target  static   
nfs-client.target  disabled   
nss-lookup.target  static   
nss-user-lookup.target  static   
paths.target  static   
poweroff.target  disabled   
printer.target  static   
reboot.target  disabled   
remote-cryptsetup.target  disabled   
remote-fs-pre.target  static   
remote-fs.target  enabled   
rescue.target  indirect   
rpc_pipefs.target  static   
rpcbind.target  static   
runlevel0.target  disabled   
runlevel1.target  indirect   
runlevel2.target  indirect   
runlevel3.target  indirect   
runlevel4.target  indirect   
runlevel5.target  static   
runlevel6.target  disabled   
shutdown.target  static   
sigpwr.target  static   
sleep.target  static   
slices.target  static   
smartcard.target  static   
sockets.target  static   
sound.target  static   
suspend-then-hibernate.target  static   
suspend.target  static   
swap.target  static   
sysinit.target  static   
system-update-pre.target  static   
system-update.target  static   
time-sync.target  static   
timers.target  static   
umount.target  static   
zfs-import.target  disabled   
zfs.target  enabled   
[email protected]  disabled   
cnf.timer  disabled   
e2scrub_all.timer  disabled   
fstrim.timer  disabled   
logrotate.timer  disabled   
man-db.timer  disabled   
[email protected]  static   
paccache.timer  disabled   
shadow.timer  static   
sysstat-collect.timer  disabled   
sysstat-summary.timer  disabled   
systemd-tmpfiles-clean.timer  static   
updatedb.timer  masked   
xfs_scrub_all.timer  disabled

Ne vidim nista ovde sto bi pozivalo initdb skriptu, ali sigurno negde jeste u ovima.

 

[webad]

Zaboravio sam da naglasim jos jedno moje ogranicenje:
Meni su svi pg-ovi vec godinama unazad na centos-u, a trenutno su na v10.

Nisam pitao, da li se napravi prazan “data” dir?
Nakon boot-a, pre nego sto se mount-uje disk i pokrene PG, ako se ustanovi da se napravio problematican direktorijum, da li si analizirao log-ove?
Bacicu pogled ako vidim nesto u ovom spisku (ali iskreno ne polazem mnogo nade )
Stvari koje mi padaju na pamet:

• 
  []Mozda se prilikom shutdown-a umount-uje disk, a ostanu PG procesi
  []Mozda u cron-u, anacron-u, at-u ili nekom drugom servisu tog tipa imas neke poslove koji pokusavaju pre nego sto PG bude osposobljen ili nakon sto bude onesposobljen
  []Mozda u PG postoji neki proces koji zove eksterni rad sa sistemom (copy to program)
  []Da mount-ujes na drugu lokaciju i postgresql.conf promenis “data_directory” da gleda na tu novu lokaciju. U zavisnosti da li ce problem da se pokaze ponovo na novoj ili staroj lokaciji moze da ukaze da li problem pravi neki PG-ov proces ili mozda postoji neka druga spodoba koja pokusava nesto da radi

 

[Branimir_Maksimovic]

postgres je dizejblovan, tako da nema procesa (mislim), ne bi trebalo. Ovo je od skora pocelo da se pojavljuje, i to sam pre par dana
uopste primetio zato sto naprasno data set nije mogao da se mount-uje. Nepogresivo nesto pravi data dir svaki put, pretpostavljam
prilikom boot-a. Probacu ovo sto si naveo, u nadi da cu otkriti sta pravi taj data dir.
U cron-u i systemd timerima nema nista sumnjivo.
Kad dodjem kuci probacu da premestim data dir pa da vidim sta ce da se desi.

 

[webad]

webad:

da li se napravi prazan “data” dir?

Mozes li dati odgovor na ovo? Ako nije prazan, sta je sadrzaj, tj da li se kreira “sve (ili delimicno) sto treba” za funkcionisanje PG-a?

postgres je dizejblovan, tako da nema procesa (mislim), ne bi trebalo.

Ako ne gresim, disejblovan je za automatsko paljenje pri boot-u. To nema veze sa time da se on kasnije upali, radi normalno, pokrene proces gasenja, umount-uje disk, a PG nije zavrsio sa gasenjem, tj neki procesi/upiti ostaju da se izvrsavaju i potencijalno pokusaju da skladiste svoj rezultat.

Ovo pricamo o kucnom ili kakvom racunaru pa je u pitanju db server koji se redovno restartuje? Ako je tako, nisam imao iskustva ni sa takvom situacijom sto se tice dela da je “data” na sporednom disku.

 

[Branimir_Maksimovic]

Kad stignem kuci mogu da vidim dal je prazan data dir nakon boot-a.
Procesa nema zato sto je to moja kucna masina, i startujem samo kad testiram nesto. Ali neki pg proces se definitivno startuje
a da ja za to ne znam

 

[Branimir_Maksimovic]

Napravi prazan data dir i nista vise. Hm.

 

[webad]

Jako zanimljivo. Ako je prazan, onda nije greskom vrsena regularna inicijalizacija baze, tj nije initdb (valjda) nego nesto drugo mozda zahteva postojanje direktorijuma jednostavnim “mkdir -p”. Moze neka pomoc biti i vreme kreiranja direktorijuma za pretragu po log-ovima.
Dodatno igranje koje mi pada na pamet kako da utvrdimo da li se dir pravi prilikom paljenja ili prilikom gasenja:

• 
  []ugasiti
  []upaliti neki live sistem preko diska ili usb-a
  []videti da li postoji data dir
  []zatim se vratiti na “normalan” sistem

Moguci scenariji su:

• 
  []ne postoji data dir u live + ne postoji data dir u normalnom
  
  • 
    [
  • ]try again
  []postoji data dir u live
  
  • 
    [
  • ]znaci da je nastao prilikom gasenja
  []ne postoji data dir u live, ali postoji u normalnom
  
  • 
    [
  • ]znaci da je nastao prilikom paljenja

 

[Branimir_Maksimovic]

Ma jednostavno ugasim, sacekam neko vreme, pa upalim. Vreme kreiranja data dira ce otkriti da li je u shutdown ili boot vremenu.

 

[Branimir_Maksimovic]

Znaci nesto prilikom boot-a to pravi.

 

[webad]

Nisam pametan da kazem ko je kriv…
Ko je owner, ko ima ovlascenja nad direktorijumom? root ili neko treci?
Po vremenu kreiranja juriti po log-ovima da li je iko ista prijavio.
Mozda ici i ekstremno pa brisati PG (naravno backup pre toga) i videti da li ce se i dalje dogadjati.

Da li imas nesto kao “/etc/postgresql/…/postgresql.conf”?
U startu nisam pitao, mozda pomogne u istrazi: koji OS i verzija PG-a?

 

[Branimir_Maksimovic]

Nema /etc/post…, Manjaro, Zadnja verzija 11.2.
Vlasnik dira je postgress.

Nego mislim mozda je to neki servis systemd-a koji tako
kreira direktorijume i negde je zakucano da taj data dir
mora postojati. Kazem, to je od skora neka izmena, ranije
nije bilo.

 

[Branimir_Maksimovic]

Nasao sam sta je!
systemd kreira tmp direktorijume iz /var/lib/tmpfiles.d
I gle

usr/lib/tmpfiles.d >>> cat postgresql.conf   
d /var/lib/postgres/data 700 postgres postgres
usr/lib/tmpfiles.d >>> pacman -Qo postgresql.conf   
/usr/lib/tmpfiles.d/postgresql.conf is owned by postgresql 11.2-3

Znaci postgres je napusao da se tu pravi tmp dir ;(
Kakav bedak.
Nista jedan rm i resen problem. Bar sam naucio kako da nateram systemd
da pravi direktorijume