# ubuntu-livecd **Repository Path**: dev_ma/ubuntu-livecd ## Basic Information - **Project Name**: ubuntu-livecd - **Description**: Ubuntu livecd patch and config - **Primary Language**: Unknown - **License**: GPL-2.0 - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 1 - **Created**: 2025-06-08 - **Last Updated**: 2026-08-03 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # Ubuntu LiveCD * [Virtual machine](#virtual-machine) * [patch.livecd.sh](#patchlivecdsh) * [Dir entry of file grub.cfg](#dir-entry-of-file-grubcfg) * [Persistent file](#persistent-file) * [Configure live system](#configure-live-system) * [Disable auto login](#disable-auto-login) * [noeject](#noeject) * [Disable unnecessary services](#disable-unnecessary-services) * [Auto start GUI](#auto-start-gui) * [Install ibus](#install-ibus) * [Hardware clock local time](#hardware-clock-local-time) * [clean-tmp.service](#clean-tmpservice) * [snapd.apparmor.service](#snapdapparmorservice) * [livecd.mounts.finalrd](#livecdmountsfinalrd) * [finalrd](#finalrd) * [systemd shutdown scripts](#systemd-shutdown-scripts) * [Build systemd](#build-systemd) * [Livecd hibernate](#livecd-hibernate) * [Use swap partition](#use-swap-partition) * [Use swap file](#use-swap-file) * [zramswap.sh](#zramswapsh) * [Allow GUI hibernate](#allow-gui-hibernate) * [Grub boot iso file](#grub-boot-iso-file) * [Grub check file](#grub-check-file) * [Chainload grub core](#chainload-grub-core) * [Grub UART terminal](#grub-uart-terminal) * [Install livecd grub](#install-livecd-grub) * [Build grub](#build-grub) * [Copy or move grub installed](#copy-or-move-grub-installed) * [Install gurb bootsector](#install-gurb-bootsector) * [initrd](#initrd) * [initramfs.split.sh](#initramfssplitsh) * [patch.main.cpio.sh](#patchmaincpiosh) * [Create inird cpio file](#create-inird-cpio-file) * [Create initrd](#create-initrd) * [initrd script files and functions](#initrd-script-files-and-functions) * [Lxd container and docker](#lxd-container-and-docker) * [Container working folder](#container-working-folder) * [Create container image and run it](#create-container-image-and-run-it) * [Lxd dns record](#lxd-dns-record) * [Mount in container](#mount-in-container) * [Run livecd in container](#run-livecd-in-container) * [Generate man pages of snap lxc](#generate-man-pages-of-snap-lxc) * [Prepare upperdir for overlayfs](#prepare-upperdir-for-overlayfs) * [Prepare rootfs for container](#prepare-rootfs-for-container) * [Create container](#create-container) * [Configure overlay rootfs for lxc previleged container](#configure-overlay-rootfs-for-lxc-previleged-container) * [Prepare overlay rootfs for lxc unprevileged container](#prepare-overlay-rootfs-for-lxc-unprevileged-container) * [Bind mount and map uid and gid for container root](#bind-mount-and-map-uid-and-gid-for-container-root) * [Start container](#start-container) * [Snapd in lxc container](#snapd-in-lxc-container) * [Move apt cache folder to memory](#move-apt-cache-folder-to-memory) * [Install sshd and xrdp](#install-sshd-and-xrdp) * [Move tmp folders to memory](#move-tmp-folders-to-memory) * [Run Alpine Linux in lxc container](#run-alpine-linux-in-lxc-container) * [Seperate apkcache files to repository dirs](#seperate-apkcache-files-to-repository-dirs) * [Install xfce and xrdp](#install-xfce-and-xrdp) * [apt-cacher-ng](#apt-cacher-ng) * [Move data folders](#move-data-folders) * [Offline mode](#offline-mode) * [Cache repository Releae and Packages](#cache-repository-releae-and-packages) * [Windows NTP server](#windows-ntp-server) **CAUTION** **Please backup your data before run any test.** Test with a virtual machine, i.e. VirtualBox or qemu, make sure no data loss, then move to a real machine. A live system with web browser and office tools only uses around 3G disk space, compared to Win10 60G. So that many virtual machines can run at same time. Ubuntu livecd uses "casper" as initramfs-tool to boot live system. View man page after boot up and login. ``` man casper ``` Ubuntu LiveCD iso files ``` https://releases.ubuntu.com/jammy/ubuntu-22.04.5-live-server-amd64.iso https://cdimage.ubuntu.com/lubuntu/releases/22.04.5/release/lubuntu-22.04.5-desktop-amd64.iso https://cdimage.ubuntu.com/lubuntu/releases/noble/release/lubuntu-24.04.2-desktop-amd64.iso ``` ## Virtual machine Audio plays sound from vbox vm when chose Intel HD Audio as audio controller. Avoid hang in vbox headless detachable start: add "nomodeset" to kernel cmdline. Avoid boot from CDROM, use IED slave CDROM drive instead of IDE primary CDROM drive. After test with virtual disks and every thing works well, before move to a real machine, test with physical disk. ``` # vbox lists partitions of a raw disk "c:\Program Files\Oracle\VirtualBox\VBoxManage.exe" internalcommands listpartitions -rawdisk \\.\PhysicalDrive0 # let vbox use raw disk # partition 5 and 7 are linux partitions, and not used by Windows "c:\Program Files\Oracle\VirtualBox\VBoxManage.exe" internalcommands createrawvmdk -filename sda57.raw.vmdk -rawdisk \\.\PhysicalDrive0 -partitions 5,7 -relative ``` Virtual machine can see all partitions, but can only read write assigned partitions, and cannot read write other partitions. Add another virtual disk as EFI or grub boot partition if necessary, and then can boot with the physical partition. Modify EFI mount in /etc/fstab, add "nofail" to mount options, to avoid boot blocked by missing EFI partition. Win10 diskpart ``` ( echo list disk && echo select disk 0 && echo list partition) | diskpart ``` Win10 host qemu use physical disk ``` d:\programs\qemu-w64-2.12\qemu-system-x86_64.exe -smp 2 -m 2G -drive file=\\.\PhysicalDrive0,if=virtio,format=raw,media=disk -boot order=a -snapshot ``` Refer to qemu doc file "qemu-doc.html" for help. "qemu-doc.html" comes with installed "qemu-system-x86_64.exe". Win10 host qemu "accel=tcg" requires turn on "Windows Hypervisor Platform" in "Turn Windows features on of off". Virtualbox failed to start vm when turn on Windows Hypervisior Platform: VT-x is not available. For x64 vm, vbox requires VT-x. ``` # turn off hypervisor, and reboot bcdedit /set hypervisorlaunchtype off ``` ## patch.livecd.sh Run "patch.livecd.sh" to patch livecd iso file, change content of file "grub.cfg" and set file size. ``` # get help info sh patch.livecd.sh -h # patch iso file sh patch.livecd.sh -v -p lubuntu22045 -s "resume=UUID=cb198d63-b5ca-4d95-a274-41fe82e417a8 resume_offset=659456" lubuntu-22.04.5-desktop-amd64.iso # restore iso file to original sh patch.livecd.sh -r lubuntu-22.04.5-desktop-amd64.iso ``` Old head part of livecd iso file is backup to file with ext-name ".iso.old.head". Ubuntu casper boot only allows persistent file in vfat partition. It has a bug, "/root/tmp/debconf.xxxx" folder is not deleted, if you check the folder in initramfs shell when use "break=init". After patch, it allows writable-backing and casper-sn on fs more than vfat, patch to delete "/root/$DEBCONF_TMPDIR", allow resume from hibernate. ### Dir entry of file grub.cfg ``` # ls -U /cdrom/boot/grub/ fonts grub.cfg i386-pc loopback.cfg x86_64-efi # stat /cdrom/boot/grub/grub.cfg /cdrom/boot/grub/loopback.cfg File: /cdrom/boot/grub/grub.cfg Size: 648 Blocks: 2 IO Block: 2048 regular file Device: b00h/2816d Inode: 2505 Links: 1 Access: (0444/-r--r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-08-07 15:52:06.000000000 +0000 Modify: 2023-08-07 15:52:05.000000000 +0000 Change: 2023-08-07 15:52:05.000000000 +0000 Birth: - File: /cdrom/boot/grub/loopback.cfg Size: 386 Blocks: 1 IO Block: 2048 regular file Device: b00h/2816d Inode: 2512 Links: 1 Access: (0444/-r--r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-08-07 15:52:06.000000000 +0000 Modify: 2023-08-07 15:52:05.000000000 +0000 Change: 2023-08-07 15:52:05.000000000 +0000 Birth: - # trans to hex # ( set -xv; printf "0x%04x\n" 648 386; printf "%02x " $((2023 -1900)) 8 7 15 52 05 ) + printf '0x%04x\n' 648 386 0x0288 0x0182 + printf '%02x ' 123 8 7 15 52 05 7b 08 07 0f 34 05 # grub-2.06/grub-core/fs/iso9660.c struct grub_iso9660_dir { grub_uint8_t len; grub_uint8_t ext_sectors; grub_uint32_t first_sector; grub_uint32_t first_sector_be; grub_uint32_t size; grub_uint32_t size_be; struct grub_iso9660_date2 mtime; // 7 bytes grub_uint8_t flags; grub_uint8_t unused2[6]; #define MAX_NAMELEN 255 grub_uint8_t namelen; } GRUB_PACKED; # in grub_iso9660_iterate_dir(): semicolon = grub_strrchr (ctx.filename, ';'); if (semicolon) *semicolon = '\0'; # cdrkit-1.1.11/genisoimage/iso9660.h struct iso_directory_record { unsigned char length [ISODCL(1, 1)]; // 1 byte ... char size [ISODCL(11, 18)]; // 8 bytes char date [ISODCL(19, 25)]; // 7 bytes ... unsigned char name_len [ISODCL(33, 33)]; // 1 byte char name [MAX_ISONAME+1]; // 208 bytes }; struct iso_ext_attr_record {}; ``` ## Persistent file By default, for lubuntu-22.04-desktop-amd64.iso and ubuntu-22.04-live-server-amd64.persistent.iso, persistent files must be named "writable" and "home-rw" in a FAT32 partition when partition is not labeled as "capser-rw". Set ext4 fs label to "casper-rw", then this partition can be used as persistent. Ubuntu-22.04 LiveCD can use lvm partition as persistent. Recommend to save your data in other lvm partition, to expand or reduce partition size easily, add or remove disks dynamically. Partition labled with "casper-rw" overrides "writable" file in folder specified by persistent-path (default to top folder in vfat partition). To use "writable" file in fodler persistent-path, change the partition lable to other string. "home-rw" file is always searched and used when use persistent-path. Web resource ``` https://askubuntu.com/questions/1269462/bios-uefi-template-image-for-booting-iso-files ``` Scripts handle persistent-path ``` initrd/main/scripts/casper initrd/main/scripts/casper-helpers find_cow_device() ``` At the first Lubuntu system boot up, login in as user "lubuntu" without password. Then create persistent cow file for "casper". Example below uses LVM (Logical Volume Manager) as persistent writable-backing device. ``` mkdir s sudo mount /dev/vglubuntu/lvlubuntu s sudo mkdir s/lubuntu22045 sudo dd if=/dev/zero of=s/lubuntu22045/writable bs=1M count=124 # ( set -xv; f=s/lubuntu22045/writable; mkdir -p t && d=$(sudo losetup --find --show $f) && [ -n "$d" ] && sudo mkfs.btrfs $d && sudo mount $d t && sudo mkdir t/upper t/work && sudo btrfs property set t/upper compression lzo && btrfs property get t/upper compression && sudo umount t && sudo losetup -v -d $d && rmdir t; echo rt=$? ) ``` Then reboot, login and check if "writable" file is used. ``` losetup grep "\s/\s" /proc/mounts ``` ## Configure live system The configuration will be saved to persistent file or partition. ### Disable auto login Add user and add to sudo group before disable auto login. ``` sudo adduser sudo adduser user sudo sudo adduser user lxd ``` For Lubuntu-22.04.3, disable GUI auto login: Remove all lines of section "[Autologin]" in file "/etc/sddm.conf". ``` man sddm.conf sudo crontab -e sudo crontab -l @reboot sh -xvc '( echo "[Users]"; echo "MinimumUid=1000" ) | tee /etc/sddm.conf' 2>&1 | logger -t cron.job.reboot.disable-auto-login # check and change "MinimumUid" if it does not cover desired user ID tail /etc/passwd ``` For ubuntu-22.04.5, disable GUI auto login: ``` sudo sed -i -e 's/^AutomaticLogin/#&/' /etc/gdm3/custom.conf ``` Prevent reset to empty passwd of user lubuntu. Comment off (by # at line beginning) user lubuntu in file "/etc/passwd". ``` tail /etc/passwd sudo sed -i -e 's/^lubuntu:/#&/' /etc/passwd ``` Disable tty auto login. ``` systemctl cat serial-getty@.service getty@.service # Comment off all lines in file: /usr/lib/systemd/system/getty@.service.d/autologin.conf ``` ### noeject By default, auto eject cdrom at system reboot or poweroff, by script file "/usr/sbin/casper-stop". Patch it to enable "noeject". ``` cat casper-stop.noeject.patch | sudo patch -d /usr/sbin ``` ### Disable unnecessary services Disable unattended-upgrades.service, fwupd services and apt-daily services. Download a new livecd instead of upgrade. ``` systemctl status fwupd* ( set -xv; for i in unattended-upgrades.service fwupd-refresh.timer fwupd-refresh.service fwupd.service fwupd-offline-update.service motd-news.service motd-news.timer apt-daily.timer apt-daily.service apt-daily-upgrade.timer apt-daily-upgrade.service; do systemctl --no-pager status $i; sudo systemctl stop $i; sudo systemctl disable $i; done ) # Disable auto upgrade sudo sed -i -e 's/1\(\"\;\)/0\1/' /etc/apt/apt.conf.d/10periodic /etc/apt/apt.conf.d/20auto-upgrades sudo sed -i -e 's/Prompt=lts/Prompt=never/' /etc/update-manager/release-upgrades tail /etc/apt/apt.conf.d/10periodic /etc/apt/apt.conf.d/20auto-upgrades /etc/update-manager/release-upgrades # disable motd sudo sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news sudo chmod -x /etc/update-motd.d/* # limit journal log disk usage echo SystemMaxUse=6M | sudo tee -a /etc/systemd/journald.conf # reduce syslog grep -A5 -B5 'rotate ' /etc/logrotate.d/rsyslog sudo sed -i -e 's/^\(\s\+\)weekly$/\1daily/' /etc/logrotate.d/rsyslog # clean apt cache sudo apt clean # For Ubuntu live server, disable subiquity sudo snap disable subiquity sudo systemctl stop snap-subiquity-6066.mount sudo systemctl disable snap-subiquity-6066.mount snap.subiquity.subiquity-server.service ``` Ubuntu live server UART console conflicts with Subiquity. ``` # serial-getty@ttyS0.service failed because of "Subiquity, the installer for Ubuntu Server ttyS0." systemctl cat serial-getty@.service getty@.service # comment off "[Service]" section in file: /usr/lib/systemd/system/serial-getty@.service.d/subiquity-serial.conf ``` Start UART console. ``` # run getty ttyS0 without kernel parameter "console=ttyS0" sudo systemctl start serial-getty@ttyS0.service # Manually set terminal settings after login, change cols and rows to match your terminal. # "vim" cannot display properly if terminal size set incorrectly. In PuTTy, stty rows shall be (rows reported by PuTTy - 1). ( set -xv; stty sane cols 135 rows 38 && env TERM=xterm-256color bash -i ) ``` ### Auto start GUI ``` # disable auto start GUI sudo systemctl set-default multi-user.target # enable auto start GUI sudo systemctl set-default graphical.target # check GUI status systemctl status display-manager.service # Manually start GUI sudo systemctl start graphical.target # Manually stop GUI sudo systemctl stop sddm.service ``` ### Install ibus Use apt-cacher-ng server if you have. ``` echo 'Acquire::http::Proxy "http://192.168.58.211:3142";' | sudo tee /etc/apt/apt.conf.d/00aptproxy ``` Install input method in Lubuntu-22.04.3. Fcitx4 not work. Use ibus-pinyin. ``` sudo apt install ibus-pinyin ibus ``` ### Hardware clock local time Windows has bug to set hardware clock as UTC time for decades, it assumes not accross time zones a lot. Linux system by default set hardware clock as UTC time, and can be easily configured to use local time. If a real machine swithes between Linux and Windows, hardware clock will mess up. Let hardware clock use local time, same as Windows. ``` # web resource https://askubuntu.com/questions/946516/how-to-tell-ubuntu-that-hardware-clock-is-local-time timedatectl set-local-rtc 1 --adjust-system-clock ``` ### clean-tmp.service Another solution to delete "/tmp/debconf.xxxxx" folders in "cow/upper/tmp/": use systemd service "clean-tmp.service". Install it and enable it. ``` sudo sh -xvc 'f=clean-tmp.service; d=/etc/systemd/system; [ -e "$f/$d" ] && exit 0; cp -a $f $d/ && f=$d/$f && chown 0:0 $f && chmod 0644 $f && systemctl enable clean-tmp.service' ``` ### snapd.apparmor.service In Ubuntu-22.04-live-server, "/var/snap/lxd/common/ns/" mount is required by lxd and is mounted by snap.lxd.daemon.service, which depends on snapd.apparmor.service and apparmor.service. "/var/snap/lxd/common/ns/" is not mounted when apparmor.service failed. ``` sudo systemctl status apparmor.service snapd.apparmor.service systemctl cat apparmor.service snapd.apparmor.service # # This line in apparmor.service prevent apparmor to start when run livecd. Comment off it to avoid this problem. ConditionPathExists=!/rofs/etc/apparmor.d # check lxd.daemon service in snap snap list snap services ``` ## livecd.mounts.finalrd By default, the mount point of loop device of "writable" file is hidden. It can be available by run file "livecd.mounts.finalrd" from initrd. Copy file "livecd.mounts.finalrd" to your machine, and run command below to install it. ``` # get help information sh livecd.mounts.finalrd --help # run install sudo sh livecd.mounts.finalrd install # check installed file ls -l /etc/finalrd/ tail /usr/share/debconf/confmodule ``` ## finalrd Lubuntu-22.04.3 shutdown bug, finalrd bug: Lubuntu-22.04.3 systemd-shutdown dead loop if boot from livecd iso file in ntfs partition. ``` https://bugs.launchpad.net/snapd/+bug/1920107 https://bugs.launchpad.net/ubuntu/+source/finalrd/+bug/1905166 https://github.com/grml/grml/issues/206 https://github.com/systemd/systemd/issues/14410 https://github.com/systemd/systemd/pull/14409 ``` A quick fix: replace "/lib/systemd/systemd-shutdown" with file "systemd-shutdown.sh". Run the short script in its head part to install it. "systemd-shutdown.sh" requires "livecd.mounts.finalrd". A better fix: patch source code of systemd and rebuild it. Patch file: systemd-249.11.shutdown.deadloop.patch In file systemd-249.11/src/shutdown/umount.c:435, delete_loopback(), when last call "ioctl(fd, LOOP_GET_STATUS64, &info)" failed, it shall return 0 instead of 1 at line 442. Further improvement, add a loopcnt in main() of file systemd-249.11/src/shutdown/shutdown.c:488. Force "changed=0" when loop too much. ### systemd shutdown scripts ``` # web resource https://askubuntu.com/questions/236476/how-do-i-diagnose-hang-on-shutdown # scripts ls /usr/lib/systemd/system-shutdown/ /usr/share/finalrd/ sudo ls -a /run/initramfs/ man finalrd systemctl cat finalrd.service cat /usr/bin/finalrd cat /usr/lib/finalrd/finalrd-static.conf /run/finalrd-libs.conf man systemd-tmpfiles ``` ## Build systemd Systemd source code ``` https://github.com/systemd/systemd/ https://codeload.github.com/systemd/systemd/tar.gz/refs/tags/v249 https://github.com/systemd/systemd/blob/main/src/shutdown/shutdown.c https://github.com/systemd/systemd/blob/main/src/shutdown/umount.c ``` Ubuntu systemd-249.11 source code ``` https://launchpadlibrarian.net/656926680/systemd_249.11-0ubuntu3.9.dsc https://launchpadlibrarian.net/656926679/systemd_249.11-0ubuntu3.9.debian.tar.xz https://launchpadlibrarian.net/656926678/systemd_249.11.orig.tar.gz ``` Extract source files. ``` ( set -xv; mkdir t && cd t && dpkg-source -x ../download/systemd-249.11-0ubuntu3.9.dsc && mv systemd_249.11 .. && cd .. && rm -fr t; echo rt=$? ) ``` Install dependencies, required packages are checked and reported when run build. ``` sudo apt install debhelper meson ... ``` Apply patch. ``` cat systemd-249.11.shutdown.deadloop.patch | patch -d systemd-249.11 -p 1 ``` Run build. ``` ( set -xv; history | tail -n 1; lsb_release -a; uname -a; systemd-detect-virt; cd systemd-249.11 && fakeroot -- dpkg-buildpackage --build=binary; echo rc=$? ) 2>&1 | tee systemd.build.log ``` Continue build after modify source file. ``` ( set -xv; history | tail -n 1; cd systemd-249.11 && fakeroot -- make -w -f debian/rules DH_VERBOSE=1 override_dh_auto_build; echo rc=$? ) 2>&1 | tee systemd.build.log ``` Generate patch file. ``` ( set -xv; history | tail -n1; tabs -4; diff -duwp systemd-249.11.old/src/shutdown/shutdown.c systemd-249.11/src/shutdown/shutdown.c; diff -duwp systemd-249.11.old/src/shutdown/umount.c systemd-249.11/src/shutdown/umount.c; diff -duwp systemd-249.11.old/meson.build systemd-249.11/meson.build ) 2>&1 | tee systemd-249.11.shutdown.deadloop.patch ``` Use static linked systemd-shutdown to avoid libsystemd-shared-249.so library dependency problem when run it as replacement of old "/lib/systemd/systemd-shutdown" in system reboot or poweroff. Try run, shall fail for libsystemd-shared. ``` cp systemd-249.11/build-deb/systemd-shutdown . ./systemd-shudown ``` Check dependent library. ``` env LD_TRACE_LOADED_OBJECTS=1 systemd-249.11/build-deb/systemd-shutdown | grep libsystemd-shared env LD_TRACE_LOADED_OBJECTS=1 deb.extracted/lib/systemd/systemd-shutdown env LD_TRACE_LOADED_OBJECTS=1 systemd-249.11/build-deb/systemd-shutdown.standalone ``` Strip systemd-shutdown.standalone. ``` cp systemd-249.11/build-deb/systemd-shutdown.standalone . strip -s systemd-shutdown.standalone ls -l systemd-249.11/build-deb/systemd-shutdown.standalone systemd-shutdown.standalone env LD_TRACE_LOADED_OBJECTS=1 ./systemd-shutdown.standalone # static linked systemd-sysusers.standalone systemd-tmpfiles.standalone grep standalone systemd-249.11/meson.build ls systemd-249.11/build-deb/*standalone ``` Build systemd-249.11 in ubuntu-20.04.1, change debhelper-compat from 13 to 12 in file "systemd-249.11/debian/control": Build-Depends: debhelper-compat (= 12), ``` ( set -xv; f=systemd-249.11.old/debian/control; [ -e $f.old ] || cp -a $f $f.old; sed -i -e 's/\(\sdebhelper-compat\s(=\) 13),/\1 12),/' $f; diff -duwp $f.old $f ) ``` Build failed at test-fs-util: systemd-249.11/src/test/test-fs-util.c:687 Solution: comment off this line. ``` ( set -xv; f=systemd-249.11/src/test/test-fs-util.c; [ -e $f.old ] || cp -a $f $f.old; sed -i -e 's/^\s\+assert_se(st.st_blocks == 0)\;/\/\/&/' $f; diff -duwp $f.old $f ) ``` Check meson log. ``` less systemd-249.11/build-deb/meson-logs/testlog.txt ``` Check built files. ``` grep -n " systemd-shutdown " systemd.build.log file systemd-249.11/build-*deb/systemd-shutdown file systemd-249.11/debian/install/*deb/lib/systemd/systemd-shutdown # file ./systemd-249.11/build-*deb/systemd-shutdown.standalone env LD_TRACE_LOADED_OBJECTS=1 ./systemd-249.11/build-deb/systemd-shutdown.standalone # try run systemd-shutdown.standalone ./systemd-249.11/build-deb/systemd-shutdown.standalone ``` Extract deb file. ``` ( set -xv; mkdir -p deb.extacted && cd deb.extracted && dpkg-deb --extract ../systemd_249.11-0ubuntu3.9_amd64.deb . && file lib/systemd/systemd-shutdown && ls -l lib/systemd/systemd-shutdown; echo rt=$? ) # extract deb debug files ( set -xv; cd deb.extracted && dpkg-deb --extract ../systemd-dbgsym_249.11-0ubuntu3.9_amd64.ddeb .; echo rt=$? ) # find usr/lib/debug/ -type f -exec sh -c 'objdump -t $1 | grep delete_loopback && echo $1' x \{\} \; ``` Binary patch systemd-shutdown is possible, the "return" entrance is "safe_close(*fd)" in assembly codes. Before jump to this, some register (r12d or rax) shall be patched. That will be another work. ## Livecd hibernate ***CAUTION*** ***File system will be crashed if resume from a miss matched hibernate swap, i.e. fs is modifed since hibernate and before resume, by other OS or livecd.*** Hibernate requires swap partition or swap file, but not zram swap. ``` Web resources https://www.kernel.org/doc/Documentation/power/swsusp.txt https://www.kernel.org/doc/Documentation/power/basic-pm-debugging.txt https://www.kernel.org/doc/Documentation/power/swsusp-and-swap-files.txt https://wiki.archlinux.org/title/Power_management/Suspend_and_hibernate https://askubuntu.com/questions/1534335/hibernate-from-command-line-works-but-not-from-menu https://askubuntu.com/questions/1240123/how-to-enable-the-hibernate-option-in-ubuntu-20-04 ``` ### Use swap partition ``` sudo mkswap /dev/sdb8 sudo swapon /dev/sdb8 sudo blkid /dev/sdb8 sudo systemctl hibernate # use swap partition set resumeswap="resume=UUID=8b05f6cf-8c3b-4da8-b27c-619ee7761ae7" # use swap file, UUID is the partition contains swap file set resumeswap="resume=UUID=cb198d63-b5ca-4d95-a274-41fe82e417a8 resume_offset=659456" # append UUID of swap partition to kernel parameter: console=tty console=ttyS0,115200 no_suspend_console break=premount $resumeswap # grub set var set iso_path="" #set iso_path=/ubuntu-22.04.5-live-server-amd64.iso echo iso_path=$iso_path set isoscan="" #set isoscan="iso-scan/filename=$iso_path" echo isoscan=$isoscan set krndbg="" #set krndbg="console=tty1 console=ttyS0,115200 loglevel=7 ignore_loglevel keep no_console_suspend" #set krndbg="$krndbg initcall_debug debug retain_initrd break=momuntroot,casper-bottom,bottom,init" echo krndbg=$krndbg set resumeswap="" #set resumeswap="resume=UUID=cb198d63-b5ca-4d95-a274-41fe82e417a8 resume_offset=827392" echo resumeswap=$resumeswap set casperboot="boot=casper $isoscan persistent persistent-path=ubuntu22045server noprompt noeject username=xxxusername hostname=ubtsvr22045 cloud-init=disabled" echo casperboot=$casperboot # grub load kernel linux /casper/vmlinuz nomodeset keep $krndbg $resumeswap $casperboot ... # run resume in initrd premount break shell sh -xv /scripts/local-premount/resume ``` "blkid -p" reports SWAPTYPE is swsuspend (not swap) when system hibernated on that swap. Option "-p" (--probe, low-level superblocks probing, bypass cache) is necessary, otherwise resume failed. ``` # use swap partition blkid -p -o value -s TYPE /dev/sdb8 # use swap file blkid -p -O $((659456 * 4096)) /dev/vglubuntu/lvlubuntu ``` ### Use swap file On file systems other than Btrfs, the value of "resume_offset=" can be obtained by running filefrag -v swap_file. The desired value is in the first row of the physical_offset column. ``` # 512M swap file for 1G RAM # 2G swap file for 16G RAM sudo dd if=/dev/zero bs=1M count=512 of=s/swapfile.1 sudo mkswap s/swapfile.1 sudo chmod a-rw,u+rw s/swapfile.1 sudo filefrag -v s/swapfile.1 ``` Cron job to use swap file. ``` sudo crontab -e sudo crontab -l @reboot sh -xvc 'f=/media/writable-backing/swapfile.1; [ -e "$f" ] || f=/media/isodevice/swapfile.1; [ -s $f ] && swapon $f' 2>&1 | logger -t cron.job.reboot.swapon ``` ### zramswap.sh Hibernate while keep zram swap, may fail to resume. Install "zramswap.sh" to avoid this problem. Run the short script in its head part to install it. ``` head zramswap.sh systemctl status zram-config.service systemctl cat zram-config.service cat /usr/bin/init-zram-swapping cat /usr/bin/end-zram-swapping # man systemd-sleep ls /etc/pm/ /usr/lib/pm-utils/sleep.d/ /lib/systemd/system-sleep/ ``` For linux-5.15.0-119-generic, when "mem_limit" reached, "backing_dev" is used. Size of "backing_dev" is big enough in tests. "disksize" shall be not bigger than "mem_limit", otherwise fs of backing_dev crashes when disk is full in worst case (all data is incompressible). An echo "huge" to "writeback" tiggers flush to "backing_dev" and free memory used. ``` sudo crontab -l # run every 15 minutes to push idle and huge pages to backing_dev */15 * * * * sh -xvc 'szr=/sys/block/zram0; for i in idle huge; do echo $i > $szr/writeback && sleep 1; done && echo all > $szr/idle' 2>&1 | logger -t cron.job.zramswap.writeback ``` ### Allow GUI hibernate Lubuntu-22.04.3 "Hibernate" ( mainmenu -> Leave -> Hibernate, lxqt-leave --hibernate) button not work. Run command "lxqt-leave" in lxqt qterminal. "lxqt-leave --hibernate" failed: systemd: "CanHibernate" = "no" ``` man polkit man pklocalauthority pkaction | grep -i -e upower -e hibernate sudo grep -rni hibernat /var/lib/polkit-1 /etc/polkit-1/ sudo cat /var/lib/polkit-1/localauthority/50-local.d/disable-hibernate.pkla sudo cat /var/lib/polkit-1/localauthority/10-vendor.d/com.ubuntu.desktop.pkla # run in qterminal pkcheck --action-id org.freedesktop.login1.hibernate --detail --process $$ # after change polkit config sudo systemctl restart polkit.service # sudo systemctl restart dbus.service # run hibernate directly sudo /usr/lib/systemd/systemd-sleep hibernate # enable printk while do hibernate cat /proc/sys/kernel/printk grep -rn . /sys/module/printk/parameters/ echo 1 | sudo tee /sys/module/printk/parameters/ignore_loglevel echo 0 | sudo tee /sys/module/printk/parameters/console_suspend ``` Crontab job to enable: lxqt-leave --hibernate ``` sudo crontab -e sudo crontab -l # enable hibernate @reboot sh -xvc 'f=/var/lib/polkit-1/localauthority/50-local.d/disable-hibernate.pkla; [ -e $f ] && sed -i -e "s/^ResultActive=no/ResultActive=yes/" $f; f=/var/lib/polkit-1/localauthority/10-vendor.d/com.ubuntu.desktop.pkla; sed -i -e "/^\[[^]]\+\shibernate\s/,/^ResultActive=no/{s/ResultActive=no/ResultActive=yes/}" $f' 2>&1 | logger -t cron.job.reboot.enable-hibernate ``` Ubuntu-24.04 needs more to enable hibernate. ``` # show hibernate menu item sudo sed -i -e 's/^NoDisplay=true/#&/' /usr/share/Lubuntu/applications/lxqt-hibernate.desktop # sudo crontab -l @reboot sh -xvc 'f=/sys/power/resume && [ "0:0" = "$(cat $f)" ] || exit 0; resume=; resume_offset=0; eval $(grep -o "resume[^ ]*=[^ ]\+" /proc/cmdline) && resume=${resume#UUID=} && [ -n "$resume" ] && resume=$(readlink -f /dev/disk/by-uuid/$resume) && resume=${resume#/dev/} && [ -n "$resume" ] && f=/sys/power/resume_offset && echo $resume_offset | sudo tee $f && cat $f && f=/sys/power/resume && cat /sys/class/block/$resume/dev | sudo tee $f && cat $f' 2>&1 | logger -t cron.job.reboot.enable-hibernate-resume ``` If "Hibernate" button is still not visable, create a shortcut of command "lxqt-leave --hibernate". ## Grub boot iso file Grub 2.04 on uefi system (x86_64-efi) failed to boot iso file, "rmmod tpm" before run loopback to avoid this problem. ``` # web resouces https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=975835 https://askubuntu.com/questions/1520398/supergrub2-boot-iso-error-out-of-range ``` Lubuntu-18.04-desktop-amd64.iso cannot boot by grub2 (grub 2.02) loopback. "read out of range" error reported when load kernel image "/casper/vmlinuz" from loopback iso file in NTFS, and "unaligend pointer" error reported when load "/casper/initrd.lz". Avoid the problem, run chkdsk and defrag NTFS (may try ntfsfix), defragment iso file: move the iso file to other partition, then defragment the NTFS partition and move iso file back. ### Grub check file Check file size and checksum ``` # checksum in grub sha256sum (hd1,2)/path/file # compare files in grub, reports file size cmp /grub/grub.cfg /grub/custom.cfg ``` For UEFI boot, set gfxmode and gfxpayload for fixed screen resolution. ``` # web resource https://askubuntu.com/questions/761149/how-to-set-display-resolution-while-having-to-use-nomodeset-on-boot set gfxmode=1024x768 set gfxpayload=keep ``` ### Chainload grub core Chainload grub core in UEFI boot (grub_platform=efi). Chainload grub core not work in legacy BIOS boot (no-uefi, grub_platform=pc), use multiboot. ``` menuentry "Load other grub core" { set root=(hd0,1) set pc=/grub-2.12.i386-pc/grub/i386-pc/core.img set ec=/grub-2.12.x86_64-efi/grub/x86_64-efi/core.efi # if [ x"$grub_platform" = x"pc" ]; then echo multiboot $pc multiboot $pc else echo chainloader $ec chainloader $ec fi } # grub load grub.cfg echo Delete old loop. loopback -d loop set iso_path=/data/lubuntu-22.04.3-desktop-amd64.iso echo Setup loopback: $iso_path loopback loop (hd3)/$iso_path set root=(loop) echo root=$root export iso_path root configfile /boot/grub/loopback.cfg # # insert debug lines in boot menu echo grub_cpu: $grub_cpu echo grub_platform: $grub_platform echo prefix: $prefix echo config_file: $config_file echo gfxmode: $gfxmode echo gfxpayload: $gfxpayload echo root: $root echo iso_path: $iso_path # sleep --verbose --interruptible 9 if [ x"$config_file" = x"" ]; then set config_file=$prefix/grub.cfg; fi if sleep --verbose --interruptible 5; then true else configfile $config_file fi ``` ### Grub UART terminal Append serial terminal to grub. ``` # web resource https://www.gnu.org/software/grub/manual/grub/html_node/Serial-terminal.html menuentry "Enable UART terminal" { serial --unit=0 --speed=115200 serial --speed=115200 terminal_output --append serial terminal_input --append serial } ``` ### Install livecd grub Ubuntu livecd has both grub i386-pc and x86_64-efi. By default, only grub i386-pc is installed. Grub-efi can be installed from cdrom manually. For grub i386-pc, the process is similar. ``` dpkg --no-pager -l grub* ls /usr/lib/grub/i386-pc/ find /cdrom/ -name 'grub*.deb' apt policy grub-efi find /cdrom/ -name 'grub*.deb' -exec dpkg-deb --extract \{\} grub-efi \; ls grub-efi/usr/lib/grub/ ``` Install extracted x86_64-efi to ext4 partition. ``` sudo mount /dev/sdxxx d ( set -xv; history | tail -n 1; rtd=$(pwd)/d; btd=$rtd/grub-2.12.x86_64-efi; efid=$btd; d=/dev/sdxxx; itd=$(pwd)/grub-efi/usr; export PATH=$itd/bin:$itd/sbin:$PATH; sudo grub-install --boot-directory=$btd --target=x86_64-efi --efi-directory=$btd --directory=$itd/lib/grub/x86_64-efi --no-bootsector --removable --recheck --verbose --debug --force $d; echo rt=$? ) 2>&1 | tee grub-install.x86_64-efi.log sudo mv grub-install.x86_64-efi.log d/grub-2.12.x86_64-efi/ ls d/grub-2.12.x86_64-efi/ d/grub-2.12.x86_64-efi/EFI/ d/grub-2.12.x86_64-efi/grub/x86_64-efi/ ``` ``` # Generate core.efi with different partition, hd number can be omitted. # Grub files will be in dest dir "/grub-2.12.x86_64-efi.livecd/grub/" in dest partition. # ( set -xv; history | tail -n 1; p=1; d=$(pwd)/grub-efi/usr; export PATH=$d/bin:$d/sbin:$PATH; grub-mkimage --verbose --directory $d/lib/grub/x86_64-efi --prefix "(,$p)/grub-2.12.x86_64-efi/grub" --output core.p$p.efi --format x86_64-efi --compression auto fat ext2 ntfs part_msdos part_gpt lvm; echo rt=$? ) 2>&1 | tee grub-mkimage.x86_64-efi.log ( set -xv; history | tail -n 1; p=1; d=$(pwd)/grub-efi/usr; export PATH=$d/bin:$d/sbin:$PATH; grub-mkimage --verbose --directory $d/lib/grub/x86_64-efi --prefix "(,$p)/grub-2.12.x86_64-efi.livecd/grub" --output core.p$p.efi --format x86_64-efi --compression auto fat ext2 ntfs part_msdos part_gpt lvm; echo rt=$? ) 2>&1 | tee grub-mkimage.x86_64-efi.log ls -l core.p1.efi ``` Copy installed grub files to new partition. ``` sudo mount /dev/sdxp1 b sudo cp -a d/grub-2.12.x86_64-efi/ b/grub-2.12.x86_64-efi.livecd sudo cp -a b/grub-2.12.x86_64-efi.livecd/grub/x86_64-efi/core.efi b/grub-2.12.x86_64-efi.livecd/grub/x86_64-efi/core.efi.old cat core.p1.efi | sudo tee b/grub-2.12.x86_64-efi.livecd/grub/x86_64-efi/core.efi >/dev/null ls -l b/grub-2.12.x86_64-efi.livecd/grub/x86_64-efi/core.efi* cat /cdrom/boot/grub/*.cfg | sudo tee -a b/grub-2.12.x86_64-efi.livecd/grub/grub.cfg ``` ### Build grub Grub source code ``` https://git.savannah.gnu.org/cgit/grub.git/ https://ftp.gnu.org/gnu/grub/ https://git.savannah.gnu.org/git/grub.git ``` "grub-2.12.tar.gz" download from "https://ftp.gnu.org/gnu/grub/" lacks file "grub-2.12/grub-core/extra_deps.lst". It is available from git. Run build ``` # build grub i386_pc ( set -xv; history| tail -n1; d=grub-2.12; wd=$(pwd) && { f=$d/grub-core/extra_deps.lst; [ -s $f ] || echo "depends bli part_gpt" | tee $f; } && tr=i386-pc && mkdir -p $d.$tr.build && cd $d.$tr.build && sh $wd/$d/configure && make -w V=1 && make -w V=1 install DESTDIR=$wd/$d.$tr._install; echo rt=$? ) 2>&1 | tee build.grub.i386-pc.log # build grub x86_64-efi ( set -xv; history| tail -n1; d=grub-2.12; wd=$(pwd) && { f=$d/grub-core/extra_deps.lst; [ -s $f ] || echo "depends bli part_gpt" | tee $f; } && tr=x86_64-efi && mkdir -p $d.$tr.build && cd $d.$tr.build && sh $wd/$d/configure --target=x86_64 --with-platform=efi && make -w V=1 && make -w V=1 install DESTDIR=$wd/$d.$tr._install; echo rt=$? ) 2>&1 | tee build.grub.x86_64-efi.log ``` Build result ``` diff -rq grub-2.12.i386-pc._install grub-2.12.x86_64-efi._install # merge installs of i386-pc and x86_64-efi ( set -xv; mkdir grub-2.12._install && mv grub-2.12.x86_64-efi._install grub-2.12._install && cp -nav grub-2.12.i386-pc._install/usr/local grub-2.12._install/usr/ && cp -a build.grub.x86_64-efi.log build.grub.i386-pc.log; echo rt=$? ) 2>&1 | tee merge.install.log ``` Install built grub ``` # install built grub x86_64-efi to EFI partition sudo mount -t vfat /dev/sdx1 p1 ( set -xv; history | tail -n 1; rtd=$(pwd)/p1; btd=$rtd/grub-2.12.x86_64-efi; efid=$btd; d=/dev/sdx1; itd=$(pwd)/grub-2.12._install/usr/local; export PATH=$itd/bin:$itd/sbin:$PATH; sudo grub-install --boot-directory=$btd --target=x86_64-efi --efi-directory=$btd --directory=$itd/lib/grub/x86_64-efi --no-bootsector --removable --recheck --verbose --debug $d; echo rt=$? ) 2>&1 | tee grub-install.x86_64-efi.log # check installed files find e/grub-2.12.x86_64-efi/ -type f -not -name '*.mod' cat grub/x86_64-efi/load.cfg # install built grub i386-pc sudo mount -t vfat /dev/sdx5 p5 ( set -xv; history | tail -n 1; rtd=$(pwd)/p5; btd=$rtd/grub-2.12.i386-pc; d=/dev/sdx5;itd=$(pwd)/grub-2.12._install/usr/local; export PATH=$itd/bin:$itd/sbin:$PATH; sudo grub-install --boot-directory=$btd --target=i386-pc --directory=$itd/lib/grub/i386-pc --no-bootsector --removable --recheck --verbose --debug $d; echo rt=$? ) 2>&1 | tee grub-install.i386-pc.log # Add "--force" option if failed on some warning. ``` ### Copy or move grub installed Copy or move installed grub to other partition, need to generate core.img or core.efi with new partition number. ``` # Generate core.img and core.efi with different partition, hd number can be omitted. # Grub files will be in dest dir "/grub-2.12.i386-pc/grub/" in dest partition. # ( set -xv; history | tail -n 1; p=1; d=$(pwd)/grub-2.12._install/usr/local; export PATH=$d/bin:$d/sbin:$PATH; grub-mkimage --verbose --directory $d/lib/grub/i386-pc --prefix "(,$p)/grub-2.12.i386-pc/grub" --output core.p$p.img --format i386-pc --compression auto ext2 fat ntfs part_gpt part_msdos biosdisk lvm; echo rt=$? ) 2>&1 | tee grub-mkimage.i386-pc.log # ( set -xv; history | tail -n 1; p=1; d=$(pwd)/grub-2.12._install/usr/local; export PATH=$d/bin:$d/sbin:$PATH; grub-mkimage --verbose --directory $d/lib/grub/x86_64-efi --prefix "(,$p)/grub-2.12.x86_64-efi/grub" --output core.p$p.efi --format x86_64-efi --compression auto fat ext2 ntfs part_msdos part_gpt lvm; echo rt=$? ) 2>&1 | tee grub-mkimage.x86_64-efi.log ``` ### Install gurb bootsector Before change MBR, make sure to backup it. Use partition tool to get posible boot loader sectors after MBR, and backup them all. Here assuem they use first 2048 sectors along with main partition table from sector 0. ``` # backup bootsector MBR sudo dd if=/dev/sdc of=hd.msdos.mbr bs=512 count=2048 sudo cp -a hd.msdos.mbr p5/boot/grub/ sudo dd if=/dev/sdc5 of=hd.msdos.p5.mbr bs=512 count=1 sudo cp -a hd.msdos.p5.mbr hd.msdos.p5.mbr.old sudo cp -a hd.msdos.p5.mbr.old p5/boot/grub/ od -t x1a hd.msdos.p5.mbr ``` Install grub bootsector to partition 5 MBR If file system comes immediatelly after sector 0, there is not enough sectors to embed grub "core.img". ``` # install grub bootsector to MBR sudo mount /dev/sdx5 p5 # grub-probe --verbose --device-map=device.map --device /dev/sdc5 ( set -xv; history | tail -n 1; k=sdx; p=7; d=$(pwd)/grub-2.12._install/usr/local; export PATH=$d/bin:$d/sbin:$PATH; sudo grub-mkdevicemap --verbose --no-floppy --device-map=device.map && sudo parted /dev/$k -- p unit s p && cat device.map && sudo grub-bios-setup --verbose --directory=p5/grub-2.12.i386-pc/grub/i386-pc --device-map=device.map /dev/$k$p; echo rt=$? ) 2>&1 | tee grub-bios-setup.log # add "--force" if not-used blocks at the beginning of partition are not enough to embed core.img. ``` ## initrd Casper script file (initrd/main/scripts/casper, /usr/share/initramfs-tools/scripts/casper) in initrd has a bug, patch it to clean "/tmp/debconf.xxxx" folder. "zcat" is missing in initrd of Ubuntu-server-22.04.5 and Lubuntu-22.04.3, so "casper-sn.cpio.gz" not work. Patch to use "gzip -dc" instead. Apply "initramfs.casper.patch" to casper cripts in folder "/usr/share/initramfs-tools/scripts/". Then rebuild initrd by "mkinitramfs". Or, use "unmkinitramfs" extract files from "/cdrom/casper/initrd". Apply patch and then create initrd as below. Or, patch main.cpio in initrd. ### initramfs.split.sh Run this script to split initrd, codes are copied from: /usr/bin/unmkinitramfs ``` sh initramfs.split.sh /cdrom/casper/initrd ls -l early.*.cpio main.cpio* mv main.cpio.xx main.cpio.zst zstd -kd main.cpio.zst ``` ### patch.main.cpio.sh After patch, the initrd allows writable-backing and casper-sn on fs more than vfat, by adding kernel parameter "wbfstype=vfat,ex2,ext4" to "linux" line in file "grub.cfg". The patch deletes "/root/$DEBCONF_TMPDIR", and allows resume from hibernate. ``` sh patch.main.cpio.sh main.cpio ``` After patch, create initrd. ``` xz --check=crc32 --threads=0 main.cpio.patched ( set -xv; cat early.?.cpio main.cpio.patched.xz > rd && ls -l rd; echo rt=$? ) ``` You can extract files from main.cpio if you want. ``` # extract main.cpio.xx file main.cpio.xx mv main.cpio.xx main.cpio.zst ( set -xv; mkdir -p initrd && mkdir initrd/main && cd initrd/main && zstd -dc ../../main.cpio.zst | sudo cpio -id; echo rt=$? ) ``` ### Create inird cpio file Create inird cpio file from extracted folder. ``` sudo sh -xvc 'wd=$(pwd) && cd $1 && find . | LC_ALL=C sort | cpio --quiet -R 0:0 --reproducible -o -H newc > "$wd/$(echo $1 | tr "/" "-").cpio"; echo rt=$?' x initrd/main ``` ### Create initrd Create initrd with early.cpio and main.cpio. ``` # compress main cpio and prepend early cpio xz --check=crc32 --threads=0 initrd-main.cpio ( set -xv; cat early.?.cpio initrd-main.cpio.xz > rd ) # test it ( set -xv; history | tail -n 1; lsinitramfs -l rd && unmkinitramfs -v rd d && sudo diff -rq initrd/ d/; echo $? ) 2>&1 | tee t.log ``` ### initrd script files and functions ``` /etc/casper.conf initrd/main/conf/conf.d/default-boot-to-casper.conf /usr/share/initramfs-tools/scripts/casper /usr/share/initramfs-tools/scripts/casper-bottom/25adduser /usr/lib/user-setup/user-setup-apply /usr/share/debconf/confmodule diff initrd/main/init /usr/share/initramfs-tools/init less /var/log/casper.log # casper on system shutdown systemctl cat casper.service # /usr/sbin/casper-stop /usr/sbin/casper-snapshot /usr/share/initramfs-tools/scripts/casper-helpers /lib/lsb/init-functions # root_snapshot_label home_snapshot_label try_snap () do_snap_copy () ``` Livecd initrd boot casper call tree ``` # init process is initrd/main/init # # config files in folder "initrd/main/conf/conf.d" are invoked by initrd/main/init # grep -rn BOOT=casper initrd/main/conf/ initrd/main/conf/conf.d/default-boot-to-casper.conf:2: export BOOT=casper # In file initrd/main/init: . /scripts/${BOOT} mountroot read_fstab_entry # In file initrd/main/scripts/casper, function mountroot() calls: find_livefs mount_images_in_directory() -> setup_overlay(); setup_auto_log_persistence run_scripts /scripts/casper-bottom cp /etc/casper.conf "${rootmnt}/etc/" # setup_overlay() calls: find_cow_device() - root_persistence_label(); mount .... /dev/xxx /cow get_backing_device() -> setup_loop() mount -t squashfs -o ro,noatime /dev/loop0 /ubuntu-server-minimal.squashfs mount -t overlay ... /cow /root try_snap() -> do_snap_copy() -> cp -a /mnt/tmpsnap/* /root mount -o move /ubuntu-server-minimal.squashfs /root/rofs # In file initrd/main/scripts/casper-functions, function load_confmodule(): . /root/usr/share/debconf/confmodule # # list files invoke /usr/share/debconf/confmodule grep -rn load_confmodule initrd/main/scripts/ # list break points, "break=xxx" of kernel parameter at boot grep -rn maybe_break initrd/main/scripts/ initrd/main/init ``` ## Lxd container and docker Ubuntu live server is lxd ready. But docker needs to be installed manaully. Recommend to use another lvm as container storage. ### Container working folder Install systemd mount files and enable them. "bind" mount can also be used for lxd working folder. For "overlay" mount, "upper" and "work" folder shall be created manually before the first use. Stop all lxd related services and sockets before change mount. Copy systemd mount files to folder "/etc/systemd/system/" and enable them. ``` mnt-data.mount var-snap-lxd-common-lxd.mount ``` Change docker container working folder. ``` sudo systemctl stop docker.service docker.socket sudo mkdir -p /mnt/data/var_lib_docker printf "%s\n" "{" '"data-root": "/mnt/data/var_lib_docker",' '"storage-driver": "overlay2"' "}" | sudo tee -a /etc/docker/daemon.json sudo mv /var/lib/docker/ /var/lib/docker.old ``` ### Create container image and run it Install binfmt support for various CPU arch. ``` sudo apt-get install qemu-user-static binfmt-support ``` There are many online images ready to run. Here create image from scratch. Change "--arch=armhf" as desired, i.e. "--arch=riscv64". Delete "http://10.100.93.1:3142/" if not use apt-cacher server. ``` # if you are in a regular lxc container, it shall run with mknod allowed: lxc config set ubtsvr security.syscalls.intercept.mknod=true # prepare rootfs ( set -xv; history | tail -n 1; date -Is; systemd-detect-virt; uname -a; lsb_release -a; sudo debootstrap --verbose --keyring=/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg --foreign --arch=armhf --include=autoconf,automake,libtool,bash-completion,bc,bsdmainutils,btrfs-progs,build-essential,ca-certificates,cloud-guest-utils,command-not-found,curl,device-tree-compiler,dosfstools,git,gnupg,initramfs-tools,less,libinline-files-perl,libnss-mdns,libnss-systemd,libpam-systemd,linux-base,locales,man-db,net-tools,openssh-server,overlayroot,patch,rsync,ssl-cert,sudo,systemd-timesyncd,tree,u-boot-tools,unzip,vim,wget,zstd --components=main,universe,multiverse jammy rootfs http://10.100.93.1:3142/ports.ubuntu.com/ && sudo chroot rootfs debootstrap/debootstrap --verbose --second-stage && sudo chroot rootfs sh -xvc 'adduser user; adduser user sudo; apt clean; rm -fr /var/cache/* /var/lib/apt/lists/*'; echo rt=$? ) 2>&1 | tee debootstrap.log # create metadata.yaml for lxc ( set -xv; ( echo architecture: "amd64"; echo creation_date: $(date +%s) ) | tee metadata.yaml && tar -cJf metadata.tar.xz metadata.yaml && rm metadata.yaml; echo rt=$? ) ``` For docker image, rootfs files must in folder ".". For lxc image, rootfs files must in folder "rootfs" if tar contains metadata.yaml for "lxc image import metafata.and.rootfs.tar", while rootfs files must in folder "." for "lxc image import metadata.tar rtfs.tar". ``` # import docker image docker image import rtfs.tar.xz armhf # run docker container docker run -it --rm --name armv7l --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:rw armhf /sbin/init docker run -it --rm --name riscv64 --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:rw riscv64 /sbin/init # run privileged container docker run -it --rm --name armv7l --privileged -v /mnt/data/work:/mnt/work armhf /sbin/init # import lxc image lxc image import metadata.tar.xz rootfs.tar.xz --alias armv71 # run lxc container lxc launch armv7l armhf --console # check files in container lxc file edit armhf/etc/hosts lxc file pull armhf/bin/bash /tmp/ -v # change init process from "/sib/init" to "/bin/bash" lxc config set systemd raw.lxc 'lxc.init.cmd = /bin/bash' ``` Systemd-networkd failed in container. ``` systemd-networkd[226]: Could not create manager: Protocol not supported "systemd-networkd.socket" starts OK but failed when "systemd-networkd.service" failed. ``` Workaround for this problem: use static ip ``` # sudo crontab -l @reboot /bin/sh -xvc 'sleep 6 && ip addr add dev eth0 10.100.93.198/24 && ip route add default via 10.100.93.1; f=/etc/resolv.conf; [ -s $f ] || { systemctl stop systemd-resolved.service; systemctl disable systemd-resolved.service; rm -f $f; ( echo nameserver 10.100.93.1; echo search lxd ) > $f; }' 2>&1 | logger -t cron.job.reboot.1 ``` ### Lxd dns record Host can add host-record for containers dns query. "host-record" is not supported by old version of dnsmasq, followed by "local" to prevent forward query to upstream server. Client DNS query slow down while dnsmasq try to reach upstream server configured in "/etc/resolv.conf" when no internet available. ``` ( set -xv; s="$(printf "%s\n" "host-record=my.fileserver.com,192.168.58.211" "local=/fileserver.com/")"; lxc network set lxdbr0 raw.dnsmasq="$s"; echo rt=$? ) ``` ### Mount in container By default, privileged (unconfined) container allow mount and umount. If configured properly, regular (no-privilegded) container can mount and umount. Umount returns quickly but "/proc/mounts" updated slowly in container. Wait a second after umount before check if umounted. Device mapper does not work properly in privileged container. Use "losetup --partscan". "losetup --partscan" in privileged container works and "lsblk" can see partitions, but dev node needs manually create. ``` sudo sh -xvc 'f=$1; p=$(losetup --find); t=${p#/dev/}p1; { [ -b /dev/$t ] && [ -e /dev/mapper/${t} ]; } || ( kpartx -d $p; losetup -d $p; sleep 1; [ -e $p ] || { mknod $p b 7 ${p#*loop} || exit $?; }; losetup --partscan $p "$f" && sleep 1 && m=$(lsblk -r $p | grep $t | cut -d " " -f 2) && n=${m#*:} && m=${m%:*} && { [ -z "$m" ] && m=253 || true; } && { [ -z "$n" ] && n=0 || true; } && cd /dev/ && rm -f $t && mknod $t b $m $n && cd mapper && rm -f $t && ln -s ../$t . )' x image ``` ## Run livecd in container Livecd squashfs images provide full set of rootfs files, which can be used as rootfs of container. This avoids to install and keep copy of rootfs for each container, so that a host can easily run dozens of containers with 2G memory (with zram swap) and even no disk usage. ### Generate man pages of snap lxc Ubuntu server 20.04 and newer livecd provides snap lxd. ``` # ubuntu snapd lxc man pages, man page tile: LXD - Command line client ( set -xv; mkdir man1 && cd man1 && lxc manpage . && find . -not -type d | while read -r i; do i=${i#./}; sed -i -e ':a;s/\(\\fBlxc[^-(]*\)-\([^(]\+(1)\\fP\)/\1.\2/g;ta' "$i" && gzip -v "$i" || exit $?; done; echo $? ) 2>&1 | tee m.log ``` ### Prepare upperdir for overlayfs Use overlayfs to keep changes for each container. Btrfs wastes space for partition smaller than 256M, use ext2 or zfs. Old zfs does not support overlayfs, use OpenZFS 2.2 and newer for overlayfs upperdir. ``` # create upperdir # create ext2 as upperdir sudo sh -xvc 't=$1; f=$t.ext2; [ -e $f ] && { echo "File exists: $f"; exit 1; }; dd if=/dev/zero of=$f bs=128M count=1 && mkfs.ext2 $f && mkdir -p $t && mount -t ext2 $f $t && mkdir -p $t/upper $t/work; echo $?' x /mnt/data/lxd_ubt-sr3_data # expand ext2 on line, may fail when container is using it sudo sh -xvc 't=$1; n=$2; f=$t.ext2; i=$( stat -c %s $f ) && j=$(( i / 128 / 1024 / 1024 )) && { [ $(( j * 128 * 1024 * 1024 )) -lt $i ] && j=$(( j + 1 )) && n=$(( n - 1 )) || true; } && dd if=/dev/zero of=$f bs=128M seek=$j count=$n || exit $?; i=$( losetup -a | grep -e $f ) && i=${i%%:*} && lsblk $i && { grep -e " $t " /proc/mounts && df $t || true; } && losetup -c $i && lsblk $i; resize2fs $f && { grep -e " $t " /proc/mounts && df $t || true; }; echo $?' x /mnt/data/lxd_ubt-sr3_data 1 # # umount before shrink sudo umount -v /mnt/sr3.rt* /mnt/data/lxd_ubt-sr3_data # # shrink ext2 after umount sudo sh -xvc 't=$1; grep -e " $t " /proc/mounts && { echo "File mount at: $t"; exit 1; }; f=$t.ext2; e2fsck -v -f $f && resize2fs -M $f && e2fsck $f && i=$( dumpe2fs $f | grep -e "Block count:" -e "Block size:" | { read -r i && read -r j && i=${i##* } && j=${j##* } && echo $(( i * j )); } ) && [ -n "$i" ] && dd if=/dev/null of=$f bs=$i seek=1 count=0 && e2fsck -n $f; echo $?' x /mnt/data/lxd_ubt-sr3_data # sync overlayfs upperdir online after change in upperdir sudo mount -o remount /mnt/sr3.rtfs/ # create btrfs as upperdir sudo sh -xvc 't=$1; d=$2; [ -e $t.btrfs ] || { dd if=/dev/zero of=$t.btrfs bs=256M count=1 && mkfs.btrfs $t.btrfs || exit $?; }; mkdir -p $t && { grep -e " $t btrfs " /proc/mounts || mount -t btrfs $t.btrfs $t; } && btrfs subvolume create $t/@$d && mkdir -p $t/@$d/upper $t/@$d/work && btrfs property set $t/@$d/upper compression zstd; echo $?' x /mnt/data/lxd.data sr3 # expand btrfs on line # btrfs compress data when cpu idle, so append enough space before install packages and shrink after install # sudo sh -xvc 't=$1; n=$2; i=$( stat -c %s $t.btrfs ) && j=$(( i / 128 / 1024 / 1024 )) && [ $j -ne 0 ] && [ $i -eq $(( j * 128 * 1024 * 1024 )) ] && dd if=/dev/zero of=$t.btrfs bs=128M seek=$j count=$n && i=$( losetup -a | grep -e $t.btrfs ) && i=${i%%:*} && losetup -c $i && mkdir -p $t && { grep -e " $t btrfs " /proc/mounts || mount -t btrfs $t.btrfs $t; } && btrfs filesystem resize max $t && lsblk $i; echo $?' x /mnt/data/lxd.data 1 # check usage btrfs filesystem usage /mnt/data/lxd_ubt-sr3_data # shrink btrfs on line sudo btrfs filesystem resize -128M /mnt/data/lxd_ubt-sr3_data sudo sh -xvc 't=$1; i=$(btrfs filesystem usage -b $t | grep -e "Device size:") && i=${i##* } && [ -n "$i" ] && dd if=/dev/null of=$t.btrfs bs=$i seek=1 count=0 && i=$( losetup -a | grep -e $t.btrfs ) && i=${i%%:*} && lsblk $i && losetup -c $i && lsblk $i; echo $?' x /mnt/data/lxd_ubt-sr3_data ``` ### Prepare rootfs for container If cdrom changed after upper folder subvolume created, run command below to match them. ``` # search and rename matched subvolume sudo sh -xvc 'n=$1; d=$2; [ -z "$n" ] && n=/mnt/data/lxd_ubt-${d}_data; t=/mnt/$d; getmnt() { grep $2 -e " $1 " /proc/mounts; }; getmnt $t || { mkdir -p $t && mount /dev/$d $t || exit $?; }; getmnt $n || { mkdir -p $n && mount -v -t btrfs $n.btrfs $n || exit $?; }; find $n/ -maxdepth 1 -mindepth 1 -type d -name "@sr*" | while read -r i; do cmp $t/.disk/info $i/upper/etc/cdrom.info && { j=${i##*/} && [ "@$d" = "$j" ] && exit; k=$n/@$d && [ -d "$k" ] && { mv $k $k- || exit $?; }; mv $i $k || exit $?; }; done ' x /mnt/data/lxd.data sr3 ``` Run command below to overlay mount rootfs for container. ``` # mount cdrom and squashfs, overlay # # Ubuntu-18.04-server "lxc.rootfs.path = overlayfs" uses workdir "olwork" instead of "work". # # mount ext2 overlay sudo sh -xvc 'n=$1; d=$2; [ -z "$n" ] && n=/mnt/data/lxd_ubt-${d}_data; t=/mnt/$d; getmnt() { grep $2 -e " $1 " /proc/mounts; }; getmnt $t || { mkdir -p $t && mount /dev/$d $t || exit $?; }; w=$( find /mnt/$d/casper/ -maxdepth 1 -name '*.squashfs' -a ! -name '*generic-hwe*' | sed -e 's/.squashfs//' | sort -r | while read -r f; do i=${f##*/}; i=${i##*.}; t=/mnt/$d.$i; getmnt $t -q || { mkdir -p $t && mount -t squashfs $f.squashfs $t || exit $?; }; echo -n ":$t"; done || exit $? ) || exit $?; [ -z "$w" ] && exit 1; t=/mnt/$d.rtfs; getmnt $t && exit; [ -s $n.ext2 ] && { getmnt $n || { mount -v $n.ext2 $n || exit $?; }; }; i=/mnt/$d/.disk/info; j=$n/upper/etc/cdrom.info; [ -e ${j%/*} ] && { cmp $i $j || exit $?; } || { mkdir -p ${j%/*} && cp -a $i $j || exit $?; }; mkdir -p $n/upper $n/work $n/olwork $t; w="lowerdir=${w#:},upperdir=$n/upper,workdir=$n/work"; mount -t overlay -o "$w" overlay $t && ls -la $t/ $n/upper && getmnt $t && true umount -v $t; echo $?' x "" sr3 # # mount btrfs overlay sudo sh -xvc 'n=$1; d=$2; t=/mnt/$d; getmnt() { grep $2 -e " $1 " /proc/mounts; }; getmnt $t || { mkdir -p $t && mount /dev/$d $t || exit $?; }; w=$( find /mnt/$d/casper/ -maxdepth 1 -name '*.squashfs' -a ! -name '*generic-hwe*' | sed -e 's/.squashfs//' | sort -r | while read -r f; do i=${f##*/}; i=${i##*.}; t=/mnt/$d.$i; getmnt $t -q || { mkdir -p $t && mount -t squashfs $f.squashfs $t || exit $?; }; echo -n ":$t"; done || exit $? ) && [ -z "$w" ] && exit $?; x=${n%/*}/lxd_ubt-${d}_data; t=/mnt/$d.rtfs; getmnt $t && exit; [ -s $n.btrfs ] || exit $?; getmnt $x || { mkdir -p $x; mount -v -t btrfs -o subvol=@$d $n.btrfs $x || exit $?; }; i=/mnt/$d/.disk/info; j=$x/upper/etc/cdrom.info; [ -e ${j%/*} ] && { cmp $i $j || exit $?; } || { mkdir -p ${j%/*} && cp -a $i $j || exit $?; }; mkdir -p $x/upper $x/work $x/olwork $t; w="lowerdir=${w#:},upperdir=$x/upper,workdir=$x/work"; mount -v -t overlay -o "$w" overlay $t && ls -la $t/ $x/upper && getmnt $t; echo $?' x /mnt/data/lxd.data sr3 # check mount ps -few | grep fuse losetup -a | grep -e sr3 grep -e sr3 /proc/mounts sudo lsns | grep lxd ( set -xv; sudo nsenter -t $( sudo lsns | grep " root .*/lxd/.*/daemon.start" | tr -s " " "\t" | cut -f 4 ) -m grep -e sr3 /proc/mounts ) ``` ### Create container Rootfs folder of lxc container must exist before start it. If rootfs folder not exist, lxc container fails to start, hook.pre-start is not called. Workaround is using an empty image and dir storage pool to create container, or create it manually. ``` # create dir storage pool lxc storage list lxc -v storage create folder dir # create empty lxc image # If rootfs folder not exist, lxc container fails to start, hook.pre-start is not called. sh -xvc '( echo "architecture: \"amd64\""; echo "creation_date: $(date +%s)" ) | tee metadata.yaml && mkdir rootfs && tar --owner=root --group=root -cf img.tar metadata.yaml rootfs; echo $?' # tar -tvf img.tar # import image lxc image import --alias emptyrootfs img.tar lxc image list # create an emptyrootfs container with dir storage pool lxc init --storage folder emptyrootfs ubt-sr3 # create an empty container with dir storage pool lxc init --storage folder --empty ubt-sr3 lxc list lxc config show ubt-sr3 lxc list -c ns46tSDbPdp # # check lxc container storage usage lxc storage volume list folder lxc storage volume ls -c etndcuU default # set lxc container description lxc config set ubt-sr3 description ubt2204xdg --property # skip this if container is created from emptyrootfs image # create container rootfs folder, if it does not exist, cannot start # If rootfs folder not exist, lxc container fails to start, hook.pre-start is not called. sudo sh -xvc 'n=$1; d=/var/snap/lxd/common/lxd/containers/$n/rootfs; sudo nsenter -t $( sudo lsns | grep -e " root .*/lxd/.*/daemon.start" | tr -s " " "\t" | cut -f 4 ) -m mkdir -p $d' x ubt-sr3 # check container rootfs folder sudo sh -xvc 'n=$1; nsenter -t $( lsns | grep -e " root .*/lxd/.*/daemon.start" | tr -s " " "\t" | cut -f 4 ) -m ls -la /var/snap/lxd/common/lxd/containers/$n/rootfs' x ubt-sr3 ``` ### Configure overlay rootfs for lxc previleged container "lxc.rootfs.path = overlayfs:/lower:/upper" only works for privileged container. ``` lxc config set ubt-sr3 security.privileged true lxc config set ubt-sr3 security.nesting true # # set lxc container rootfs from ubuntu-22.05.5-live-server iso file, multi lowerdir sh -xvc 'n=$1; shift; printf "%s\n" "$@" | lxc config set $n raw.lxc -; echo $?' x ubt-sr3 "lxc.rootfs.path = overlayfs:/mnt/sr3.installer:/mnt/sr3.ubuntu-server:/mnt/sr3.ubuntu-server-minimal:/mnt/data/lxd_ubt-sr3_data/upper" ``` ### Prepare overlay rootfs for lxc unprevileged container "lxc.rootfs.path = overlayfs:/lower:/upper" cannot work for unprevileged container. "lxc.lxcrootfs.path = dir:/foder" needs idmap to avoid recursively chown the rootfs of the container. Only linux kernel 5.19 and newer kernel support "lxc.rootfs.options = idmap=container" for overlayfs idmapped mount (mount option: X-mount.idmap). Squashfs needs linux kernel 6.2 to support idmapped mount. FAT and ext4 idmapped mount needs linux kernel 5.12. For old kernel, use "bindfs" to do uimapping and gipmapping. "X-mount.idmap" needs mount in util-linux-v2.39+, Ubuntu-22.04.5 installs mount v2.37.2. Use mount-idmapped tool to do "X-mount.idmap", check "idmapped" in file /proc/mounts to confirm idmap works. "mount-idmapped" does not support overlayfs. ``` https://github.com/brauner/mount-idmapped ``` lxc.hook.pre-start and lxc.hook.pre-mount can be used to check container internal ownership and permission status on rootfs files after idmap. Bindfs is used to do idmap for unprevileged container. ``` # install bindfs # for linux kernel 5.19 and newer, can skip this and use "lxc.rootfs.options = idmap=container" # # add "universe" to /etc/apt/sources.list #sudo add-apt-repository universe sudo vi /etc/apt/source.list sudo apt update sudo apt install bindfs # # fuse user_allow_other echo user_allow_other | sudo tee -a /etc/fuse.conf ``` #### Bind mount and map uid and gid for container root ``` # bind mount container rootfs in lxd daemon namespace, not work, lxc remaps uid and gid when container start # "Too many open files" errors (EMFILE) when "ulimit -n" is small for bindfs process, when search (find) deeply in bindfs folder. Unprevileged container also has this problem when use the bindfs folder. # https://github.com/nix-community/impermanence/issues/206 # sudo sh -xvc 'n=$1; s=$2; t=$3; [ -z "$s" ] && s=/mnt/${n#*-}.rtfs; [ -z "$t" ] && t=/mnt/${n#*-}.rtovl; getmnt() { grep -e " $1 " /proc/mounts; }; getmnt $t && exit; uo=""; go=""; i="$( lxc config get $n volatile.idmap.next | tr -s "{}[]" "\n" | grep -e "Hostid" | while read -r i; do k=uo; j=${i#\"Isuid\":true}; [ "$i" = "$j" ] && k=go; j=${j#*Hostid\":}; j=${j%%,*}; echo "$k=$j"; done )" && eval "$i" && [ -n "$uo" ] && [ -n "$go" ] && lowerdir="" && upperdir="" && eval "$( getmnt $s | grep -o -e ",lowerdir=.*,workdir=" | tr -s "," "\n" )" && [ -n "$lowerdir" ] && [ -n "$upperdir" ] && mg=$s.mg && { echo "$lowerdir" | grep -e ":" || mg=$lowerdir; } && { getmnt $mg || { mkdir -p $mg && mount -v -t overlay -o lowerdir=$lowerdir overlay $mg; }; } && i=$s.bd && { getmnt $i || { { [ $( ulimit -n ) -lt 65535 ] && ulimit -n 65535 || true; } && mkdir -p $i && bindfs -o allow_other,suid,dev --uid-offset=$uo --gid-offset=$go $mg $i; }; } && mount -v -o remount,ro $s && chown $uo:$go $upperdir $upperdir/etc && chown -vR $uo:$go $upperdir/../*work && ls -laR $upperdir/../*work && mkdir -p $t && mount -v -t overlay -o ro,lowerdir=$s.bd,upperdir=$upperdir,workdir=${upperdir%/upper}/work overlay $t && ls -la $t/ $t/var/ && ls -laR $upperdir/../*work && echo "lxc.rootfs.path = overlayfs:$s.bd:$upperdir" | lxc config set $n raw.lxc -; echo $?' x ubt-sr3 # mount-idmapped, work in Ubuntu-22.04.5-server hwe kernel linux 6.8.0-40-generic sudo sh -xvc 'n=$1; s=$2; t=$3; [ -z "$s" ] && s=/mnt/${n#*-}.rtfs; [ -z "$t" ] && t=/mnt/${n#*-}.rtovl; getmnt() { grep -e " $1 " /proc/mounts; }; getmnt $t && exit; uo=""; i="$( lxc config get $n volatile.idmap.next | grep -o "{\"Isuid\":true,[^}]\+}" | tr -d "{}\"" | tr -s "," "\n" | tr -s ":" "=" | grep -e "^Hostid=" )" && i="uo=${i#*=}" && eval "$i" && [ -n "$uo" ] && lowerdir="" && upperdir="" && eval "$( getmnt $s | grep -o -e ",lowerdir=.*,workdir=" | tr -s "," "\n" )" && [ -n "$lowerdir" ] && [ -n "$upperdir" ] && echo "$lowerdir" | tr -s ":" "\n" | while read -r i; do getmnt $i.idmp || { mkdir -p $i.idmp && mount-idmapped --map-mount "b:0:$uo:65535" $i $i.idmp || exit $?; }; done && mount -v -o remount,ro $s && chown $uo:$uo $upperdir $upperdir/etc && chown -R $uo:$uo $upperdir/../work && i=${upperdir%/upper} && j=$( echo "$lowerdir" | tr -s ":" "\n" | while read -r i; do printf "%s:" $i.idmp; done ) && j=${j%:} && [ -n "$j" ] && mkdir -p $t && mount -v -t overlay -o ro,lowerdir=$j,upperdir=$i/upper,workdir=$i/work overlay $t && ls -la $s/ && ls -la $t/ $t/var/ && ls -l $i && ls -laR $i/work && echo "lxc.rootfs.path = overlayfs:$j:$i/upper" | lxc config set $n raw.lxc -; echo $?' x ubt-sr3 # enable nesting if need snapd in container lxc config set ubt-sr3 security.nesting true # After lxc container started, host can umount all sr3 related folders. Keep idmapped mount and upperdir mount to enable reboot. sudo sh -xvc 'd=$1; umount -v /mnt/$d.rtovl /mnt/$d.rtfs /mnt/$d.filesystem /mnt/$d.installer; grep $d /proc/mounts' x sr3 ``` ### Start container ``` # start lxc start --console ubt-sr3 lxc -v --debug start ubt-sr3 --console lxc console ubt-sr3 lxc console ubt-sr3 --show-log # run bash lxc exec ubt-sr3 -- bash -i # check log lxc info ubt --show-log sudo less /var/log/syslog # lxc list all lxc list -h ( set -xv; lxc list -c $( lxc list -h | sed -n -e '/^[ ]\+Pre-defined column /,/^$/{/^[ ] \+\(Pre-defined\|L\) /d; /^[ ]*$/d; s/^[ ]\+\(.\) - .*$/\1/; p}' | sed -e ':a; N; $!ba; s/\n/,/g' ) ) lxc list -c ns46tSDbPdp ``` Run lubuntu-22.04.5 and ubuntu-22.04.5 desktop livecd in lxc container, boot hung and journal log is flooding by acpid.service: Condition check resulted in ACPI event daemon being skipped. ``` # workaround for acpid problem echo "mv /etc/acpi/events /etc/acpi/events.bak" | lxc exec ubt-sr3 -- bash -xv # stop snapd if it keeps in restarting loop, caused by no disk space or no "security.nesting", or "open files(ulimit -n)" for bindfs is not enough. sudo systemctl stop snapd.service snapd.socket snapd.seeded # # https://discuss.linuxcontainers.org/t/snap-inside-privileged-lxd-container/13691/8 ( set -xv; f=$(which udevadm); [ -e "$f.old" ] || mv "$f" "$f.old"; ln -s $(which true) $f; echo $? ) ln -s /bin/true /usr/local/bin/udevadm # gnome-shell process busy when firefox in foreground ( set -xv; export DBUS_SESSION_BUS_ADDRESS=/dev/null; firefox & ) ``` ### Snapd in lxc container Snapd in unprevileged container needs "security.nesting" and 125M disk space. Snapd first run needs large disk space for folder /var/lib/snapd/, afer first run, that folder can be reduced and upperdir disk can be shrinked. ``` # allow user run lxc command sudo adduser user lxd # snapd in unprivileged container need security.nesting lxc config set ubt-sr3 security.nesting true # reduce snap space usage, after snapd first run OK and poweroff container sudo sh -xvc 'rf=$1; d=/var/lib/snapd; u=$( grep -e " $rf " /proc/mounts | grep -o -e "upperdir=[^,]\+" ) && u=${u#upperdir=} && u=${u%/upper}; find $rf$d/snaps/ -maxdepth 1 -not -type d | while read -r i; do n=${i##*/}; j=$rf$d/seed/snaps/$n; [ -e $j ] && rm $i && ln -sr $j $i; j=$u$d/seed/snaps/$n; [ -e $j ] && rm $j; done; mount -o remount $rf; echo $?' x /mnt/sr3.rtfs # check snapd disk usage sudo du -sh /mnt/data/lxd_ubt-sr3_data/upper/var/lib/snapd # use snapd seed files in lowerdir, reduce disk usage sudo sh -xvc 'd=$1; t=$2; s=/var/lib/snapd; find /mnt/ -maxdepth 1 -type d -name "$d.*" -not -name "$d.rtfs" -not -name "$d.rtbind" | while read -r d; do find $d$s/seed/snaps/ -type f | while read -r f; do i=$t/upper${f#$d}; [ -e $i ] && ls -l $f $i && cmp $f $i && rm $i; done; done; rmdir -p $t$s/seed/snaps; mount -o remount /mnt/$d.rtfs' x sr3 /mnt/data/lxd_ubt-sr3_data # remove snapd # # stop snapd sudo systemctl stop snapd snapd.socket snapd.seeded # snapd.apparmor # # remove snapd #sudo du -sh /mnt/data/lxd_ubt_data/upper/var/lib/snapd sudo du -sh /var/lib/snapd # sudo apt purge snapd #sudo apt autoremove sudo apt-mark hold snapd printf "%s\n" "Package: snapd" "Pin: release a=*" "Pin-Priority: -10" | sudo tee /etc/apt/preferences.d/nosnap.pref grep -e /snap/ -e /var/snap/ /proc/mounts | cut -d " " -f 2 | xargs -t sudo umount sudo rm -rf /snap /var/snap /var/lib/snapd /var/cache/snapd /home/*/snap /root/snap # reinitialize snap, revert all ( set -xv; t=lxd_ubt-sr3_data/upper; sudo find $t/ -name 'snap*' -type d -exec rm -fr \{\} \; ; sudo find $t/ -name 'snap*' -exec rm -f \{\} \; ; sudo rm -fr $t/var/cache/snapd ) sudo mount -oremount /mnt/sr3.rtfs/ ``` ### Move apt cache folder to memory ``` # move apt cache foder to memory ( set -xv; printf "%s\n" /var/lib/apt/lists /var/run/lib_apt_lists /var/cache/apt /var/run/cache_apt /var/cache/debconf /var/run/cache_debconf /var/lib/ubuntu-advantage /var/run/lib_ubuntu-advantage /var/cache/apt-xapian-index /var/run/cache_apt-xapian-index /var/lib/swcatalog /var/run/lib_swcatalog /var/cache/swcatalog /var/run/cache_swcatalog /var/cache/app-info /var/run/cache_appinfo | while read -r d && read -r t; do sudo mkdir -p $t ${d%/*} && sudo rm -fr $d && sudo ln -sr $t $d; done; ) # create cache folders in memory at boot ( set -xv; f=/etc/tmpfiles.d/tmp.conf; [ -s $f ] || sudo cp /usr/lib/tmpfiles.d/tmp.conf $f; printf "%s\n" "D /var/run/tmp 1777 root root -" "D /var/run/lib_apt_lists 0755 root root -" "D /var/run/cache_apt 0755 root root -" "D /var/run/cache_debconf 0755 root root -" "D /var/run/lib_ubuntu-advantage 0755 root root -" "D /var/run/cache_apt-xapian-index 0755 root root -" "D /var/run/lib_swcatalog 0755 root root -" "D /var/run/cache_swcatalog 0755 root root -" "D /var/run/cache_app-info 0755 root root -" | sudo tee -a $f; f=/etc/fstab; grep -e "^tmpfs /tmp " $f || echo "tmpfs /tmp tmpfs nosuid,nodev 0 0" | sudo tee -a $f; echo $? ) # # increase space for folder /run/ if apt command reports no space left. sudo mount -o remount,size=512M /run # sudo dpkg --configure -a ``` ### Install sshd and xrdp Sshd is already installed and running in server rootfs. For desktop rootfs, sshd package is available in server livecd and can be instlled from it. To access desktop GUI in container, install and run xrdp in container, then use rdp client to connect. ``` # xrdp is in "universe", if depend package version mismatch, enable "xxxxxx-security" vi /etc/apt/sources.list sudo apt update sudo apt install openssh-server sudo apt install xrdp xserver-xorg-core xserver-xorg-input-all xorgxrdp sudo systemctl set-default multi-user.target # im-config im-config -c cat ~/.xinputrc cat /etc/X11/xinit/xinputrc # # for lubuntu-18.04 lxde and lubuntu-20.04 lxqt, not use ibus # fcitx input chinese only work in Leafpad when fcitx-frontend-gtk3 is not installed. # fcitx may freeze desktop GUI, remove it from Indicator Applets: right click task bar -> Panel Settings (lxpanelctl config) -> Panel Applets -> Indicator applets -> Preferences -> uncheck "Indicator Applications". sudo apt install fcitx fcitx-googlepinyin fcitx-module-dbus fcitx-frontend-gtk2 fcitx-frontend-gtk3 # for lubuntu-18.04 LXDE printf "%s\n" "fcitx -d" "exec lxsession -e LXDE -s Lubuntu" | tee -a ~/.xsession # for lubuntu lxqt sudo apt install ibus ibus-pinyin systemctl stop sddm.service echo "startlxqt" | tee -a ~/.xsession # for ubuntu wayland sudo systemctl stop gdm printf "%s\n" "export GNOME_SHELL_SESSION_MODE=ubuntu" "export XDG_CURRENT_DESKTOP=ubuntu:GNOME" "export XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg" | tee -a ~/.xsessionrc Lubuntu-18.04.5 LXDE autostart configure Start menu -> Preferences -> Default applications for LXSession -> LXSession configuration -> Autostart. Lubuntu-22.04.6 LXQt autostart configure Start menu -> Preferences -> LXQt Settings -> Session Settings -> LXQt Session Settings -> Autostart. ``` Windows RDP client ``` # username in Windows RDP client (mstsc.exe) saved in registry HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client\Default HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client\Servers # list remote servers recent used reg query "HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client\Default" # list UsernameHint for remote servers reg query "HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client\Servers" /s ``` Known problems IME (ibus or fcitx) does not work for Ubuntu-22.04 and Lubuntu-22.04 snap Firefox. Ubuntu-24.04 and Lubuntu-24.04 have same problem. Gnome-shell process keeps in busy when firefox runs in foreground. ### Move tmp folders to memory ``` # cleanup tmp folder sudo rm -fr /mnt/data/lxd.data/@sr*/upper/tmp # move cache folders # tracker3 file index use disk space du -sh ~/.cache/tracker3/ ps -few | grep track systemctl --user list-unit-files | grep track systemctl --user status track* ( set -xv; systemctl --user list-unit-files | grep track | cut -d " " -f 1 | while read -r i; do systemctl --user stop $i; systemctl --user mask $i; done ) tracker3 reset --filesystem --rss rm -fr ~/.cache/tracker3/ # # clean up mesa_shader_cache, after logout GUI ps -few | grep mesa pkill -f "mesa" rm -fr ~/.cache/mesa_shader_cache # rm -fr .cache ln -s /tmp ~/.cache # firefox cache about:cache about:config # move firefox cache to memory ( set -xv; d=$HOME/snap/firefox/common/.cache && [ -d "$d" ] && rm -fr $d && ln -s /tmp $d; d=$HOME/snap/firefox/common/.mozilla/firefox; find $d/ $HOME/.mozilla/firefox/ -maxdepth 1 -type d -name '*.default' -o -name '*.default-release' | while read -r i; do for j in storage security_state; do j=$i/$j; [ -e $j ] || continue; rm -fr $j; ln -v -s /tmp $j; done; done ) # rm -fr $HOME/snap/snapd-desktop-integration/common/.cache ln -s /tmp $HOME/snap/snapd-desktop-integration/common/.cache # rm $HOME/snap/firefox/common/.mozilla/firefox/*.default/*.sqlite* # delete extra emoji files of ibus sudo find /usr/share/ibus/dicts/ -maxdepth 1 -name 'emoji-*.dict' -not -name '*-zh*' -not -name '*-yue*' -not -name '*-en*' -not -name '*-root*' -exec rm -v \{\} \; ``` ## Run Alpine Linux in lxc container Create container rootfs from Alpine Linux cdrom, then run it. ``` # create container lxc init --storage folder emptyrootfs alpine lxc config show alpine # set rootfs dir echo "lxc.rootfs.path = dir:/mnt/sr4.rtfs" | lxc config set alpine raw.lxc - lxc config set alpine security.nesting true # add cdrom and usb lxc config device add alpine sr4 disk source=/mnt/sr4 path=/media/cdrom sudo mount /dev/sdm /mnt/usb1 lxc config device add alpine usb disk source=/mnt/usb1 path=/media/usb # prepare rootfs sudo sh -xvc 'n=$1; d=$2; t=/mnt/$d; getmnt() { grep -e " $1 " /proc/mounts; }; getmnt $t || { mkdir -p $t && mount /dev/$d $t || exit $?; }; t=/mnt/$d.rtfs; getmnt $t && exit 1; i=zram5; echo 1321M | tee /sys/block/$i/disksize && mkfs.ext2 /dev/$i && mkdir -p $t && mount -v -t ext2 /dev/$i $t || exit $?; cd $t && zcat /mnt/sr4/boot/initramfs-* | cpio -id || exit $?; rm -fr newroot .modloop lib/firmware lib/modules lib/modules-load.d etc/modprobe.d; [ -L "sbin/init" ] && exit 1; cp -a init sbin/ && sed -i -e "2i set -xv\nKOPT_apkovl=/media/usb/alpine.apkovl.tar.gz\n" -e "s@^set .*cat /proc/cmdline@true &@" -e "s/^eend \\\$?/true &/" -e "s@^exec .* switch_root @mv /sbin/init /sbin/init.rm\nln -s /bin/busybox /sbin/init\nexec /sbin/init\n\n&@" -e "s/^[ \tu]*mount /true &/" -e "s@^sysroot=/sysroot@sysroot=/@" sbin/init || exit $?; diff -u init sbin/init; t=etc/local.d; mkdir -p $t; f=$t/runonce.start; printf "%s\n" "#!/bin/sh" "grep -q -e \"^console::respawn:\" /etc/inittab && exit" "{" "sleep 2" "sed -i -e \"s@tty1::respawn:/sbin/getty @console::respawn:/sbin/getty 38400 console\n#&@\" -e \"s@^tty[S0-9]*::respawn:/sbin/getty @#&@\" /etc/inittab" "kill -HUP 1" "f=/etc/network/interfaces; grep -q -e eth0 \$f || ( echo \"auto eth0\"; echo \"iface eth0 inet dhcp\" ) >> \$f" "rc-update add networking boot" "apk add openssh-client openssh-server" "rc-update add sshd default" "rc-service sshd start" "} 2>&1 | logger -t ${f##*/} &" >> $f && chmod a+rx $f || exit $?; t=etc/runlevels/default; mkdir -p $t && [ -L "$t/local" ] || { ln -s /etc/init.d/local $t/ || exit $?; }; uo=""; i="$( lxc config get $n volatile.idmap.next | grep -o "{\"Isuid\":true,[^}]\+}" | tr -d "{}\"" | tr -s "," "\n" | tr -s ":" "=" | grep -e "^Hostid=" )" && i="uo=${i#*=}" && eval "$i" && [ -n "$uo" ] && chown -R $uo:$uo .; echo $?' x alpine sr4 # run lxc start --console alpine ``` ### Seperate apkcache files to repository dirs ``` # https://wiki.alpinelinux.org/wiki/Apk_spec ( set -xv; sd=/media/usb/cache; i=/media/usb/apks; mkdir -p "$i" && cd "$i" && mkdir -p main/x86_64 community/x86_64 && ( cd $sd && ls -1 ) > fl && grep -e 'APKINDEX.*.tar.gz' fl | while read -r i; do tar -xf "$sd/$i" && grep -e "^P:alpine-base$" APKINDEX && td=main || td=community; td=$td/x86_64; echo "$i" > "$td/APKINDEX.tar.gz.txt"; mv -v "$sd/$i" "$td/APKINDEX.tar.gz"; sed -e '/^[PV]:/!d; s/^V/-/' APKINDEX | sed -e '/^P:/{N;s/[P:\n]//g}' | while read -r i; do j=$(grep -e "^$i\.\?[^.]*\.apk" fl) && { mv -v "$sd/$j" "$td/$i.apk" || exit $?; } || true; done || exit $?; done || exit $?; rm -f .SIGN.RSA.* DESCRIPTION APKINDEX fl; echo $? ) 2>&1 | tee m.log ``` Then the repository folder can be used as apk source. ``` printf "%s\n" "" "" /media/usb/apks/main /media/usb/apks/community >> /etc/apk/repositories ``` ### Install xfce and xrdp ``` # https://wiki.alpinelinux.org/wiki/Xfce # apk add openssh-server apk add util-linux e2fsprogs dosfstools adduser user # # Alpine Linux v3.11 does not have "setup-desktop", need to manual install. # for Alpine Linux v3.17 and newer #setup-desktop xfce apk add eudev dbus rc-service dbus start setup-xorg-base apk add xfce4 xfce4-terminal # # chinese font wget https://mirrors.edge.kernel.org/alpine/v3.19/community/x86_64/font-wqy-zenhei-0.9.45-r3.apk apk add font-wqy-zenhei-0.9.45-r3.apk apk add /media/usb/cache/font-wqy-zenhei-0.9.45-r3.b3399bf3.apk # # Apline Linux v3.11 xrdp xorg session not work because it has no xorgxrdp package, use xvnc session. apk add xrdp xorgxrdp xorg-server ( set -xv; for i in dbus xrdp xrdp-sesman sshd; do rc-service $i start; done; echo $? ) rc-service lightdm status # # Alpine Linux v3.11 has no firefox, use chromium. apk add firefox apk add libreoffice # Alpine Linux v3.17 and older, chinese input not work. ``` ## apt-cacher-ng Use apt-cacher-ng as apt proxy server to speed up apt work. ``` # install apt-cacher-ng sudo apt-get install apt-cacher-ng # check files installed dpkg -L apt-cacher-ng # html page for web browser vi /usr/lib/apt-cacher-ng/userinfo.html # check config grep -rni 'CacheDir\|acng-doc' /etc/apt-cacher-ng/ grep -rni remap /etc/apt-cacher-ng/ grep -rni 'Port\|Offlinemode' /etc/apt-cacher-ng/ sudo cat /etc/apt-cacher-ng/security.conf ls -l /var/log/apt-cacher-ng/ tail -f /var/log/apt-cacher-ng/apt-cacher.log /var/log/apt-cacher-ng/apt-cacher.err # ls -l /var/lib/apt-cacher-ng /var/cache/apt-cacher-ng ls -l /usr/share/doc/apt-cacher-ng/ ``` ### Move data folders ``` pgrep -a apt sudo systemctl stop apt-cacher-ng # sudo sh -xvc 'rd=/mnt/data; for d in /var/lib/apt-cacher-ng /var/cache/apt-cacher-ng; do [ -e $d.old ] || { i=$(echo $d | tr -s "/" "_"); cp -a $d $rd/$i && mv $d $d.old && ln -s $rd/$i $d; }; done' ``` ### Offline mode When apt-cacher-ng cannot reach outside website, client failed to run "apt update". To solve the problem, set "Offlinemode: 1" and restart apt-cacher-ng. When "Offlinemode: 1", cached "Release" file is not updated when client run "apt update". When "Offlinemode: 0" (default), cached "Release" file is always updated when client run "apt update". Toggle "Offlinemode" on network interface up and down. Use file "99_enp0s8.sh" to auto toggle "Offlinemode", run the short sript at its head part to install it. ### Cache repository Releae and Packages ``` # cache repository Releae InRelease and Packages.xz # http://192.168.58.101:3142/acng-doc/html/config-serv.html#repmap # https://hub.docker.com/r/gijoe88/docker-apt-cacher-ng # # check if Release file and Packages.xz are cached: http://192.168.58.101:3142/acng-report.html Click "Start Scan and/or Expiration" in "Scan and semi-automatic expiration". # grep -rn uburep /etc/apt-cacher-ng/ ls -l /etc/apt-cacher-ng/backends_ubuntu* /usr/lib/apt-cacher-ng/ less /usr/lib/apt-cacher-ng/ubuntu_mirrors find /var/cache/apt-cacher-ng/ -iname 'packages*' -o -iname 'release*' -o -iname 'inrelease*' -exec ls -l \{\} \; | grep /uburep/ find /var/cache/apt-cacher-ng/ -iname 'packages*' -o -iname 'release*' -o -iname 'inrelease*' -exec ls -l \{\} \; | grep -v /uburep/ # add lines in file "/etc/apt-cacher-ng/acng.conf" to cache Release file for these repositories grep 'Remap-secdeb' /etc/apt-cacher-ng/acng.conf Remap-ubuportsrep: ports.ubuntu.com /ubuntu-ports ; ports.ubuntu.com/ubuntu-ports Remap-secubu: security.ubuntu.com /ubuntu ; security.ubuntu.com/ubuntu Remap-uburcneerep: repos.rcn-ee.com/ubuntu ; repos.rcn-ee.com/ubuntu Remap-debrcenrep: repos.rcn-ee.com/debian ; repos.rcn-ee.com/debian # restart apt-chacher-ng and monitor log tail -f /var/log/apt-cacher-ng/apt-cacher.log ``` ## Windows NTP server Except Linux NTP server, Win7 and Win10 have NTP server, it can be used to sync time. ``` # web resources https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-time-service-tools-and-settings https://learn.microsoft.com/en-us/troubleshoot/windows-server/active-directory/configure-authoritative-time-server https://support.hanwhavision.com/hc/en-us/articles/26570683589529-How-to-Setup-an-NTP-Server-on-Windows-10 https://support.nortekgroup.com/hc/en-us/articles/9657669911452-Use-Windows-10-as-NTP-Server # cmd.exe commands w32tm /? w32tm /query /configuration /verbose w32tm /query /status /verbose w32tm /query /peers /verbose w32tm /dumpreg /subkey:TimeProviders\NtpServer # regedit Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpServer # change: Enabled=0 Enabled=1 # Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config # change: AnnounceFlags=0x0a AnnounceFlags=0x05 # check Windows Time service status sc query | find /i "time" sc query W32Time # apply config changes w32tm /config /update sc stop W32Time sc start W32Time # allow firewall income netsh advfirewall firewall add rule name="NTP Server" dir=in protocol=udp localport=123 profile=any enable=yes action=allow # test w32tm /stripchart /computer:192.168.58.1 /samples:4 /dataonly ``` Client side ``` # sntp report: "not in sync, skipping this server" # Set Windows 10 NTP server "AnnounceFlags=0x05" to avoid this problem. # test ntp server sntp -d -d -d 192.168.58.1 # systemd timesyncd systemctl cat systemd-timesyncd.service man timesyncd.conf man systemd.time less /etc/systemd/timesyncd.conf timedatectl status timedatectl show-timesync timedatectl timesync-status # # configure systemd-timesyncd to user Windows 10 NTP server # Change lines in file /etc/systemd/timesyncd.conf # Then restart systemd-timesyncd.service # sudo systemctl restart systemd-timesyncd.service # ( set -xv; ( echo "FallbackNTP=192.168.58.1"; echo "RootDistanceMaxSec=5356800" ) | sudo tee -a /etc/systemd/timesyncd.conf ) ``` Windows clinet sync to NTP server ``` # https://learn.microsoft.com/en-us/troubleshoot/windows-server/active-directory/w32time-sync-specialpollinterval-ignored # # In Windows client, set ntp server and change the Automatic Sync Interval to 5 minutes # w32tm /config /manualpeerlist:"192.168.58.1,0x9" /syncfromflags:manual /update # # set poll interval reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpClient /v SpecialPollInterval /t REG_DWORD /d 300 /f # # set max time offset reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config /v MaxPosPhaseCorrection /t REG_DWORD /d 0xFFFFFFFF /f # # config w32time service sc config w32time start= delayed-auto sc start w32time # # force sync time w32tm /config /update w32tm /resync /force ```