SOLUTION BELOW

The actual bug

I have never been in a more confusing situation regarding Linux.

I have a Dell XPS 15 9560, which had a dual boot Windows 10 / EndeavourOS setup. It was running fine for months. 10 days ago I updated Linux and after restart it couldn't boot anymore. It got stuck at "A start job is running for /dev/disk/by-uuid/..." (which is the root partition).

First, with the help of a friend of mine who is quite knowledgeable about Linux (he runs vanilla Arch, etc), we spent 5 hours trying to fix it but had no luck.

Then I decided to back up everything and do a fresh install. Aaaand the same error happened again on the first boot. Then I though "ok, probably some problem with Arch, lets try Fedora". Nope. Some similar error about not finding the root partition. (Here I must say that the kernel which was shipped with the ISO was working fine, but after updating to the latest one, it failed.) Here I thought "ok, then it might be a problem with the latest kernel, let's install EndeavourOS with the LTS kernel." Nope, LTS kernel also didn't boot. Then I tried Ubuntu and it worked, but that's not solving the problem. Then I decided to put another nvme drive in the laptop and try there. The same error again.

Now the greatest part: If I put the nvme drive into an external usb case, EndeavourOS installs, updates, boots without any problem, no sign of the error.

So now I don't know how to proceed... Maybe there is something wrong with the pcie port in my laptop, but except for the booting problem, windows is working, I can also mount and access every partition in the ssd through a live usb. So no other signs of problem with the port whatsoever.

I would be grateful for any advice as I've lost several days trying to solve this and I am out of ideas...

Solution: The last working kernels are from 11. August 2023 (both linux and linux-lts) linux-6.4.10.arch1-1 and linux-lts-6.1.45-1. You can download them from here: linux / linux-lts and install them with

sudo pacman -U the_path_to_the_package

Thank you all for the help!

  • eldain@feddit.nl
    ·
    1 year ago

    According to your logs when installing the kernel, your nvme modules are nowhere to be found, and basic system tools are unavailable. Your core system seems severely borked... can you do a memtest (broken ram can corrupt your storage and your attemps at finding the problem) and check your smartctl --all of that ssd?

  • rotopenguin@infosec.pub
    ·
    1 year ago

    AHAHAHA that is a proper insane bug. One PCIe device shouldn't be able to slap others off of the "bus", "we're not on a bus all you did was mess up your own personal lanes mate".

  • abrer@lemmy.one
    ·
    1 year ago

    So this occurs after an update. Is it not possible to boot into the prior kernel?

    If possible to boot into the prior kernel, can you inspect logs or the journal to see where your error is cropping up?

    This issue sounds like a regression of sorts with a driver, but log/debug would help confirm. This would be one worth reporting to upstream if you can rescue some logs (I gather you can if you can boot the disk from another enclosure).

    If you can boot into the machine, investigate note from the journal:

    • journalctl --list-boots
    • journalctl -b -1,
      • where -1 was the prior boot, -2, the one before that, etc

    -- If you are booting into a live environment or are otherwise mounting the disk:

    • journalctl -D /var/log/journal/ID_GOES_HERE
    • example path: /var/log/journal/2dff8304d5114c44bfb1311357a3cd87

    -- Keep us posted.

    If truly a driver regression, but you can boot from the prior kernel (if you don't have it, install it via livecd or so), definitely report this one and remain on the prior kernel until resolved. Bleeding edge things.

    • oiram15@lemmy.sdf.org
      ·
      1 year ago

      I have already wiped everything, so no logs... The only way to have it booting is to install EndeavourOS using the offline installer, which is using kernel 6.4.8. There is an option to install the LTS kernel alongside. So the system is booting with 6.4.8, but after updating, neither the new 6.4.12, or the LTS, which is 6.2, doesn't boot. I haven't tried booting with the LTS kernel before updating, to see if the same kernel is working before and after or not. I will try to reinstall it using the offline installer and then try to gather some logs after updating.

  • Guidonsia@lemmy.ml
    ·
    1 year ago

    I had se similar issue but with bluetooth, worked on windows not on linux. Fixed it by resetting the BIOSwith the CMOS battery thing