> What newer better features is it meant to provide, or what problems is it meant to solve or address in relation to X11 and why should it take so long?
IMHO, Wayland is meant to address a few issues that can't really be addressed in X11.
Integrating the 'display server' with the window manager eliminates some unfixable syncrhonization issues in the X11 design, mostly around creation of new windows and how they're placed (or something like that, I don't quite remember), but the downside is that window managers need to integrate a compositor and have more behavior for that.
The other big thing is that X11 is a distributed system with no security between parts. Any connected X client can read or generate keyboard events at any time, or read any part of the screen. This is not ideal for lots of reasons, but the Wayland way requires different approaches to make things like screenshots, screen capture, and global hotkeys work. Also, because each window manager/compositor has its own display server (although many use the wlroots library for the base), consensus building is needed or a program that wants a hotkey needs to know how to do it in X and in all the popular Wayland compositors. Consensus building is hard.
From what I understand, Wayland required new driver architecture, and drivers for Nvidia were slow to arrive.
X11 was built around networking and has always allowed for a display server located near your monitor and clients distributed elsewhere in a network. This used to work well, but requires proper client architecture to handle the asynchronous nature of networks for best results. There's a fraction of users who make use of this (including me) in their regular use case for X. Waypipe was slow to arrive, and isn't as convenient as ssh -X, but should help provide for this use case.
There's probably some other stuff. But basically, Wayland is not a drop in replacement for X11 and X11 works for a lot of people, so they don't see a big reason to switch and disrupt their way of life.
It continues because most of the people who were working on Xorg prefer to work on Wayland. There's a lot of people relying on the work of a few people who do the deep work on display servers, so you know... either you use what they're making, or you use the older things they made, or you go to CLIs. :)
https://en.wikipedia.org/wiki/Windows_Display_Driver_Model
except Microsoft had that up and running in 2006 but politics has kept Linux from doing the same. Specifically, WDDM and Wayland simplify the graphics interface to an API that lets you blast out a rectangle of pixels so unlike the old HAL model you weren't specifying an API where it can draw primitives like lines and circles, the API allows user-space applications to run software on the GPU to make pixel rectangles that either get blasted to the screen or composited.
Microsoft is really good at maintaining backwards compatible with the UI (I can still run Office '97 on my PC) and they figured out how to make it so old applications don't know what happened.
Incompetence at GUI work is a hallmark of the politics of Linux, just as power management "just not working" is a hallmark of the politics of the Intel-Microsoft-Dell alliance.