We should make sure first, that the Haskell updates from upstream are really applied to
our build schedule in exactly that order (as I fear order of rebuilds matters a lot).

As an easy fix in the past I was successful by selectively building parts of the failed Haskell module, but I easily ended up in dependency circles (where this library hash doesn't come in
handy at all!)

As I last resort I can think of a script checking the build error logs and trying to
guess package names which needs to be rebuilt:

Configuring attoparsec-0.13.2.4...
Error:
    The following packages are broken because other packages they depend on are missing. These broken packages must be rebuilt before they can be used.
installed package tasty-1.3.1 is broken due to missing package ansi-terminal-0.11-2SXi8ZhU18i2uWLidRUotS, async-2.2.2-K8T9LglWxlG5HgD5vvGjbo, optparse-applicative-0.16.1.0-JDPEASK1GJS1Nsq2qjjZCq
installed package tasty-quickcheck-0.10.1.2 is broken due to missing package QuickCheck-2.14.2-io0WylueSG4seqSTUQCKM, optparse-applicative-0.16.1.0-JDPEASK1GJS1Nsq2qjjZCq, random-1.2.0-1bl1BCDTJTv7mTz0mMuFeh

So in this example this means haskell-tasty needs a rebuild against new versions of
haskell-async, haskell-optparse-applicative and tasty-quickcheck one against haskell
quickcheck, haskell-optparse-applicative and haskell-random.

I also fear we cannot simply drop Haskell easilily for things like 'pandoc' or
'shellcheck'.

"As I last resort I can think of a script checking the build error logs and trying to
guess package names which needs to be rebuilt:"

We do that already: the buildmaster does reschedule the respective packages upon report of error. However, as you noted: Dependency cycles make this approach fail, too, because then, everything depending on haskell-tasty or haskell-tasty-quickchek which was already built needs to be rebuilt, too …

check the build scripts starting from:

https://git.archlinux32.org/builder/tree/bin/return-assignment#n291

maybe the regex does not work anymore or something like that …

nope, the regex looks good.

I had a (brief) look into the haskell mess and the problem seems to be, that a few haskell-* packages genuinely fail (e.g. haskell-tasty) on 32 bit.

The buildmaster ignores failed packages after some time (a few days, I think). The reasoning behind that is, that for "normal" packages (like libraries), this will not block all dependent packages (imagine libfoo and package bar, depending on libfoo. Both get an update, but libfoo fails to build. At some point, we want to have the new bar even if it's compiled against the old libfoo - this is especially true, if "libfoo" is actually just some boring makedependency, like git, or some doctohtml or whatnotelse). Unfortunately, with haskell-* this won't work, because the dependency *must* be rebuilt to update the hash (e.g. makedepends pandoc will not work in the old version).

These haskell-* packages have build errors which are not "due to haskell itself":

haskell-generic-data
haskell-libbf
haskell-profunctors
haskell-quickcheck-classes-base
haskell-rank2classes
haskell-semigroupoids
haskell-tasty
haskell-th-lift-instances
haskell-tidal

Now ghc itself fails:

"inplace/bin/mkdirhier" libraries/ghc-prim/dist-install/doc/html/ghc-prim//.
"inplace/bin/mkdirhier" libraries/ghc-prim/dist-install/build/autogen/GHC//.
"inplace/bin/genprimopcode" --make-haskell-source < compiler/stage1/build/primops.txt > libraries/ghc-prim/dist-install/build/autogen/GHC/Prim.hs
"inplace/bin/ghc-cabal" hscolour libraries/ghc-prim dist-install
ghc-cabal: The program 'hscolour' version >=1.8 is required but the version of
/usr/bin/HsColour could not be determined.
make[1]: *** [libraries/ghc-prim/ghc.mk:4: libraries/ghc-prim/dist-install/doc/html/ghc-prim/ghc-prim.haddock] Error 1
make: *** [Makefile:128: all] Error 2
==> ERROR: A failure occurred in build().
    Aborting...

and again on i686:

1. > Compressing package…

zstd: error 11 : Allocation error : not enough memory

Considering the ridiculous speed, at which upstream increases haskell pkgrels, I believe, they have similar issues as we have. But for them, they are not as serious as for us, because they have a less strict dependency-checking in place than we do.

I added two TODOs:
https://archlinux32.org/buildmaster/todos.php#TODO232 https://archlinux32.org/buildmaster/todos.php#TODO233 but didn't have time yet to actually implement this. If implemented, they should considerably speed up the haskell rebuilds.

In latest builds I see also 'ghc' itself choaking on 'llvm' for unknown reasons:

ghc: panic! (the 'impossible' happened)
  (GHC version 8.10.5:
        Failed to lookup LLVM data layout
  Target: i686-unknown-linux
  Available targets:
      i386-unknown-windows
      i686-unknown-windows
      x86_64-unknown-windows
      arm-unknown-linux-gnueabihf
      arm-unknown-linux-musleabihf
      armv6-unknown-linux-gnueabihf
      armv6-unknown-linux-musleabihf
      armv6l-unknown-linux-gnueabihf
      armv6l-unknown-linux-musleabihf
      armv7-unknown-linux-gnueabihf
      armv7-unknown-linux-musleabihf
      armv7a-unknown-linux-gnueabi
      armv7a-unknown-linux-musleabi
      armv7a-unknown-linux-gnueabihf
      armv7a-unknown-linux-musleabihf
      armv7l-unknown-linux-gnueabi
      armv7l-unknown-linux-musleabi
      armv7l-unknown-linux-gnueabihf
      armv7l-unknown-linux-musleabihf
      aarch64-unknown-linux-gnu
      aarch64-unknown-linux-musl
      aarch64-unknown-linux
      i386-unknown-linux-gnu
      i386-unknown-linux-musl
      i386-unknown-linux
      x86_64-unknown-linux-gnu
      x86_64-unknown-linux-musl
      x86_64-unknown-linux
      x86_64-unknown-linux-android
      armv7-unknown-linux-androideabi
      aarch64-unknown-linux-android
      armv7a-unknown-linux-androideabi
      powerpc64le-unknown-linux-gnu
      powerpc64le-unknown-linux-musl
      powerpc64le-unknown-linux
      s390x-ibm-linux
      i386-apple-darwin
      x86_64-apple-darwin
      arm64-apple-darwin
      aarch64-apple-darwin
      armv7-apple-ios
      aarch64-apple-ios
      i386-apple-ios
      x86_64-apple-ios
      amd64-portbld-freebsd
      x86_64-unknown-freebsd
      aarch64-unknown-freebsd
      armv6-unknown-freebsd-gnueabihf
      armv7-unknown-freebsd-gnueabihf
      arm-unknown-nto-qnx-eabi
  Call stack:
      CallStack (from HasCallStack):
        callStackDoc, called at compiler/utils/Outputable.hs:1179:37 in ghc:Outputable
        pprPanic, called at compiler/llvmGen/LlvmCodeGen.hs:113:20 in ghc:LlvmCodeGen

Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug

ghc/ghc.mk:124: ghc/stage1/package-data.mk: No such file or directory
make[1]: *** [utils/ghc-cabal/ghc.mk:59: utils/ghc-cabal/dist/build/tmp/ghc-cabal] Error 1

But for some minor issues with -latomic I'm on the way on i486 (this will also be a binary seed to bootstrap
again on i686 and pentium4). I bootstrapped a Haskell 8.4. with the binary seed with a patched ghc8.4-bin.
Then I build sucessfully a ghc 8.8.3, happy, alex and haskell-hscolour, so I am reasonably convinced, that
the Haskell compiler actually works. Now ghc 9.0.1 makes it harder to pass a -latomic to configure and
make (LIBS doesn't work in all compiler stages and some configure options –with-libs have been removed).

The LLVM targets above are easy fixable in llvm-targets by copying and modifying a close enough
architecture. The llvm-targets file doesn't contain some archane system triplets, or the triplet sniffer
has other ideas (hence the mismatches like i386-unknown-linux - which exists -and i486-unknown-linux -
which is probed during the build).

Stuck in segfaults in stage 2 compiler now:

Core was generated by `/root/haskell/ghc/src/ghc-9.0.1/inplace/lib/bin/ghc-stage2 -B/root/haskell/ghc/'.

#0  0xaf9209b0 in buflen () from /usr/lib/libc.so.6
#1  0xb0a918f4 in base_GHCziEventziThread_ensureIOManagerIsRunning1_info$def ()
   from /root/haskell/ghc/src/ghc-9.0.1/libraries/base/dist-install/build/libHSbase-4.15.0.0-ghc9.0.1.so

Just found an old Haskell on ARMv7, so it actually has been bootstrapped in the past (I was wrong).

Packages (4) ghc-8.6.1-1 ghc-libs-8.6.1-1 llvm50-5.0.2-2 llvm50-libs-5.0.2-2

Now we are stuck in

/usr/lib/ghc-8.10.5/bin/runghc: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory

Haskell has really low priority, patches are welcome..

Bootstrapping with ghc90-bin leads to:

make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Haddock/Syb.dyn_o] Segmentation fault (core dumped)
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [utils/check-ppr/ghc.mk:18: utils/check-ppr/dist-install/build/Main.dyn_o] Segmentation fault (core dumped)
make[1]: *** [utils/check-api-annotations/ghc.mk:18: utils/check-api-annotations/dist-install/build/Main.dyn_o] Segmentation fault (core dumped)
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/CompatPrelude.dyn_o] Segmentation fault (core dumped)
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Documentation/Haddock/Types.dyn_o] Segmentation fault (core dumped)
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Haddock/Utils/Json/Types.dyn_o] Segmentation fault (core dumped)
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Haddock/Backends/Xhtml/Types.dyn_o] Segmentation fault (core dumped)
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Paths_haddock.dyn_o] Segmentation fault (core dumped)
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Haddock/Backends/HaddockDB.dyn_o] Segmentation fault (core dumped)
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Haddock/Backends/Hyperlinker/Types.dyn_o] Segmentation fault (core dumped)

This turns out to be a major headache..