|
|
Rechecked today, still the same error.
I do not understand, why the packages are marked as not existent. Looking into the repo, the packages are existent and downloadable. A native installed kali linux is working perfectly.
For me it looks like the packages are not named in the packages.gz (this is just a guess), but I can not reproduce, where this is downloaded from. Is this maybe a problem in the git repository?
If there is anything I can do, please feel free to ask.
This bug is critical, maybe you might want to change the status of this bugreport.
Best
Hans |
|
|
|
I am building a Kali live ISO just now and it works for me. Also for your info, our QA builds images from kali-rolling daily, and if ever it fails, we're notified and we fix it asap. In practice it's very rare that a build fails.
So here, the issue is very likely on your side. I checked your build logs, compared it with mine, and unfortunately I didn't see any hint of what goes wrong... I'd say, maybe some custom configuration that you added is causing this. Is it public, can you share the git repo for your kali-live fork with your config? |
|
|
|
Hi Arnaud,
thanks for your response. First to say, my scripts worked well before, no issues. However, it is a little bit complicated, so I explain.
First I am booting a lkali-ive system. Then I am mounting the harddrrive with my kali-repo to /mnt in the live system.
Below /mnt I am starting the main script, with this content:
---- snip ----
#!/bin/bash
# mount -L livebuild /mnt
# mount --uuid 806a0d05-06c6-4aa3-af40-318cea130f59 /mnt
rm -v /mnt/ISO-latest/*.iso
# apt update && apt upgrade -yq && apt autoremove -yq && apt clean -yq
apt install git cdebootstrap live-build live-config apt-cacher-ng console-data curl -y
############################################################################################
# i386
# pushd /mnt/kali-linux/i386/kali-live
# First line: parallel, second line: seriell
# ./uit-update-kali.sh && ./uit-build-i386.sh & popd
# ./uit-update-kali.sh && ./uit-build-i386.sh && lb clean --purge && popd
############################################################################################
# amd64
pushd /mnt/kali-linux/amd64/kali-live
./uit-update-kali.sh && ./uit-build-amd64.sh && popd
############################################################################################
# pushd /mnt/kali-linux/armhf/kali-arm
# ./uit-build-armhf.sh & popd
# mv -f /mnt/kali-linux/i386/kali-live/images/*.iso /mnt/ISO-latest/
---- snap -----
As you see, I am push into the kali/live-build/ directory and starting another script, then pull and go back into the former folder.
Thus I can start the build script in /mnt/kali-linux/amd64/kali-live/ as I am starting it out of this folder. The scripts started:
First the update script:
--- snip ----
#!/bin/sh
### define path to the repository on your disk
### PATH=
# MOUNTPOINT=/mnt
# DEVICE=/dev/sdb1
# REPOPATH=kali-linux/var/cache/apt-cacher-ng/
### download and install needed packages
# apt-get update
# apt-get install git cdebootstrap live-build live-config apt-cacher-ng console-data curl -y
### execute any commands to fit your personal environment
loadkeys de
### prepare youre proxy and make sure, your local repo has ownership "apt-cacher-ng:apt-cacher-ng"
# cp acng.conf /etc/apt-cacher-ng/
# chown -R apt-cacher-ng:apt-cacher-ng $MOUNTPOINT/$REPOPATH
### updating the git branch
git pull origin main
### This is a test szenario, execite other scripts as needed
### This suggestion calls my very own script.
# ./uit-build-i386.sh
--- snap ----
And then the buiild command at last:
--- snip ---
#!/bin/sh
### Author Hans-J. Ullrich <[email protected]>
### License: GPL
### Set some variables, not active yet.
### architecture: i386, amd64, arm
ARCH=amd64
### Distribution, you might not want to change it.
DISTRO=kali-rolling
### Your preferred window-manager: lxde, kde, xfce, gnome
VARIANT=xfce
### Environmet for your local proxy
# PROXY=http://localhost:3142/
### language, keyboard, locales everything you might not need to change
BOOTAPP="boot=live hostname=uit-kali ignore_uuid locales=de_DE.UTF-8 keyboard-layouts=de keyboard-variants=nodeadkeys"
### Clean your system, not really necessarey, as the latter "./build.sh" does
### this either.
# lb clean
### VERY IMPORTANT: Make sure, apt-cacher-ng is started, even, if you are sure,
### it is already running!
# /etc/init.d/apt-cacher-ng stop && sleep 3 && /etc/init.d/apt-cacher-ng start
### set your needed environment and preparations.
# export http_proxy=http://localhost:3142/
lb config
### This command starts the actual build. All commands are in one line.
### However, this script might be improved, to use variables for better
"~/kali/uit-build-amd64.sh" 50L, 2110B
--- snap ---
After both scripts are finished, the main script from above will be executed furter on.
Please note: In my scripts are many lines commented out, this is not a mistake, but the result of tests I did in the past. Yes, I know, I could clean them up, but I wanted the entries have szayed, that I do not forget them.
Does this help? I know, my scripts are weired, maybe it could be improved, but until a few days ago they worked well.
Thank you for your help.
Best regards
Hans |
|
|
|
Scripts added....
uit-build-amd64.sh (2,110 bytes)
#!/bin/sh
### Author Hans-J. Ullrich <[email protected]>
### License: GPL
### Set some variables, not active yet.
### architecture: i386, amd64, arm
ARCH=amd64
### Distribution, you might not want to change it.
DISTRO=kali-rolling
### Your preferred window-manager: lxde, kde, xfce, gnome
VARIANT=xfce
### Environmet for your local proxy
# PROXY=http://localhost:3142/
### language, keyboard, locales everything you might not need to change
BOOTAPP="boot=live hostname=uit-kali ignore_uuid locales=de_DE.UTF-8 keyboard-layouts=de keyboard-variants=nodeadkeys"
### Clean your system, not really necessarey, as the latter "./build.sh" does
### this either.
# lb clean
### VERY IMPORTANT: Make sure, apt-cacher-ng is started, even, if you are sure,
### it is already running!
# /etc/init.d/apt-cacher-ng stop && sleep 3 && /etc/init.d/apt-cacher-ng start
### set your needed environment and preparations.
# export http_proxy=http://localhost:3142/
lb config
### This command starts the actual build. All commands are in one line.
### However, this script might be improved, to use variables for better
### configuration possibilities.
# ./build.sh --arch $ARCH --distribution $DISTRO --variant $VARIANT --verbose -- --apt-http-proxy $PROXY --bootappend-live $BOOTAPP
./build.sh --arch $ARCH --distribution $DISTRO --variant $VARIANT --verbose -- --bootappend-live "$BOOTAPP"
### these options can be used for further commands after build
### for example, if you want to burn a dvd, shutdown, whatever
### Examples for my personal needs below
#/etc/init.d/apt-cacher-ng stop
### Suggestions for handling the built image.
# cp /home/username/live-build-config/images/*.iso /home/user/
# wodim /path_to_your_live_build/live-build-config/images/your_image_name.iso
### Some opportunities at unattended build run, what to do after the build succeeded.
# dd if=/mnt/kali-linux/amd64/live-build-config/images/kali-linux-rolling-live-lxde-amd64.iso of=/dev/sdf bs=8192 status=progress
# mv -f /mnt/kali-linux/amd64/live-build-config/images/*.iso /mnt/ISO-latest/
# halt -p
# reboot
# shutdown -h now
exit
uit-build-kali-64_only.sh (1,130 bytes)
#!/bin/bash
# mount -L livebuild /mnt
# mount --uuid 806a0d05-06c6-4aa3-af40-318cea130f59 /mnt
rm -v /mnt/ISO-latest/*.iso
# apt update && apt upgrade -yq && apt autoremove -yq && apt clean -yq
apt install git cdebootstrap live-build live-config apt-cacher-ng console-data curl -y
############################################################################################
# i386
# pushd /mnt/kali-linux/i386/kali-live
# First line: parallel, second line: seriell
# ./uit-update-kali.sh && ./uit-build-i386.sh & popd
# ./uit-update-kali.sh && ./uit-build-i386.sh && lb clean --purge && popd
############################################################################################
# amd64
pushd /mnt/kali-linux/amd64/kali-live
./uit-update-kali.sh && ./uit-build-amd64.sh && popd
############################################################################################
# pushd /mnt/kali-linux/armhf/kali-arm
# ./uit-build-armhf.sh & popd
# mv -f /mnt/kali-linux/i386/kali-live/images/*.iso /mnt/ISO-latest/
mv -f /mnt/kali-linux/amd64/kali-live/images/*.iso /mnt/ISO-latest/
sleep 10
loadkeys de
shutdown +10
uit-update-kali.sh (755 bytes)
#!/bin/sh
### define path to the repository on your disk
### PATH=
# MOUNTPOINT=/mnt
# DEVICE=/dev/sdb1
# REPOPATH=kali-linux/var/cache/apt-cacher-ng/
### download and install needed packages
# apt-get update
# apt-get install git cdebootstrap live-build live-config apt-cacher-ng console-data curl -y
### execute any commands to fit your personal environment
loadkeys de
### prepare youre proxy and make sure, your local repo has ownership "apt-cacher-ng:apt-cacher-ng"
# cp acng.conf /etc/apt-cacher-ng/
# chown -R apt-cacher-ng:apt-cacher-ng $MOUNTPOINT/$REPOPATH
### updating the git branch
git pull origin main
### This is a test szenario, execite other scripts as needed
### This suggestion calls my very own script.
# ./uit-build-i386.sh
|
|
|
|
I added the scripts, because the bugsystem ignored the comments in my description (should have masked them, I suppose). The uploaded scripts are unchanged! |
|
|
|
Umpf, last build-script was uncomplete. So here again:
--- snip -----
### License: GPL
### Set some variables, not active yet.
### architecture: i386, amd64, arm
ARCH=amd64
### Distribution, you might not want to change it.
DISTRO=kali-rolling
### Your preferred window-manager: lxde, kde, xfce, gnome
VARIANT=xfce
### Environmet for your local proxy
# PROXY=http://localhost:3142/
### language, keyboard, locales everything you might not need to change
BOOTAPP="boot=live hostname=uit-kali ignore_uuid locales=de_DE.UTF-8 keyboard-layouts=de keyboard-variants=nodeadkeys"
### Clean your system, not really necessarey, as the latter "./build.sh" does
### this either.
# lb clean
### VERY IMPORTANT: Make sure, apt-cacher-ng is started, even, if you are sure,
### it is already running!
# /etc/init.d/apt-cacher-ng stop && sleep 3 && /etc/init.d/apt-cacher-ng start
### set your needed environment and preparations.
# export http_proxy=http://localhost:3142/
lb config
### This command starts the actual build. All commands are in one line.
### However, this script might be improved, to use variables for better
### configuration possibilities.
# ./build.sh --arch $ARCH --distribution $DISTRO --variant $VARIANT --verbose -- --apt-http-proxy $PROXY --bootappend-live $BOOTAPP
./build.sh --arch $ARCH --distribution $DISTRO --variant $VARIANT --verbose -- --bootappend-live "$BOOTAPP"
### these options can be used for further commands after build
### for example, if you want to burn a dvd, shutdown, whatever
### Examples for my personal needs below
#/etc/init.d/apt-cacher-ng stop
### Suggestions for handling the built image.
# cp /home/username/live-build-config/images/*.iso /home/user/
# wodim /path_to_your_live_build/live-build-config/images/your_image_name.iso
### Some opportunities at unattended build run, what to do after the build succeeded.
# dd if=/mnt/kali-linux/amd64/live-build-config/images/kali-linux-rolling-live-lxde-amd64.iso of=/dev/sdf bs=8192 status=progress
# mv -f /mnt/kali-linux/amd64/live-build-config/images/*.iso /mnt/ISO-latest/
# halt -p
# reboot
# shutdown -h now
exit
--- snap --- |
|
|
|
Did you trying cleaning up the dists directories in the apt-cacher-ng cache? Something like:
find /var/cache/apt-cacher-ng/ -name dists -type d
Then rm -fr the list of directories from above. Then retry a build. |
|
|
|
Nope, as descripbed, and see in my script, I am doung an "lb clean --purge" before. So the whole live-build tree is empty, like doing from scratch. |
|
|
|
Please use the 3 back ticks to enclose your code snippets, it looks much better:
```
your code
```
|
|
|
|
So the whole live-build tree is empty, like doing from scratch.
Yes but if you use a caching proxy (apt-cacher-ng), then packages and repo metadata are fetched from your local cache, which is in /var/cache/apt-cacher-ng I suppose.
I used apt-cacher-ng for many years, and I had many weird failures caused by it. Clearing the dists directory in your apt-cacher-ng cache is always a good starting point to deal with weird errors. |
|
|
|
Please note, the problem appears not in an installed system, nor by doing an upgrade of a booted live-system. It appears only at building a live system (running "lb build"). So I guess it might have soemthing to do with the build sources, where the packages are downloaded from. As you know, when building, first of all there are downloaded three package list files, (I am not sure, I believe, it is called "packages.gz", but I do not know exactlly). I believe, they are not corect (but that is just a guess). All in all, this error is looking strange at all.
What I could do, might, just delete the whole stuff, get everything from git again by scratch and tray again. But I strongly suppose, the result might be the same.
What do you think? |
|
|
|
I am not using apt-cacher. In my script it is commented out. Please take a look at the scripts i attached. In the past I tried apt-cacher-ng, but had lots of problems, so I don`t use it since years. |
|
|
|
I am not using apt-cacher. In my script it is commented out.
Ah ok, I see that now, thanks. |
|
|
|
I see that in your logs:
Ign:1 file:/packages ./ InRelease
Get:2 file:/packages ./ Release [1235 B]
Get:2 file:/packages ./ Release [1235 B]
Ign:3 file:/packages ./ Release.gpg
Get:4 file:/packages ./ Packages [1682 B]
What's in this directory? Can you attach those files? Most likely the issue comes from there. |
|
|
|
Note that the line lb config in your script is not needed, since it's done in the script ./build.sh anyway. You should comment it out. |
|
|
|
Sorry Arnaud, I could not answer prior, as the bugsystem has a 3600s delasy.
First: I need to execute "lb config" seperated, just to create my settings, with German language and keyboard etc. The ./build.sh is fine, when using default settings.
In the meantime I think, I dsicovered the cause of the issue.
But when you do an "apt update" in the livefile, compare the files /var/cache/apt/archives/pkgcache.bin of the environtment file and the pkgcache.bin file which is downlaoded during the live buld process findable
in ../kali-live/live-build/chroot/var/cache/apt/. The pkgcache.bin in the chroot is much smaller (guess missing al the entries). Size in kali-native = 43685K, size in chroot = 43441K. Also the size of srcpkgcache.bin differs, but not taht much.
This is something I can not change, even when overwriting, the pkgcache.bin is always overwritten when starting a build.
I also tried fresh downloaded livefile iso, butthe downoadable iso makes no difference. This proves, my image live environment is ok. Still believe, the cause is on the site of the repository (as above discribed). |
|
|
|
I see that in your logs:
Ign:1 file:/packages ./ InRelease
Get:2 file:/packages ./ Release [1235 B]
Get:2 file:/packages ./ Release [1235 B]
Ign:3 file:/packages ./ Release.gpg
Get:4 file:/packages ./ Packages [1682 B]
These messages are normal and generated by apt. Not sure where they come from, but this message exist already since years. I suppose these are files downloaded from the servers. |
|
|
|
These messages are normal and generated by apt. Not sure where they come from, but this message exist already since years. I suppose these are files downloaded from the servers.
These messages means that you have a local repository configured in your apt sources.list, in the chroot environment during the build. I don't have it in my logs, to clarify.
Let me past a longer extract of the logs on your side:
[2026-04-16 15:50:32] lb chroot_archives chroot install
P: Configuring file /etc/apt/sources.list
Ign:1 file:/packages ./ InRelease
Get:2 file:/packages ./ Release [1235 B]
Get:2 file:/packages ./ Release [1235 B]
Ign:3 file:/packages ./ Release.gpg
Get:4 file:/packages ./ Packages [1682 B]
Hit:5 http://kali.download/kali kali-rolling InRelease
Get:6 http://kali.download/kali kali-rolling/non-free Sources [115 kB]
Get:7 http://kali.download/kali kali-rolling/contrib Sources [85.1 kB]
Get:8 http://kali.download/kali kali-rolling/non-free-firmware Sources [11.1 kB]
Get:9 http://kali.download/kali kali-rolling/main Sources [17.7 MB]
Fetched 17.9 MB in 4s (4498 kB/s)
What's written above is that APT fetches metadata from 2 repositories: http://kali.download/kali/ and file:/packages. You need to check what is this local repository packages that is enabled on your side, and what's in there. |
|
|
|
I have tried running your script uit-build-amd64.sh and it works for me.
Previously I asked you if you have a public repo of your kali-live. I can't help you troubleshot further without that. I have no idea what modifications you made in your kali-live repo. If it's public I can have a look and find what's the issue. But if it's private then there's nothing more I can do. |
|
|
|
Hmm, I do not have a public repo in my kali-.live.
FYI I downloaded the latest ISO from the kali site and used that as environment eliminating any bad things from the past.
But same results.
Then I deleted the complete git tree (the live-build tree) and installed a new one, using
clone https://gitlab.com/kalilinux/build-scripts/live-build-config.git
This would eliminate second bad things from the past.
Then did run my uit-config.sh scriipt (to get my settings) anf after it my uit-build-amd64.sh script.
Still same results.
Now I am lost, where else the failure could be. And please remember: Everything worked before! I have nothing changed on my side.
I still believe, something has changed on the repo or server side, which we ae un aware of.
However, to reproduce it, maybe you could try this:
-
Download the Installer-ISO from kali and create a bootable usb-stick.
-
Boot it and create the git tree (clone https://gitlab.com/kalilinux/build-scripts/live-build-config.git)
-
Change the name of the "live-build-config" to "kali-live". (this was recommended by kali several months ago)
-
Put one or two external debian packages into ../kali-live/config/packages.chroot/
-
Execute my scripts (uit-config and uit-build) from within the ../kali-live/ folder.
-
You then should see the error.
If you still do not see it, then please let me know, then I will recheck everything again and do from scratch again.
Please let the bugreport open though, then I can report any progress. |
|
|
|
Ok, good news!
Due to your advice I examined the chroot, where are these entries
Ign:1 file:/packages ./ InRelease
Get:2 file:/packages ./ Release [1235 B]
Get:2 file:/packages ./ Release [1235 B]
Ign:3 file:/packages ./ Release.gpg
Get:4 file:/packages ./ Packages [1682 B]
Hit:5 http://kali.download/kali kali-rolling InRelease
are from.
They are created by the content of deb packages in packages.chroot, which I added to get them in kali-live installed, too.
Now I emptied the folder "packages.chroot" and the issue is gone. What I do not understand, why this issue is caused by these packages. In the past I built about 30 to 40 ISO-images
including these packages and everything worked fine. Maybe dependencies have changed?
As we have found the reason, I now can go further and take a closer look. As this bugreport can be closed, I would like to suggest that wecould let it open for a week or so.
Thus I could write my results here for other people, who might read here, too.
I can only say: Thank you very, very much for your help and your time you spent for me!
Best regards and have a nice weekend!
Hans |
|
|
|
So, after some tests, I found the reason for the issue.
In packages.chroot/ I added several packages which I have included into the livefile I built. Among several other packages was the package
gstreamer1.0-pipewire_1.6.2-1_amd64.deb
which was needed by another package and missing in kali-repo (thus I had to edit it, too, of course).
Now things in kali had changed, but this package recommended older versions from the kali-repo, which are no more existen in kali. Thus the errors.
As my added packages in packages.chroot do not need this gstreamer package any more (because of changes in the dependencies), I could safely emove the gstreamer package off packages.chroot and now
everything is working as wanted.
So the cause of the issue was not easy to find, as I chnged nothing, was unaware (and could not well see), that dependencies had changed due to new versions, and the logs also showed the cause not
clearly enough.
This bugreport can safely be closed.
Last but not least, I have to say: I would not have found the cause of the issue without your help, Arnaud!
So again, thank you very much, please close this bugreprt.
Best
Hans |
|
