We are using the same "live-build" than Debian and we are not doing anything special AFAIK. The source code of everything is available, please check it out if you want to find out how this is happening.

Sorry about reopening this issue, but it needs to be pointed out that I wouldn't have been reporting this if I had found that the Debian Live ISOs were using the AL extensions, which they aren't (I checked this with the debian-live-9.x.x-amd64-xfce images).

So Kali does appear to be doing something different than other Linux distros with regards to ISO generation, especially Debian, and I was really hoping I wouldn't have to go through the steps of having to install a full Kali system, then figure out what is the exact process that the Kali maintainers use when generating ISOs, and then analyse your scripts, sources and whatever patches you applied, to figure out what it is that Kali does differently from other distros with regards to ISO generation.

As far as I'm concerned, that's at least one full day of work that I'd like to avoid as, just like yours, my time is in short supply, and, again, my expectation is that someone on the Kali project may be able to provide, in a few minutes work, what will take me at least a full day.

Finally managed to find that the AL extension is something that the xorriso/libisofs people have pulled as part of their own Arbitrary Attribute Interchange Protocol (AAIP): https://dev.lovelyhq.com/libburnia/libisofs/blob/master/doc/susp_aaip_2_0.txt

Unfortunately, it doesn't look like this got documented further than that project.

I guess the default options used by Kali do enable the AAIP extensions, whereas other distros don't, hence the use of AL.

I'll have a chat with libcdio to find out whether they want to handle AL in their library. At any rate, this issue can be considered closed.

Debian started to use live-wrapper instead of live-build for generating the official images. Currently both tools are available in Debian and we are still relying on live-build which has way more features than live-wrapper.

Our live-build ISO build process is documented here:
https://docs.kali.org/development/live-build-a-custom-kali-iso

Hi,

i am the developer of xorriso and have inspected kali-linux-2017.1-amd64.iso
what AAIP info is in and why that might be.

My favorite theory is that xorriso -as mkisofs option --hardlinks is used.
Regrettably i was yet unable to find the program or script by which the
xorriso command line gets composed.
https://docs.kali.org/development/live-build-a-custom-kali-iso does not
help much, because it only mentions "./build.sh".
Isn't there any online display of the xorriso options used ?

If my theory is true:
Option --hardlinks is not very useful with distro ISOs. Linux will after
mount ignore the recorded extra info and i assume that re-uniting hardlinks
when extracting files from the Kali ISO by xorriso means is not a goal.
On the other hand, it served as trigger for a bug discussion on libcdio
mailing list. So it has its merits now.

In general, the AAIP extension of libisofs is legitimized by the System
Use Protocol which also legitimizes Rock Ridge.
The problem with libcdio is due to a mistaken bug fix, which introduced
a whitelist of permissible SUSP entries, in which "AL" is missing.
So only the ISO 9660 names but not the Rock Ridge names were shown.

Have a nice day :)

Thomas

Hi,

thanks to Ady Ady, whom i know from SYSLINUX mailing list, i see a candidate
for my suspicion in
https://anonscm.debian.org/cgit/debian-live/live-build.git/plain/scripts/build/binary_iso

which has:

XORRISO_OPTIONS="${XORRISO_OPTIONS} -no-emul-boot -boot-load-size 4 -boot-info-table --hardlinks"

My test proposal is to remove the option --hardlinks from the variable,
to produce an ISO, and to check whether it still has "AL" entries by

xorriso -xattr on -indev $x -find / -has_any_xattr -exec get_any_xattr -- \
2>&1 | less

With kali-linux-2017.1-amd64.iso it reports

file: .

isofs.di="\002\011\002\004\001\222\001p"
isofs.st="1492308594"
isofs.nt="\001\001\001\377"

file: .disk

isofs.di="\002\011\002\004\001\230\015D"

...

A KALI ISO made without option --hardlinks should yield no output on stdout
from above xorriso run. (Only some info messages on stderr.)

While the use of --hardlinks is not wrong, it is somewhat exotic.
The projects of libcdio and libisofs have to thank you for putting that
exotic species into the wild.
But the KALI LINUX project might want to join the community of less
playful ISO producers, nevertheless.

Have a nice day :)

Thomas

Urm, i forgot to mention that "$x" is the path to the freshly produced ISO.

Hi,

Ady also pointed me to the commit which introduced --hardlinks

https://anonscm.debian.org/cgit/debian-live/live-build.git/commit/scripts/build/binary_iso?id=9c974b26bf532ba76f6d66423310f6d95b1ff1f7

by Raphaël Hertzog from december 2016 which says:

"Instead of renaming kernel for syslinux, create hardlinks"

I currently fail to imagine how --hardlinks can help with renaming or
avoiding to rename a SYSLINUX kernel.

The only non-AAIP aspect of that option is that the File Serial
Numbers in Rock Ridge entry PX of hardlink siblings get set to the
same (inode) number.
But afaik, Linux ignores the inode number in PX and generates an
own one from the higher bits of the directory entry byte address.

So how did adding --hardlinks make the KALI ISO better ?

Have a nice day :)

Thomas

Hi Thomas, I wanted to have the kernel image available under two different names for the differing needs of syslinux and grub-efi. Before we introduced EFI support, we would rename the kernel to cope with the short filenames required by syslinux and it was OK since syslinux was the only bootloader at that time.

Now with the addition of the EFI support, to avoid losing spaces due to two copies of the same kernel image, I created an hardlink with the name needed for syslinux. Thus I added the --hardlinks option to ensure that this property was kept in the ISO image.

I can get rid of the option... but I would like to be sure that I'm not needlessly growing the image by doing it.

Thanks for your analysis BTW!

Hi,

the unification of hardlink content inside the ISO is a base feature
of libisofs and of mkisofs, too.

To be sure, i now tested with xorriso-1.4.6 (without option --hardlinks)
and mkisofs-2.01.01a64.
Then i let xorriso show the data blocks of the files by:

$ xorriso -indev test.iso -find / -exec report_lba --
...
Report layout: xt , Startlba , Blocks , Filesize , ISO image path
File data lba: 0 , 33 , 3906 , 7998480 , '/NBX'
File data lba: 0 , 33 , 3906 , 7998480 , '/NBX.hardlink'

So files which were hardlinks on disk share their data storage in the ISO.
This content sharing in a read-only filesystem is not 1:1 equivalent
to a hardlink relation, nevertheless.

The purpose of option --hardlinks is to mark real hardlink relations and
to prepare for restoring of above two files as hardlinks by xorriso
commands. See below.

Further the recorded extra info is necessary to determine during
incremental backups whether a hardlink family gained or lost members.
(The same info can speed up incremental backup provided that device
numbers are stable over reboot.)

So if KALI would strive for copying its files out of the ISO and for
getting the hardlink relation between both files restored to disk, then
the option --hardlinks would be necessary.

As said, Linux does not map Rock Ridge File Serial Numbers to ino_t:

$ ls -li input_files_on_disk
total 15624
3539013 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX
3539013 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX.hardlink

$ ls -li /mnt/iso
total 15623
1159 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX
1163 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX.hardlink

(Note the contradiction of hard link counter and ino_t numbers. The ino_t
numbers are the byte addresses of the ISO Directory Records divided by 32.)

So cp -a or other archive-grade copying would create two file copies.
But option --hardlinks at ISO production time gives xorriso the info
to keep the hardlinks united when copying them to disk:

$ xorriso -hardlinks on -osirrox on -indev test.iso -extract / restored_files_on_disk
...
Extracted from ISO image: file '/'='...absolute.path.../restored_files_on_disk'

$ ls -li restored_files_on_disk
total 15624
39102028 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX
39102028 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX.hardlink

(I advise to use in backup situations the xorriso command -for_backup
which combines processing of hardlinks, xattr, ACL, and MD5 checksumming.)

Have a nice day :)

Thomas

Thanks Thomas for this information. I uploaded a new version of live-build to Debian unstable that no longer uses --hardlinks. It's version 1:20170807.

It will migrate to Kali in approximately 10 days.