View Issue Details

IDProjectCategoryView StatusLast Update
0004109Kali LinuxGeneral Bugpublic2017-08-07 15:33
ReporterAkeo Assigned Torhertzog  
PrioritylowSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Product Version2017.1 
Summary0004109: Kali ISOs use the nonstandard Rock Ridge extension 'AL' - What does it do? Where is it documented?
Description

All,

The KALI ISOs, such as kali-linux-2017.1-amd64.iso, are using Rock Ridge extensions and for the most part, they seem to be compliant with the Rock Ridge standard, through the use of the 'CE', 'NM', 'PX', 'SL', 'SP' and 'TF' SUSP's (https://en.wikipedia.org/wiki/Rock_Ridge#Design_and_contents).

However, it also appears to use the 'AL' SUSP, which seems to be a very nonstandard extension and creates all kind of issues with ISO processing tools (such as the ones based on the latest GNU libcdio, as it rejects 'AL' as a valid Rock Ridge extension, which results in name mangling and other unwanted behaviour - c.f. http://git.savannah.gnu.org/cgit/libcdio.git/tree/lib/iso9660/rock.c?h=release-0.94#n169).

No other Linux distribution I know of, which, in their vast majority, also tend to use Rock Ridge, seem to be using that 'AL' Rock Ridge extension, and I couldn't find that extension documented in any of the IEEE standards, or even in the source for popular ISO generation tools such as genisoimage, so the logical conclusion is that this either is something that is very specific to Kali, or something that comes from the ISO generation tool, but that Kali deliberately chose to enable. Either way, I would expect the the people in charge of the ISO generation should have a very clear idea of why this extension was enabled, what it does, and where it is documented, in which case I will kindly ask that they release that information.

Thanks,

/Pete

Steps To Reproduce

You may need to use a custom tool to see the Rock-Ridge SUSP's being used by an ISO. Or you could use libcdio and add a cdio_warn("'%c%c'\n", *chr, *(chr + 1)); line in the default: case of the code I pointed out, so that it prints the extensions it does not recognizes. I can however guarantee that the latest Kali ISO does use the 'AL' SUSP, even though 'AL' as an extension doesn't seem to be documented anywhere.

Activities

rhertzog

rhertzog

2017-07-24 09:00

administrator   ~0006947

We are using the same "live-build" than Debian and we are not doing anything special AFAIK. The source code of everything is available, please check it out if you want to find out how this is happening.

Akeo

Akeo

2017-07-24 10:06

reporter   ~0006948

Sorry about reopening this issue, but it needs to be pointed out that I wouldn't have been reporting this if I had found that the Debian Live ISOs were using the AL extensions, which they aren't (I checked this with the debian-live-9.x.x-amd64-xfce images).

So Kali does appear to be doing something different than other Linux distros with regards to ISO generation, especially Debian, and I was really hoping I wouldn't have to go through the steps of having to install a full Kali system, then figure out what is the exact process that the Kali maintainers use when generating ISOs, and then analyse your scripts, sources and whatever patches you applied, to figure out what it is that Kali does differently from other distros with regards to ISO generation.

As far as I'm concerned, that's at least one full day of work that I'd like to avoid as, just like yours, my time is in short supply, and, again, my expectation is that someone on the Kali project may be able to provide, in a few minutes work, what will take me at least a full day.

Akeo

Akeo

2017-07-24 19:07

reporter   ~0006949

Finally managed to find that the AL extension is something that the xorriso/libisofs people have pulled as part of their own Arbitrary Attribute Interchange Protocol (AAIP): https://dev.lovelyhq.com/libburnia/libisofs/blob/master/doc/susp_aaip_2_0.txt

Unfortunately, it doesn't look like this got documented further than that project.

I guess the default options used by Kali do enable the AAIP extensions, whereas other distros don't, hence the use of AL.

I'll have a chat with libcdio to find out whether they want to handle AL in their library. At any rate, this issue can be considered closed.

rhertzog

rhertzog

2017-07-25 16:24

administrator   ~0006952

Debian started to use live-wrapper instead of live-build for generating the official images. Currently both tools are available in Debian and we are still relying on live-build which has way more features than live-wrapper.

Our live-build ISO build process is documented here:
https://docs.kali.org/development/live-build-a-custom-kali-iso

scdbackup

scdbackup

2017-07-25 19:51

reporter   ~0006953

Hi,

i am the developer of xorriso and have inspected kali-linux-2017.1-amd64.iso
what AAIP info is in and why that might be.

My favorite theory is that xorriso -as mkisofs option --hardlinks is used.
Regrettably i was yet unable to find the program or script by which the
xorriso command line gets composed.
https://docs.kali.org/development/live-build-a-custom-kali-iso does not
help much, because it only mentions "./build.sh".
Isn't there any online display of the xorriso options used ?

If my theory is true:
Option --hardlinks is not very useful with distro ISOs. Linux will after
mount ignore the recorded extra info and i assume that re-uniting hardlinks
when extracting files from the Kali ISO by xorriso means is not a goal.
On the other hand, it served as trigger for a bug discussion on libcdio
mailing list. So it has its merits now.

In general, the AAIP extension of libisofs is legitimized by the System
Use Protocol which also legitimizes Rock Ridge.
The problem with libcdio is due to a mistaken bug fix, which introduced
a whitelist of permissible SUSP entries, in which "AL" is missing.
So only the ISO 9660 names but not the Rock Ridge names were shown.

Have a nice day :)

Thomas

scdbackup

scdbackup

2017-07-26 09:07

reporter   ~0006955

Hi,

thanks to Ady Ady, whom i know from SYSLINUX mailing list, i see a candidate
for my suspicion in
https://anonscm.debian.org/cgit/debian-live/live-build.git/plain/scripts/build/binary_iso

which has:

XORRISO_OPTIONS="${XORRISO_OPTIONS} -no-emul-boot -boot-load-size 4 -boot-info-table --hardlinks"

My test proposal is to remove the option --hardlinks from the variable,
to produce an ISO, and to check whether it still has "AL" entries by

xorriso -xattr on -indev $x -find / -has_any_xattr -exec get_any_xattr -- \
2>&1 | less

With kali-linux-2017.1-amd64.iso it reports

file: .

isofs.di="\002\011\002\004\001\222\001p"
isofs.st="1492308594"
isofs.nt="\001\001\001\377"

file: .disk

isofs.di="\002\011\002\004\001\230\015D"

...

A KALI ISO made without option --hardlinks should yield no output on stdout
from above xorriso run. (Only some info messages on stderr.)

While the use of --hardlinks is not wrong, it is somewhat exotic.
The projects of libcdio and libisofs have to thank you for putting that
exotic species into the wild.
But the KALI LINUX project might want to join the community of less
playful ISO producers, nevertheless.

Have a nice day :)

Thomas

scdbackup

scdbackup

2017-07-26 09:08

reporter   ~0006956

Urm, i forgot to mention that "$x" is the path to the freshly produced ISO.

scdbackup

scdbackup

2017-07-26 09:31

reporter   ~0006957

Hi,

Ady also pointed me to the commit which introduced --hardlinks

https://anonscm.debian.org/cgit/debian-live/live-build.git/commit/scripts/build/binary_iso?id=9c974b26bf532ba76f6d66423310f6d95b1ff1f7

by Raphaƫl Hertzog from december 2016 which says:

"Instead of renaming kernel for syslinux, create hardlinks"

I currently fail to imagine how --hardlinks can help with renaming or
avoiding to rename a SYSLINUX kernel.

The only non-AAIP aspect of that option is that the File Serial
Numbers in Rock Ridge entry PX of hardlink siblings get set to the
same (inode) number.
But afaik, Linux ignores the inode number in PX and generates an
own one from the higher bits of the directory entry byte address.

So how did adding --hardlinks make the KALI ISO better ?

Have a nice day :)

Thomas

rhertzog

rhertzog

2017-07-29 10:48

administrator   ~0006959

Hi Thomas, I wanted to have the kernel image available under two different names for the differing needs of syslinux and grub-efi. Before we introduced EFI support, we would rename the kernel to cope with the short filenames required by syslinux and it was OK since syslinux was the only bootloader at that time.

Now with the addition of the EFI support, to avoid losing spaces due to two copies of the same kernel image, I created an hardlink with the name needed for syslinux. Thus I added the --hardlinks option to ensure that this property was kept in the ISO image.

I can get rid of the option... but I would like to be sure that I'm not needlessly growing the image by doing it.

Thanks for your analysis BTW!

scdbackup

scdbackup

2017-07-29 12:02

reporter   ~0006960

Hi,

the unification of hardlink content inside the ISO is a base feature
of libisofs and of mkisofs, too.

To be sure, i now tested with xorriso-1.4.6 (without option --hardlinks)
and mkisofs-2.01.01a64.
Then i let xorriso show the data blocks of the files by:

$ xorriso -indev test.iso -find / -exec report_lba --
...
Report layout: xt , Startlba , Blocks , Filesize , ISO image path
File data lba: 0 , 33 , 3906 , 7998480 , '/NBX'
File data lba: 0 , 33 , 3906 , 7998480 , '/NBX.hardlink'

So files which were hardlinks on disk share their data storage in the ISO.
This content sharing in a read-only filesystem is not 1:1 equivalent
to a hardlink relation, nevertheless.

The purpose of option --hardlinks is to mark real hardlink relations and
to prepare for restoring of above two files as hardlinks by xorriso
commands. See below.

Further the recorded extra info is necessary to determine during
incremental backups whether a hardlink family gained or lost members.
(The same info can speed up incremental backup provided that device
numbers are stable over reboot.)


So if KALI would strive for copying its files out of the ISO and for
getting the hardlink relation between both files restored to disk, then
the option --hardlinks would be necessary.

As said, Linux does not map Rock Ridge File Serial Numbers to ino_t:

$ ls -li input_files_on_disk
total 15624
3539013 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX
3539013 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX.hardlink

$ ls -li /mnt/iso
total 15623
1159 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX
1163 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX.hardlink

(Note the contradiction of hard link counter and ino_t numbers. The ino_t
numbers are the byte addresses of the ISO Directory Records divided by 32.)

So cp -a or other archive-grade copying would create two file copies.
But option --hardlinks at ISO production time gives xorriso the info
to keep the hardlinks united when copying them to disk:

$ xorriso -hardlinks on -osirrox on -indev test.iso -extract / restored_files_on_disk
...
Extracted from ISO image: file '/'='...absolute.path.../restored_files_on_disk'

$ ls -li restored_files_on_disk
total 15624
39102028 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX
39102028 -rwxr-xr-x 2 thomas thomas 7998480 Jul 27 23:26 NBX.hardlink

(I advise to use in backup situations the xorriso command -for_backup
which combines processing of hardlinks, xattr, ACL, and MD5 checksumming.)

Have a nice day :)

Thomas

rhertzog

rhertzog

2017-08-07 15:33

administrator   ~0006973

Thanks Thomas for this information. I uploaded a new version of live-build to Debian unstable that no longer uses --hardlinks. It's version 1:20170807.

It will migrate to Kali in approximately 10 days.

Issue History

Date Modified Username Field Change
2017-07-21 11:33 Akeo New Issue
2017-07-24 09:00 rhertzog Assigned To => rhertzog
2017-07-24 09:00 rhertzog Status new => closed
2017-07-24 09:00 rhertzog Resolution open => no change required
2017-07-24 09:00 rhertzog Note Added: 0006947
2017-07-24 10:06 Akeo Status closed => feedback
2017-07-24 10:06 Akeo Resolution no change required => reopened
2017-07-24 10:06 Akeo Note Added: 0006948
2017-07-24 19:07 Akeo Note Added: 0006949
2017-07-24 19:07 Akeo Status feedback => assigned
2017-07-25 16:24 rhertzog Note Added: 0006952
2017-07-25 19:51 scdbackup Note Added: 0006953
2017-07-26 09:07 scdbackup Note Added: 0006955
2017-07-26 09:08 scdbackup Note Added: 0006956
2017-07-26 09:31 scdbackup Note Added: 0006957
2017-07-29 10:48 rhertzog Note Added: 0006959
2017-07-29 12:02 scdbackup Note Added: 0006960
2017-08-07 15:33 rhertzog Status assigned => resolved
2017-08-07 15:33 rhertzog Resolution reopened => fixed
2017-08-07 15:33 rhertzog Note Added: 0006973