View Issue Details

IDProjectCategoryView StatusLast Update
0005613Kali LinuxFeature Requestspublic2021-07-07 15:24
ReporterBalling Assigned Torhertzog  
PrioritynormalSeverityminorReproducibilityunable to reproduce
Status assignedResolutionopen 
Product Versionkali-dev 
Summary0005613: Deploy support for (p)diff incremental for apt-get update on https://http.kali.org/kali
Description

Сущ:1 http://files.viva64.com/deb viva64-release InRelease
Игн:2 http://dl.google.com/linux/chrome/deb stable InRelease
Сущ:3 http://dl.google.com/linux/chrome/deb stable Release
Пол:5 https://cdn-aws.deb.debian.org/debian experimental InRelease [78.9 kB]
Пол:7 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Packages.diff/Index [27.8 kB]
Пол:8 https://cdn-aws.deb.debian.org/debian experimental/main i386 Packages.diff/Index [27.8 kB]
Пол:9 https://cdn-aws.deb.debian.org/debian experimental/main Translation-en.diff/Index [27.8 kB]
Пол:10 https://cdn-aws.deb.debian.org/debian experimental/main i386 Contents (deb).diff/Index [27.9 kB]
Пол:11 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Contents (deb).diff/Index [27.9 kB]
Пол:6 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling InRelease [30.5 kB]
Пол:12 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Packages 2019-08-04-0831.20.pdiff [9,063 B]
Пол:12 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Packages 2019-08-04-0831.20.pdiff [9,063 B]
Пол:13 https://cdn-aws.deb.debian.org/debian experimental/main i386 Packages 2019-08-04-0831.20.pdiff [9,097 B]
Пол:13 https://cdn-aws.deb.debian.org/debian experimental/main i386 Packages 2019-08-04-0831.20.pdiff [9,097 B]
Пол:14 https://cdn-aws.deb.debian.org/debian experimental/main Translation-en 2019-08-04-0831.20.pdiff [6,018 B]
Пол:14 https://cdn-aws.deb.debian.org/debian experimental/main Translation-en 2019-08-04-0831.20.pdiff [6,018 B]
Пол:15 https://cdn-aws.deb.debian.org/debian experimental/main i386 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB]
Пол:15 https://cdn-aws.deb.debian.org/debian experimental/main i386 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB]
Пол:16 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/contrib Sources [66.0 kB]
Пол:17 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB]
Пол:17 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB]
Пол:18 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free Sources [136 kB]
Пол:19 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main Sources [12.8 MB]
Пол:20 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main i386 Packages [17.0 MB]
Пол:21 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main amd64 Packages [17.1 MB]
Пол:22 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main amd64 Contents (deb) [36.9 MB]
Пол:23 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main i386 Contents (deb) [36.9 MB]
Пол:24 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free i386 Packages [168 kB]
Пол:25 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free amd64 Packages [192 kB]
Пол:26 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free i386 Contents (deb) [858 kB]
Пол:27 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free amd64 Contents (deb) [925 kB]
Пол:28 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/contrib amd64 Packages [108 kB]

As you can see debian has incremental apt-get update (.pdiff and .diff) while with kali I have to download every day (practically) 122 MiB!!!
Obviously you will benefit from it as well. So please! It is as cool as https support. Thanks.

Activities

crash

crash

2019-08-09 09:25

reporter   ~0010837

Hi Balling,
I guess it is under evaluation.

Balling

Balling

2019-08-09 09:33

reporter   ~0010838

crash, the real problem that I cannot find pdiff server side open source implementation(( I suppose debian (and Amazon which help them) should have it, though...

About evaluation, I suppose it will be like 3-5 % more CPU time, but many, many times less bandwidth use... Because you do not need to do anything on kali destro side, it is all already supported (by apt).

crash

crash

2019-08-09 09:43

reporter   ~0010840

Yeah sure. I totally understand your point. (:

Regarding the evaluation, I meant I guess the pdiff implementation is under evaluation (:

Balling

Balling

2019-08-09 09:46

reporter   ~0010841

crash, cannot you ask debian for it? It can't be propriatory! I mean it is debian and all.

crash

crash

2019-08-09 09:47

reporter   ~0010842

I will chase it. (:

Balling

Balling

2019-08-09 10:37

reporter   ~0010843

crash, that was hard to google but --pdiff=mirror in debmirror can do the trick, no? Are you using it, of are you using apt-mirror (apt-cacher(-ng))?

Balling

Balling

2019-08-09 10:54

reporter   ~0010844

crash,
I found the utility debian use to generate diffs and pdiffs. Tell that to maintainers. It is dak (Debian Archive Kit).
https://wiki.debian.org/DebianRepository/Setup

dak generate-index-diffs

https://github.com/Debian/dak/blob/50ced4f9160d0fc5627a6b4ce717865165d87891/config/debian/dinstall.functions#L164

rhertzog

rhertzog

2019-08-13 20:42

administrator   ~0010863

We are using reprepro to manage our repositories. It doesn't support "pdiff" files natively though it provides some "example" script that is supposed to be able to generate those:
/usr/share/doc/reprepro/examples/pdiff.example.gz

I'm not sure whether that script still works and there are many warnings in the top-level comment. It needs some tests and further investigation...

Balling

Balling

2019-08-13 21:19

reporter   ~0010864

BTW, another benefit, I suppose as now kali update repo on every boot, it is even better for your servers.

"example" script

No! It is implemetation from 2006-2014! Do not use it please! https://salsa.debian.org/brlink/reprepro/commits/debian/docs/pdiff.example

You must use dak for this purpose. Reprepro is in python, and dak is as well.

Look for example how it is done in pureos. Thanks.

rhertzog

rhertzog

2019-08-15 13:38

administrator   ~0010875

Old code is not necessarily broken and no we won't switch to "dak" just for this feature. Changing the archive management software is not an easy operation and has many implications on other parts of our infrastructure.

BTW reprepro is written in C, but the pdiff example script is written in Python.

Balling

Balling

2019-08-15 13:47

reporter   ~0010878

Anyway, this script is derived from dak
https://github.com/ionos-enterprise/reprepro/blob/d18e2ed95520e34aca4f7f84c1364c2cac4da51e/docs/manual.html#L609
So you can as well use dak...

rhertzog

rhertzog

2019-08-15 14:51

administrator   ~0010888

Note to myself: look into using "rredtool" to generate those .diff files.

Balling

Balling

2019-09-20 15:31

reporter   ~0011137

@rhertzog
By the way if you open https://pkg.kali.org/derivative/kali-roll/ it has a name "Data about this Debian derivative",
so you are managing the difference with Debian, maybe you could use --pdiff=mirror in debmirror to clone all data which is the same as Debian and then just provide all derivative packets without diff and pdiff? There should be a way not to recompile pdiff and diff files...

Issue History

Date Modified Username Field Change
2019-08-04 14:34 Balling New Issue
2019-08-09 09:25 crash Note Added: 0010837
2019-08-09 09:33 Balling Note Added: 0010838
2019-08-09 09:43 crash Note Added: 0010840
2019-08-09 09:46 Balling Note Added: 0010841
2019-08-09 09:47 crash Note Added: 0010842
2019-08-09 10:37 Balling Note Added: 0010843
2019-08-09 10:54 Balling Note Added: 0010844
2019-08-13 20:42 rhertzog Note Added: 0010863
2019-08-13 20:42 rhertzog Assigned To => rhertzog
2019-08-13 20:42 rhertzog Status new => assigned
2019-08-13 21:19 Balling Note Added: 0010864
2019-08-15 13:38 rhertzog Note Added: 0010875
2019-08-15 13:47 Balling Note Added: 0010878
2019-08-15 14:51 rhertzog Note Added: 0010888
2019-09-20 15:31 Balling Note Added: 0011137