View Issue Details

IDProjectCategoryView StatusLast Update
0005613Kali Linux[All Projects] Feature Requestspublic2021-07-07 15:24
ReporterBalling Assigned Torhertzog  
PrioritynormalSeverityminorReproducibilityunable to reproduce
Status assignedResolutionopen 
Product Versionkali-dev 
Target VersionFixed in Version 
Summary0005613: Deploy support for (p)diff incremental for apt-get update on
DescriptionСущ:1 viva64-release InRelease
Игн:2 stable InRelease
Сущ:3 stable Release
Пол:5 experimental InRelease [78.9 kB]
Пол:7 experimental/main amd64 Packages.diff/Index [27.8 kB]
Пол:8 experimental/main i386 Packages.diff/Index [27.8 kB]
Пол:9 experimental/main Translation-en.diff/Index [27.8 kB]
Пол:10 experimental/main i386 Contents (deb).diff/Index [27.9 kB]
Пол:11 experimental/main amd64 Contents (deb).diff/Index [27.9 kB]
Пол:6 kali-rolling InRelease [30.5 kB]
Пол:12 experimental/main amd64 Packages 2019-08-04-0831.20.pdiff [9,063 B]
Пол:12 experimental/main amd64 Packages 2019-08-04-0831.20.pdiff [9,063 B]
Пол:13 experimental/main i386 Packages 2019-08-04-0831.20.pdiff [9,097 B]
Пол:13 experimental/main i386 Packages 2019-08-04-0831.20.pdiff [9,097 B]
Пол:14 experimental/main Translation-en 2019-08-04-0831.20.pdiff [6,018 B]
Пол:14 experimental/main Translation-en 2019-08-04-0831.20.pdiff [6,018 B]
Пол:15 experimental/main i386 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB]
Пол:15 experimental/main i386 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB]
Пол:16 kali-rolling/contrib Sources [66.0 kB]
Пол:17 experimental/main amd64 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB]
Пол:17 experimental/main amd64 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB]
Пол:18 kali-rolling/non-free Sources [136 kB]
Пол:19 kali-rolling/main Sources [12.8 MB]
Пол:20 kali-rolling/main i386 Packages [17.0 MB]
Пол:21 kali-rolling/main amd64 Packages [17.1 MB]
Пол:22 kali-rolling/main amd64 Contents (deb) [36.9 MB]
Пол:23 kali-rolling/main i386 Contents (deb) [36.9 MB]
Пол:24 kali-rolling/non-free i386 Packages [168 kB]
Пол:25 kali-rolling/non-free amd64 Packages [192 kB]
Пол:26 kali-rolling/non-free i386 Contents (deb) [858 kB]
Пол:27 kali-rolling/non-free amd64 Contents (deb) [925 kB]
Пол:28 kali-rolling/contrib amd64 Packages [108 kB]

As you can see debian has incremental apt-get update (.pdiff and .diff) while with kali I have to download every day (practically) 122 MiB!!!
Obviously you will benefit from it as well. So please! It is as cool as https support. Thanks.



2019-08-09 09:25

reporter   ~0010837

Hi Balling,
I guess it is under evaluation.


2019-08-09 09:33

reporter   ~0010838

crash, the real problem that I cannot find pdiff server side open source implementation(( I suppose debian (and Amazon which help them) should have it, though...

About evaluation, I suppose it will be like 3-5 % more CPU time, but many, many times less bandwidth use... Because you do not need to do anything on kali destro side, it is all already supported (by apt).


2019-08-09 09:43

reporter   ~0010840

Yeah sure. I totally understand your point. (:

Regarding the evaluation, I meant I guess the pdiff implementation is under evaluation (:


2019-08-09 09:46

reporter   ~0010841

crash, cannot you ask debian for it? It can't be propriatory! I mean it is debian and all.


2019-08-09 09:47

reporter   ~0010842

I will chase it. (:


2019-08-09 10:37

reporter   ~0010843

crash, that was hard to google but --pdiff=mirror in debmirror can do the trick, no? Are you using it, of are you using apt-mirror (apt-cacher(-ng))?


2019-08-09 10:54

reporter   ~0010844

I found the utility debian use to generate diffs and pdiffs. Tell that to maintainers. It is dak (Debian Archive Kit).

dak generate-index-diffs


2019-08-13 20:42

administrator   ~0010863

We are using reprepro to manage our repositories. It doesn't support "pdiff" files natively though it provides some "example" script that is supposed to be able to generate those:

I'm not sure whether that script still works and there are many warnings in the top-level comment. It needs some tests and further investigation...


2019-08-13 21:19

reporter   ~0010864

BTW, another benefit, I suppose as now kali update repo on every boot, it is even better for your servers.

 >"example" script

No! It is implemetation from 2006-2014! Do not use it please!

You must use dak for this purpose. Reprepro is in python, and dak is as well.

Look for example how it is done in pureos. Thanks.


2019-08-15 13:38

administrator   ~0010875

Old code is not necessarily broken and no we won't switch to "dak" just for this feature. Changing the archive management software is not an easy operation and has many implications on other parts of our infrastructure.

BTW reprepro is written in C, but the pdiff example script is written in Python.


2019-08-15 13:47

reporter   ~0010878

Anyway, this script is derived from dak
So you can as well use dak...


2019-08-15 14:51

administrator   ~0010888

Note to myself: look into using "rredtool" to generate those .diff files.


2019-09-20 15:31

reporter   ~0011137

By the way if you open it has a name "Data about this Debian derivative",
so you are managing the difference with Debian, maybe you could use --pdiff=mirror in debmirror to clone all data which is the same as Debian and then just provide all derivative packets without diff and pdiff? There should be a way not to recompile pdiff and diff files...

Issue History

Date Modified Username Field Change
2019-08-04 14:34 Balling New Issue
2019-08-09 09:25 crash Note Added: 0010837
2019-08-09 09:33 Balling Note Added: 0010838
2019-08-09 09:43 crash Note Added: 0010840
2019-08-09 09:46 Balling Note Added: 0010841
2019-08-09 09:47 crash Note Added: 0010842
2019-08-09 10:37 Balling Note Added: 0010843
2019-08-09 10:54 Balling Note Added: 0010844
2019-08-13 20:42 rhertzog Note Added: 0010863
2019-08-13 20:42 rhertzog Assigned To => rhertzog
2019-08-13 20:42 rhertzog Status new => assigned
2019-08-13 21:19 Balling Note Added: 0010864
2019-08-15 13:38 rhertzog Note Added: 0010875
2019-08-15 13:47 Balling Note Added: 0010878
2019-08-15 14:51 rhertzog Note Added: 0010888
2019-09-20 15:31 Balling Note Added: 0011137