View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0005613 | Kali Linux | [All Projects] Feature Requests | public | 2019-08-04 14:34 | 2021-07-07 15:24 |
| Reporter | Balling | Assigned To | rhertzog | ||
| Priority | normal | Severity | minor | Reproducibility | unable to reproduce |
| Status | assigned | Resolution | open | ||
| Product Version | kali-dev | ||||
| Target Version | Fixed in Version | ||||
| Summary | 0005613: Deploy support for (p)diff incremental for apt-get update on https://http.kali.org/kali | ||||
| Description | Сущ:1 http://files.viva64.com/deb viva64-release InRelease Игн:2 http://dl.google.com/linux/chrome/deb stable InRelease Сущ:3 http://dl.google.com/linux/chrome/deb stable Release Пол:5 https://cdn-aws.deb.debian.org/debian experimental InRelease [78.9 kB] Пол:7 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Packages.diff/Index [27.8 kB] Пол:8 https://cdn-aws.deb.debian.org/debian experimental/main i386 Packages.diff/Index [27.8 kB] Пол:9 https://cdn-aws.deb.debian.org/debian experimental/main Translation-en.diff/Index [27.8 kB] Пол:10 https://cdn-aws.deb.debian.org/debian experimental/main i386 Contents (deb).diff/Index [27.9 kB] Пол:11 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Contents (deb).diff/Index [27.9 kB] Пол:6 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling InRelease [30.5 kB] Пол:12 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Packages 2019-08-04-0831.20.pdiff [9,063 B] Пол:12 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Packages 2019-08-04-0831.20.pdiff [9,063 B] Пол:13 https://cdn-aws.deb.debian.org/debian experimental/main i386 Packages 2019-08-04-0831.20.pdiff [9,097 B] Пол:13 https://cdn-aws.deb.debian.org/debian experimental/main i386 Packages 2019-08-04-0831.20.pdiff [9,097 B] Пол:14 https://cdn-aws.deb.debian.org/debian experimental/main Translation-en 2019-08-04-0831.20.pdiff [6,018 B] Пол:14 https://cdn-aws.deb.debian.org/debian experimental/main Translation-en 2019-08-04-0831.20.pdiff [6,018 B] Пол:15 https://cdn-aws.deb.debian.org/debian experimental/main i386 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB] Пол:15 https://cdn-aws.deb.debian.org/debian experimental/main i386 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB] Пол:16 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/contrib Sources [66.0 kB] Пол:17 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB] Пол:17 https://cdn-aws.deb.debian.org/debian experimental/main amd64 Contents (deb) 2019-08-04-0831.20.pdiff [40.1 kB] Пол:18 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free Sources [136 kB] Пол:19 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main Sources [12.8 MB] Пол:20 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main i386 Packages [17.0 MB] Пол:21 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main amd64 Packages [17.1 MB] Пол:22 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main amd64 Contents (deb) [36.9 MB] Пол:23 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/main i386 Contents (deb) [36.9 MB] Пол:24 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free i386 Packages [168 kB] Пол:25 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free amd64 Packages [192 kB] Пол:26 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free i386 Contents (deb) [858 kB] Пол:27 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/non-free amd64 Contents (deb) [925 kB] Пол:28 https://ftp.acc.umu.se/mirror/kali.org/kali kali-rolling/contrib amd64 Packages [108 kB] As you can see debian has incremental apt-get update (.pdiff and .diff) while with kali I have to download every day (practically) 122 MiB!!! Obviously you will benefit from it as well. So please! It is as cool as https support. Thanks. | ||||
|
|
Hi Balling, I guess it is under evaluation. |
|
|
crash, the real problem that I cannot find pdiff server side open source implementation(( I suppose debian (and Amazon which help them) should have it, though... About evaluation, I suppose it will be like 3-5 % more CPU time, but many, many times less bandwidth use... Because you do not need to do anything on kali destro side, it is all already supported (by apt). |
|
|
Yeah sure. I totally understand your point. (: Regarding the evaluation, I meant I guess the pdiff implementation is under evaluation (: |
|
|
crash, cannot you ask debian for it? It can't be propriatory! I mean it is debian and all. |
|
|
I will chase it. (: |
|
|
crash, that was hard to google but --pdiff=mirror in debmirror can do the trick, no? Are you using it, of are you using apt-mirror (apt-cacher(-ng))? |
|
|
crash, I found the utility debian use to generate diffs and pdiffs. Tell that to maintainers. It is dak (Debian Archive Kit). https://wiki.debian.org/DebianRepository/Setup dak generate-index-diffs https://github.com/Debian/dak/blob/50ced4f9160d0fc5627a6b4ce717865165d87891/config/debian/dinstall.functions#L164 |
|
|
We are using reprepro to manage our repositories. It doesn't support "pdiff" files natively though it provides some "example" script that is supposed to be able to generate those: /usr/share/doc/reprepro/examples/pdiff.example.gz I'm not sure whether that script still works and there are many warnings in the top-level comment. It needs some tests and further investigation... |
|
|
BTW, another benefit, I suppose as now kali update repo on every boot, it is even better for your servers. >"example" script No! It is implemetation from 2006-2014! Do not use it please! https://salsa.debian.org/brlink/reprepro/commits/debian/docs/pdiff.example You must use dak for this purpose. Reprepro is in python, and dak is as well. Look for example how it is done in pureos. Thanks. |
|
|
Old code is not necessarily broken and no we won't switch to "dak" just for this feature. Changing the archive management software is not an easy operation and has many implications on other parts of our infrastructure. BTW reprepro is written in C, but the pdiff example script is written in Python. |
|
|
Anyway, this script is derived from dak https://github.com/ionos-enterprise/reprepro/blob/d18e2ed95520e34aca4f7f84c1364c2cac4da51e/docs/manual.html#L609 So you can as well use dak... |
|
|
Note to myself: look into using "rredtool" to generate those .diff files. |
|
|
@rhertzog By the way if you open https://pkg.kali.org/derivative/kali-roll/ it has a name "Data about this Debian derivative", so you are managing the difference with Debian, maybe you could use --pdiff=mirror in debmirror to clone all data which is the same as Debian and then just provide all derivative packets without diff and pdiff? There should be a way not to recompile pdiff and diff files... |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2019-08-04 14:34 | Balling | New Issue | |
| 2019-08-09 09:25 | crash | Note Added: 0010837 | |
| 2019-08-09 09:33 | Balling | Note Added: 0010838 | |
| 2019-08-09 09:43 | crash | Note Added: 0010840 | |
| 2019-08-09 09:46 | Balling | Note Added: 0010841 | |
| 2019-08-09 09:47 | crash | Note Added: 0010842 | |
| 2019-08-09 10:37 | Balling | Note Added: 0010843 | |
| 2019-08-09 10:54 | Balling | Note Added: 0010844 | |
| 2019-08-13 20:42 | rhertzog | Note Added: 0010863 | |
| 2019-08-13 20:42 | rhertzog | Assigned To | => rhertzog |
| 2019-08-13 20:42 | rhertzog | Status | new => assigned |
| 2019-08-13 21:19 | Balling | Note Added: 0010864 | |
| 2019-08-15 13:38 | rhertzog | Note Added: 0010875 | |
| 2019-08-15 13:47 | Balling | Note Added: 0010878 | |
| 2019-08-15 14:51 | rhertzog | Note Added: 0010888 | |
| 2019-09-20 15:31 | Balling | Note Added: 0011137 |