Wiki . Forums . Bugs . Translation . Privacy Policy | Search . Simple . Advanced | Reports . Help | Matrix . 10.x . 11.x . 12.x . 13.x . Future . Recent . Contribs

Bug 8834 - install deltarpm to reduce updates sizes
Summary: install deltarpm to reduce updates sizes
Status: CLOSED FIXED
Alias: None
Product: SME Server 10.X
Classification: SME Server
Component: New Feature Request (show other bugs)
Version: 10.0a1
Hardware: --- ---
: P3 normal
Target Milestone: 10.0a2
Assignee: SME Server bug team
QA Contact: SME Server bug team
URL:
: 2878 (view as bug list)
Depends on:
Blocks:
 
Reported: 2015-02-05 15:52 CET by RequestedDeletion
Modified: 2017-06-14 11:21 CEST (History)
6 users (show)

See Also:
Package: smeserver-yum


Attachments
smeserver-yum-2.6.0.bz8834.DeltaRpm.patch (1.97 KB, patch)
2016-07-24 23:04 CEST, Stephane de Labrusse
Details | Diff
smeserver-yum-2.6.0.bz8834.DeltaRpm.patch (1.96 KB, patch)
2016-07-24 23:07 CEST, Stephane de Labrusse
Details | Diff
smeserver-yum-2.6.0.bz8834.DeltaRpm.patch (5.31 KB, patch)
2016-07-31 21:56 CEST, Stephane de Labrusse
Details | Diff
smeserver-yum-2.6.0.bz8834.DeltaRpm.patch (5.08 KB, patch)
2016-08-01 16:24 CEST, Stephane de Labrusse
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description RequestedDeletion 2015-02-05 15:52:58 CET
As per title. yum-presto is available from upstream base repo.

Description : Yum-presto is a plugin for yum that looks for deltarpms rather
            : than rpms whenever they are available.  This has the potential of
            : saving a lot of bandwidth when downloading updates.
            : 
            : A Deltarpm is the difference between two rpms.  If you already
            : have foo-1.0 installed and foo-1.1 is available, yum-presto will
            : download the deltarpm for foo-1.0 => 1.1 rather than the full
            : foo-1.1 rpm, and then build the full foo-1.1 package from your
            : installed foo-1.0 and the downloaded deltarpm.
Comment 24 Terry Fage 2017-06-14 11:21:06 CEST
Heads up

This produces an error
[root@sme10a3vm ~]# config setprop yum DeltaRpmProcess disable
[root@sme10a3vm ~]# config show yum

Correct syntax, disabled NOT disable

[root@sme10a3vm ~]# config setprop yum DeltaRpmProcess disabled
[root@sme10a3vm ~]# config show yum
Comment 23 RequestedDeletion 2017-06-05 00:12:13 CEST
Released with Koozali SME Server 10alpha3.
Comment 22 Terry Fage 2017-06-03 08:37:52 CEST
VERIFICATION

= Environment: 
SME 10.0alpha3 x64 clean install updated as at 03 Jun 17		

= ORIGINAL PROBLEM:
Bug 8834 - install deltarpm to reduce updates sizes

= RESOLUTION:
Fixed in smeserver-yum-2.6.0-16.el7.sme

* Tue Mar 07 2017 Jean-Philipe Pialasse <tests@pialasse.com> 0.96-18.sme
- Removed Message-Id validation, as it rejects MS account validation email [SME: 10139]

= CURRENT VERSION INSTALLED:
[root@sme10a3vm ~]# cat /etc/e-smith-release
SME Server release 10.0alpha3

[root@sme10a2base ~]# uname -r
3.10.0-514.6.2.el7.x86_64

[root@sme10a3vm ~]# rpm -qa smeserver-yum
smeserver-yum-2.6.0-16.el7.sme.noarch

= TESTING:
Released in sme10a3 and as part of other bug verifications rpm ver now at 

[root@sme10a3vm ~]# rpm -qa smeserver-yum
smeserver-yum-2.6.0-16.el7.sme.noarch

No errors observed or detected in Logs

= UPDATED VERSION INSTALLED:
[root@sme10a3vm ~]# rpm -qa smeserver-yum
smeserver-yum-2.6.0-16.el7.sme.noarch

[root@sme10a3vm ~]# rpm -qa smeserver-yum
smeserver-yum-2.6.0-16.el7.sme.noarch

Can be set enabled/disabled via config property

[root@sme10dev6 ~]# config setprop yum DeltaRpmProcess disabled 
[root@sme10dev6 ~]# signal-event yum-modify 

Option to enable is also now available from the server-manager Software Installer form

Default setting

[root@sme10a3vm ~]# config show yum
yum=service
    AutoInstallUpdates=disabled
    CheckContribs=enabled
    DeltaRpmProcess=disabled
    DownloadOnly=disabled
    EnableGroups=no
    GPGCheck=0
    PackageFunctions=disabled
    RandomDelay=120
    check4updates=daily
    status=enabled

Enable Delta RPM via Server-Manager

[root@sme10a3vm ~]# config show yum
yum=service
    AutoInstallUpdates=disabled
    CheckContribs=enabled
    DeltaRpmProcess=enabled
    DownloadOnly=enabled
    EnableGroups=no
    GPGCheck=0
    PackageFunctions=disabled
    RandomDelay=120
    check4updates=daily
    status=enabled

Disable

[root@sme10a3vm ~]# config show yum
yum=service
    AutoInstallUpdates=disabled
    CheckContribs=enabled
    DeltaRpmProcess=disabled
    DownloadOnly=disabled
    EnableGroups=no
    GPGCheck=0
    PackageFunctions=disabled
    RandomDelay=120
    check4updates=daily
    status=enabled

Enable via config property

[root@sme10a3vm ~]# config setprop yum DeltaRpmProcess enabled
[root@sme10a3vm ~]# config show yum
yum=service
    AutoInstallUpdates=disabled
    CheckContribs=enabled
    DeltaRpmProcess=enabled
    DownloadOnly=disabled
    EnableGroups=no
    GPGCheck=0
    PackageFunctions=disabled
    RandomDelay=120
    check4updates=daily
    status=enabled

[root@sme10a3vm ~]# signal-event yum-modify
[root@sme10a3vm ~]# yum update
Loaded plugins: fastestmirror, smeserver
Loading mirror speeds from cached hostfile
 * base: centos.mirror.digitalpacific.com.au
 * smeaddons: ibsgaarden.dk
 * smeextras: ibsgaarden.dk
 * smeos: ibsgaarden.dk
 * smeupdates: ibsgaarden.dk
 * updates: centos.mirror.digitalpacific.com.au
No packages marked for update

Disable

[root@sme10a3vm ~]# config setprop yum DeltaRpmProcess disable
[root@sme10a3vm ~]# config show yum
yum=service
    AutoInstallUpdates=disabled
    CheckContribs=enabled
    DeltaRpmProcess=disable
    DownloadOnly=disabled
    EnableGroups=no
    GPGCheck=0
    PackageFunctions=disabled
    RandomDelay=120
    check4updates=daily
    status=enabled

[root@sme10a3vm ~]# signal-event yum-modify
[root@sme10a3vm ~]# yum update
Loaded plugins: fastestmirror, smeserver
Loading mirror speeds from cached hostfile
 * base: centos.mirror.digitalpacific.com.au
 * smeaddons: ibsgaarden.dk
 * smeextras: ibsgaarden.dk
 * smeos: ibsgaarden.dk
 * smeupdates: ibsgaarden.dk
 * updates: centos.mirror.digitalpacific.com.au
No packages marked for update

Can verify that it appears the changes incorporated in this bug fix have resolved this bugs issue.

No errors observed or detected in Logs associated with this bug.

Marking verified as released in sme10 alpha3

= PROBLEM FIXED:
Install deltarpm to reduce updates sizes 

= VERIFIED OR REOPEN:
Verified

= DOCUMENTATION IMPACT:
https://wiki.contribs.org/DB_Variables_Configuration#yum
should be expanded here re may actually cause a slower update
and
info added to 
https://wiki.contribs.org/SME_Server:Documentation:Administration_Manual:Chapter13#Software_Installer_Panel

= SUGGESTED RELEASE NOTES:
Fixed in smeserver-yum-2.6.0-16.el7.sme

* Tue Mar 07 2017 Jean-Philipe Pialasse <tests@pialasse.com> 0.96-18.sme
- Removed Message-Id validation, as it rejects MS account validation email [SME: 10139]
Comment 21 Jean-Philippe Pialasse 2017-03-17 15:26:10 CET
move to sme10
Comment 20 RequestedDeletion 2016-08-02 22:50:45 CEST
(In reply to Stefano Zamboni from comment #19)
> the only package from our repo that would benefit from delta is clamd
> 
> all the others are quite small

That completely depends on where you are on the globe, costs per Mb, bandwidth, speed and limits....

Ask Terry for example... or someone in Russia or Somalia....
Comment 19 Stefano Zamboni 2016-08-02 18:53:11 CEST
the only package from our repo that would benefit from delta is clamd

all the others are quite small
Comment 18 Charlie Brady 2016-08-02 18:43:18 CEST
(In reply to Stephane de Labrusse from comment #14)
> Created attachment 5666 [details]
> smeserver-yum-2.6.0.bz8834.DeltaRpm.patch
> 
> If no objections I want to push this to smeserver-yum

I have no objections.

We should also be having a discussion about the creation of .drpm files in our repositories.
Comment 17 Stephane de Labrusse 2016-08-01 16:24:00 CEST
Created attachment 5667 [details]
smeserver-yum-2.6.0.bz8834.DeltaRpm.patch

patch pushed to cvs
Comment 16 Stephane de Labrusse 2016-08-01 16:23:00 CEST
/usr/bin/plague-client build smeserver-yum smeserver-yum-2_6_0-6_el7_sme sme10
Package smeserver-yum enqueued.  Job ID: 1405.

* Mon Aug 1 2016 stephane de Labrusse <stephdl@de-labrusse.fr> 2.6.0-6.sme
- Deltarpm is now a setting in the yum panel (disabled by default)
- Added smeserver-yum-2.6.0.bz8834.DeltaRpm.patch [SME: 8834]


delta is just a simple option (enabled/disabled) and disabled by default

set resolved fixed
Comment 15 Daniel B. 2016-07-31 22:09:25 CEST
I don't think this setting deserves to be exposed in the server manager
Comment 14 Stephane de Labrusse 2016-07-31 21:56:17 CEST
Created attachment 5666 [details]
smeserver-yum-2.6.0.bz8834.DeltaRpm.patch

If no objections I want to push this to smeserver-yum

deltatrpm is disabled by default

option available (number of process) :automatic,2,4,6,disabled

an option exists in the manager now with translations
Comment 13 Jean-Philippe Pialasse 2016-07-27 05:15:03 CEST
Three things:

Try to avoid to mix type for one property, keep it simple
1- Either this is boolean type 1/0 or enabled /disabled, either this is a int, either this is a string

2- by default it should be disabled, as with good adsl connection it will in fact slow the update process, high the cpu usage, and heat the server. We tend to develop for low power server, sompeople reusing old hardware will see a significant slowering of their server while updating without any benefit if they have a correct connection or a worsening. 

3 you are also workin on pre download which mean this one is only useful for people with limited monthly bandwidth , ehich again limitbthe public and hence it should be disabled by default. 


Then you should 
Use DeltaRpm as enable/disabled with disabled by default
Use DeltaRpmProcess as a number of process defaulting to 2. 


This reduce the leaning curve for cli users, reduce the complexity of verification we will need for. A new interface. Kiss one property one goal. 

Keep the good work.
Comment 12 Stephane de Labrusse 2016-07-24 23:07:05 CEST
Created attachment 5636 [details]
smeserver-yum-2.6.0.bz8834.DeltaRpm.patch

removed some jump of lines
Comment 11 Stephane de Labrusse 2016-07-24 23:04:34 CEST
Created attachment 5635 [details]
smeserver-yum-2.6.0.bz8834.DeltaRpm.patch

a patch to add delta-rpm

enabled by default to 2 parallel processes which is the default setting.

you can disabled globally

[root@sme10dev6 ~]# config setprop yum DeltaRpmProcess disabled 
[root@sme10dev6 ~]# signal-event yum-modify 

you can set the value to '-1' and let the deltarpm find the good value to your system

you can also disabled it per repository by 

[root@sme10dev6 ~]# db yum_repositories setprop sogo DeltaRpmPercentage 0
[root@sme10dev6 ~]# signal-event yum-modify 

or set a value for not using deltarpm if the delta is higher than a certain percentage (default is 75%)

[root@sme10dev6 ~]# db yum_repositories setprop sogo DeltaRpmPercentage 50
[root@sme10dev6 ~]# signal-event yum-modify 

I know that opinions are divided on this topic but it works like a charm on my laptop since the long years.

nothing committed yet ;)
Comment 10 Daniel B. 2016-06-17 20:13:22 CEST
No it's not harmless. It costs cpu and storage space in order to reduce downloads size. The tradeoff isn't always beneficial. I can't say if it should be enabled by default or not
Comment 9 Stefano Zamboni 2016-06-17 19:21:57 CEST
(In reply to Daniel B. from comment #8)
> Note that presto / delta-rpms isn't always beneficial. If you have a fast
> and un-metered connection, presto will make things slower. Especially if you
> have a low-end CPU. That being said, I don't know globally if there's more
> install where it would be good than installs where it'd be bad. In any case,
> it should be easy to disable

it's harmless, it comes from upstream, it can be useful where bandwidth is a problem (just think about italy, you don't have to go in Africa ;-))

please add it and enable it by default

thank you

P.S. on some server centos6 where I use it, I see that it is very useful with clamav.. could we consider the idea to have deltarpms for our packages?
Comment 8 Daniel B. 2016-06-17 18:37:24 CEST
Note that presto / delta-rpms isn't always beneficial. If you have a fast and un-metered connection, presto will make things slower. Especially if you have a low-end CPU. That being said, I don't know globally if there's more install where it would be good than installs where it'd be bad. In any case, it should be easy to disable
Comment 7 Jean-Philippe Pialasse 2016-06-17 18:30:56 CEST
(In reply to Daniel B. from comment #6)
> upstream repo provides delta-rpms, but not our own repo. That's not a big
> issue as most of our package are quite small and wouldn't benefit from
> delta-rpm (with a few exceptions like clamav)

hence a good pick for SME10  even if we do not provide ourself deltarpm yet?

also would it be compatible with : yum downloader bug # 1502 ?
Comment 6 Daniel B. 2016-06-17 18:28:19 CEST
upstream repo provides delta-rpms, but not our own repo. That's not a big issue as most of our package are quite small and wouldn't benefit from delta-rpm (with a few exceptions like clamav)
Comment 5 Stefano Zamboni 2016-06-17 18:00:03 CEST
(In reply to Jean-Philippe Pialasse aka Unnilennium from comment #3)
> are deltarpm availables ?
> if no should we simply close this ?

I'm using it since epoch on SME9
Comment 4 Jean-Philippe Pialasse 2016-06-17 17:57:37 CEST
*** Bug 2878 has been marked as a duplicate of this bug. ***
Comment 3 Jean-Philippe Pialasse 2016-06-17 17:56:11 CEST
are deltarpm availables ?
if no should we simply close this ?
Comment 2 Stefano Zamboni 2015-09-24 13:08:55 CEST
I saw now it has been discussed in #2878 too

in any case, we could just close this bug and add relevant info to our wiki
Comment 1 RequestedDeletion 2015-02-05 17:07:00 CET
and dependency deltarpm