[Mono-dev] Mono repo breaks on some azure VMs but not others - CDN issue?

[David Molnar]

Hi everyone,

First, thank you for all your hard work on the Mono project. We use F# as our primary development language. Mono helps us rapidly develop for Linux as well as Windows while keeping the benefits of using F#.

Unfortunately we are seeing a problem with the CentOS repository at download.mono-project.com that we believe is related to a CDN issue. Please let me know if this is not the right place to ask about this, I would appreciate any pointers. The issue is breaking Linux deployments of our Azure hosted service by causing yum install failures of Mono on our Linux VMs. We would be grateful for a fix or workaround and we are happy to help with further troubleshooting.

Details:

We deploy Mono on Red Hat Enterprise Linux 7.2 VMs hosted on Microsoft Azure. We see yum fails to install package sqlite from the CentOS repo on download.mono-project.com from some VMs but not other VMs hosted in Azure. The issue is breaking our Linux deployments of our service because we have set yum to install Mono on startup for our Linux VMs.

We see that yum is attempting to download this URL
http://download.mono-project.com/repo/centos/repodata/d839a3db898c8bd7e9a08097357b3568ffc7b128e846b3943d1d61a94ecc2bc6-primary.sqlite.bz2

The URL is specified in http://download.mono-project.com/repo/centos/repodata/repomd.xml , which we download directly from the web site, has a last modified date of 12 April 2017. The file we have shows revision 1491476506 . Is this the most recent version of repomd.xml?

The URL gives a 404 on some of our Azure-hosted VMs and not others. We can reliably reproduce the 404 error using wget on failing machines 100% of the time.

When we examine http://download.mono-project.com/repo/centos/repodata/  from a web browser we see that the sqlite package is at this URL, updated 12 April 2017:
http://download.mono-project.com/repo/centos/repodata/4e94d255551bd1fa2ce69f946cf41519aed3b645206aa23e1aa2d06f349763ae-primary.sqlite.bz2
There is no "d839..." URL in the repodata directory.

On all our machines that successfully download the "d839..." URL, we observe from wget a reference to a CDN cache hit and a "last modified" date of Thursday 6 April 2017. Output from wget appended to this message.

Because of the CDN reference and the CDN hitting only on machines that succeed in downloading the "d839..." URL, we suspect that there is a CDN which is caching an old version of the primary.sqlite.bz2 package. Because the "d839..." URL shows upin repomd.xml, we also suspect that either repomd.xml is being cached with the previous URL or that repomd.xml has not been updated properly with a new URL for a new version of the package.

Does this error report make sense? Would it be possible to please check repomd.xml and the propagation of the repo data to relevant CDNs? If this issue is hitting us, I expect it may also break others using CentOS / RHEL as well.

Again, if this is not the best place to report, I would appreciate any pointers on where to follow up. Thanks again for all your hard work on Mono!

David Molnar


WGET output showing CDN hit:

wget http://download.mono-project.com/repo/centos/repodata/d839a3db898c8bd7e9a08097357b3568ffc7b128e846
b3943d1d61a94ecc2bc6-primary.sqlite.bz2
StatusCode        : 200
StatusDescription : OK
Content           : {66, 90, 104, 57...}
RawContent        : HTTP/1.1 200 OK
                    X-Cache: HIT
                    Accept-Ranges: bytes
                    Content-Length: 555139
                    Content-Type: application/x-bzip2
                    Date: Mon, 17 Apr 2017 06:11:49 GMT
                    ETag: "87883-54c7d707d2df2"
                    Last-Modified: Thu, 06...
Headers           : {[X-Cache, HIT], [Accept-Ranges, bytes], [Content-Length, 555139], [Content-Type,
                    application/x-bzip2]...}
RawContentLength  : 555139





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20170417/875a4994/attachment.html>