Damaged pages, the second

Use this forum for HELP at Choral Public Domain Library as well as FEEDBACK
Locked
Nikolaus Hold
Posts: 50
Joined: 13 Jul 2016 10:47

Damaged pages, the second

Post by Nikolaus Hold »

Many errors in pages caused by a script from March 2021.
A search within the CPDL for
{{Descr|
brings many hits. Existing text is no longer displayed. Partly similar errors with external links, as as already mentioned in my error note "External websites - Damaged pages".

See e.g. page https://www.cpdl.org/wiki/index.php/Catechismus
Publication date and place: 1625 Würzburg: Volmar.
{{Descr|22 Georg Vogler works
Facsimile: Bayerische StaatsBibliothek
==List of Works==}}
or Musical concerto (Bartolomeo Spighi da Prato)
First Publication date and place: 1641 Florence: Zanobi Pignoni.
{{Descr|Bartolomeo Spighi da Prato
Facsimile:
==List of Works==}}
or Two Christmas anthems, Op. 32 (Dudley Buck)
First published:
, City: Publisher. Description:
External Links:
or Abendlied (Friedrich Kuhlau)
15th published: 1906 in Volksliederbuch für Männerchor, no. 173
16th published: 1915 in Volksliederbuch für gemischten Chor, no. 179
Description:
External websites:
(Description exists, but is not displayed.)
CHGiffen
Site Admin
Posts: 1749
Joined: 16 Sep 2005 21:22
Location: Hudson, Wisconsin, USA

Re: Damaged pages, the second

Post by CHGiffen »

As with {{#ExtWeb: ... }}, {{Descr| ... }} has caused some bungled effects, stemming from the mass conversion to these templates, using ReplaceText. Earlier, I fixed a dozen or so pages from {{#ExtWeb: }} out of well over a thousand pages, since the replacement scripts were ones that I had run. I just fixed the {{Descr| }} item in this thread, and a similar search will be necessary. I didn't run the replacment scripts that caused the present problem, but I will fix the ones already pointed out. It is unfortunate, but apparently collateral that crops up when a massive restructuring of the format of works and publications pages (in an effort to put various kinds of information into templates for more efficient data handling & orgainisation). The actual "damage" is quite small as a fraction of the tens of thousands of pages that have been converted with these two (and other) template replacements. If a user encounters such a page and can fix it him/herself, that would be most helpful.

Thanks for pointing out this second instance of bungled effects. I (and hopefully others) will try to examine a large store of pages on which the errant replacement script was run, making fixes.
Charles H. Giffen
CPDL Board of Directors Chair
Admin at & Manager of ChoralWiki
CHGiffen
Site Admin
Posts: 1749
Joined: 16 Sep 2005 21:22
Location: Hudson, Wisconsin, USA

Re: Damaged pages, the second

Post by CHGiffen »

Incidentally, the culprit in the broken {{Descr| }} pages cited seems to be due to the inclusion of external links and headers (such as "==List of works==") that have characters (most notably, the "=" character) that break the template. {{#ExtWeb: }} must be used to list external links.
Charles H. Giffen
CPDL Board of Directors Chair
Admin at & Manager of ChoralWiki
Nikolaus Hold
Posts: 50
Joined: 13 Jul 2016 10:47

Re: Damaged pages, the second

Post by Nikolaus Hold »

Thanks for the answers and for taking care of the problems I described.
If a user encounters such a page and can fix it him/herself, that would be most helpful.
I believe that the person who makes a change is responsible for it. You can't just offload troubleshooting onto other users, especially when existing information has simply disappeared without a trace. Or should normal users be encouraged to always check the source code first to see if the web page in question shows what the creator intended?

I tried to understand what the problem is in the Description of Abendlied (Friedrich Kuhlau) and I read the documentation for the template Descr. Example 2 shows that the developers were well aware that existing information can disappear into nirvana. Nevertheless, it was decided to use the template in 'mass conversion'. The decision makers must have been very sure about the benefit the change would bring to CPDL users. And for 9 months afterwards, they didn't feel compelled to care about the impact.

Basically, this amounts to clandestinely setting new standards for how CPDL pages are to be built. e.g.
{{#ExtWeb: }} must be used to list external links.
I hope that the function for "add new pages" has appropriate checks built in to see if these are being adhered to. Regarding the existing pages, the developers should either try to find upward compatible solutions, or look for the problem areas themselves in advance and fix them.

I could check with James Gibb,
  • if he regularly checks if all his pages comply with the just invented standards
  • or if he even knew (unlike me) about such new standards
  • or if he can remember which of his pages could have such problems.
I've been in contact with him before, but I'm afraid he wouldn't take such questions seriously ... ;-)
choralia
Site Admin
Posts: 2848
Joined: 05 Mar 2006 19:57
Location: Rome, Italy
Contact:

Re: Damaged pages, the second

Post by choralia »

On a collaborative website all users are supposed to collaborate to maintenance and improvements. If you like, Nikolaus, I can implement improvements according to your specifications. Please let me know if you wish to receive a quotation. Otherwise, you have to accept what other users do for free, and possibly implement any desired improvements yourself.

Max
Nikolaus Hold
Posts: 50
Joined: 13 Jul 2016 10:47

Re: Damaged pages, the second

Post by Nikolaus Hold »

We agree that "what users do for free" is highly commendable - after all, I am one of the volunteers myself, albeit in a much less demanding role than you. The most important features of CPDL for me are:

1. the large offer of about 40,000 scores
2. the permanent online availability and
3. the free access and download possibility.

Everything else, the best templates etc. pales in comparison. And not everything that happens in this area is seen as an improvement by users who are not familiar with the intricacies of MediaWiki. Should the source code in the future be allowed to consist only of templates and parser thingies?
I searched in vain for documentation of the #ExtWeb construct so far.

But maybe you should see my post less as criticism and more as feedback. And that's what this forum is for, isn't it?
choralia
Site Admin
Posts: 2848
Joined: 05 Mar 2006 19:57
Location: Rome, Italy
Contact:

Re: Damaged pages, the second

Post by choralia »

Nikolaus Hold wrote: 24 Oct 2021 16:05 Should the source code in the future be allowed to consist only of templates and parser thingies?
Yes, this is what is being gradually implemented, so that:
  • all pages have a consistent structure;
  • fields contents are clearly delimited so that they can be more reliably captured by the "spider" that puts page contents into a database used for maintenance purposes; and
  • the rendering of all work pages can be centrally governed in such a way that, if we want to change the way pages are displayed while preserving the information inside, changes made on templates (or parser functions) are immediately applied to all pages.
Nikolaus Hold wrote: 24 Oct 2021 16:05 I searched in vain for documentation of the #ExtWeb construct so far.
This parser function was introduced few months ago, and not documented yet. I've just added some documentation here.

Max
BarryJ
Posts: 161
Joined: 03 May 2014 03:33
Location: Gunnison, Colorado, USA

Re: Damaged pages, the second

Post by BarryJ »

I'm afraid I was the one responsible for most of the changes to the Descr template in March. Between March 14 and March 22, I replaced '''Description:''' with {{Descr}} in about 40,000 pages. It was relatively easy at first, since the template is empty most of the time. After a while, it got complicated, since there are many ways editors have described works. In total, I ran ReplaceText about 180 times, with various different Regexp scripts (at most 250 pages at a time; each 250-page group takes 3-6 minutes). Each new script was tested on a few pages before it was used across a larger number of pages. Then at the end there were about 500 pages that had to be changed manually one by one. Almost all of the changes were restricted to work pages, but a few got applied to publication pages. I tried to be as careful as I could, but apparently I missed some errors.
I will try to chase down the remainder of errors that need to be corrected; but it would help me if I could have a few examples of the remaining issues anyone finds. You could post a reply here, on my talk page, or to the email on my user page.
I am also responsible for changing '''Edition notes:''' to the {{EdNotes}} template on many pages, and I was involved in converting '''External websites''' (etc.) to the {{#ExtWeb:}} parser function. If you find any of those, send them to me too.
Thanks for your patience.
Barry Johnston
Gunnison, U. S. A.
BarryJ
Posts: 161
Joined: 03 May 2014 03:33
Location: Gunnison, Colorado, USA

Re: Damaged pages, the second

Post by BarryJ »

I have finished going through the list Claude sent (thanks, Claude!). I found some more pages by searching; in total I examined 85 pages, 29 work pages and 56 publication pages. I made major corrections to 42 of the 85. I think I found most of the errors; if you find more, please contact me.
Barry Johnston
Gunnison, U. S. A.
Locked
https://forums1.cpdl.org/phpBB3/viewtopic.php?f=1&t=13322&p=29406&sid=24cb62cdb9eba7171be21854fab0c4b7
forums1.cpdl.org
crawl=CC-MAIN-2021-49
49137
application/http; msgtype=response
sha1:4AEQLNPNMTOPPGI5JC5QPTEQ65YVWHSO
<urn:uuid:c454c186-561f-42d7-a2a9-ab72407f0b58>
2021-11-28T23:42:44Z
190.92.152.174
text/html
sha1:4VUK6BGKRY6MCJPJTKCJSXA5NLQ5VD6H
<urn:uuid:6e0e25df-da04-4b66-95b4-c2f2aac5d721>
https://forums1.cpdl.org/phpBB3/viewtopic.php?f=1&t=13322&p=29406&sid=24cb62cdb9eba7171be21854fab0c4b7
response
<urn:uuid:4da7e8cc-db40-4cc6-b841-cd233244b2d9>
isPartOf: CC-MAIN-2021-49
publisher: Common Crawl
description: Wide crawl of the web for November/December 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-102
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/