Finding and playing public domain sheet-music

Here are some introductory notes on a preliminary search, relating to some new software I was testing and installing as a favour.

* IMSLP/Petrucci Music Library. Free Public Domain Sheet Music, with 140,000 works at May 2019. No preview, and a 20 second delay on each PDF download while an appeal for donations is displayed. It seems they’re not closely tracking Archive.org, as I couldn’t find a couple of newer public domain items I know are on Archive.org.

* Musopen is the other large repository of free sheet music. You get a visual preview of the sheets, and PDF download links are open and quick. But the results from its keyword search is very bad. You’ll do better with a site: search in Google or DuckDuckGo…

site:https://musopen.org/music/ keywords

* Metadata tagging on Archive.org is patchy for sheet music. Thought it will often have ‘sheet music’ or ‘sheetmusic’ or ‘score’ in its title. Theoretically it seems it should have a ‘sheet music’ topic tag, but often the uploaders don’t tag their upload with that. The best initial wide searches are:

sheetmusic AND mediatype:texts keywords

“sheet music” AND mediatype:texts keywords

Which together suggest Archive.org currently has about 10,000 sheet music booklets, nearly all of an age to be in the public domain. Including much 1910s and 20s rag-time, marching music, and popular ditties.

* ChoralWiki has a very large collection of public domain choral sheet music.

* The Mutopia Project has a small collection of classical works.

* Band Music a large collection of American marching-band sheet music.

* American university libraries also have very large online collections, usually general popular and light classical music.

I don’t know of any unified one-box search that will search across all of the above. It might be an interesting project to create one, but I won’t be doing that — so feel free to give it a go.

---

* For audio preview of the music score, you’ll first need to get away from your default MIDI sounds driver. Doing this used to be fiendishly complex, but is now extremely easy with the fine bit of Windows freeware called VirtualMIDISynth. Install, and it takes over from your Windows MIDI sounds driver, and then you add a free .SF2 soundfont to it — so that it has the instruments it needs to play a score with.

Then you need to OCR your sheet music into software player-readable form. For sheet music OCR is termed “OMR”. If you’re using the leading paid Sibelius sheet-music player and writer (termed ‘scorewriters’ by those in the trade), then you’ll find that works seamlessly with the leading PhotoScore “OMR” software. PhotoScore uses the SharpEye SDK, widely said by eagle-eyed music industry testers to be the most accurate at picking out and interpreting all the complex fine detail of a music score.

Then you save your successful scan to an .OPT file and Sibelius can load and play this as if it were a .MP3 file. Your new VirtualMIDISynth audio driver and soundfont makes it sound lovely.

---

I’d like to recommend some free open source OMR -> Scorewriter software, but I had no success with it and the OMR end of the process is especially lacking and clunky (Java). As someone who’s been away from the world of music for many years, I had the feeling that the robust commercial impetus in the music professions means that it’s hard for open source software makers to stand the pace.

Force Windows to open .txt files with 32-bit Notepad++

Situation: When Windows opens a .txt file it will only launch the 64-bit version of Notepad++. How does the user force Windows to open .txt files with the 32-bit Windows, when both the 32-bit and 64-bit Notepad++ are installed? The usual routes of Windows file association and using the Notepad++ Preferences settings have both failed.

Reason for wanting 32-bit: The user may prefer the older version, for light editing, for the time being. Because it supports useful plugins such as MultiClip Viewer and others.

Solution: Open the Windows Registry editor (Start: ‘Regedit’) and navigate down to…

HKEY_CURRENT_USER/software/classes/applications/notepad++.exe/shell/open/command

There replace the 64-bit file path with the 32-bit path. Which means, for me…

C:\Program Files\Notepad++\notepad++.exe

to

C:\Program Files (x86)\Notepad++\notepad++.exe

… and then exit the Registry Editor.

Some useful regex commands for Notepad++

Perhaps it’s just the influence of Inkle’s new Heaven’s Vault game, with the epigraphy of its mysterious alien inscriptions, but I’ve taken a bit of a shine to regex. My first failed tests with the Notepad++ regex were obviously with the ‘wrong type’ of regex, as I now know there are slightly different versions for Windows, Linux etc. But I’ve now found commands that do work for me.

The following were found by scouring forums and were then tested while learning more about Notepad++ and how it works (it’s a lot deeper than it looks). They’re actual working practical examples, tested and working with the latest Notepad++ on Windows 64-bit. My testing suggests that exactly the same macros run differently in the old 32-bit vs. new 64-bit Notepad++. Since (so far as I can tell) plugin activity cannot be recorded in macros, I assume the difference is due to regex support.

Please note that I am clueless about writing these things, only knowing how to search for, find and test them. So don’t ask me to advise you on their devising or tweaking. My many thanks especially to guy038 at the Notepad++ forum, and many and various others, for writing these and helping others find solutions. I found that the search-engine Yippy, based on Bing, is especially good at finding these things, and will almost inevitably lead you to guy038. But, so far as I know, he has not made a regex ‘keyring’ or a ‘cookbook’ or suchlike. Hence my need to collect some working examples here under practical headings.

All but one of these regexes (regexii?) run in the ‘Find’ or ‘Replace’ box in Notepad++. One needs to run in the ‘Mark’ tab in the same box…

On your keyboard, it’s useful to know that Ctrl + Home will take your text cursor (‘caret’) back to the top of the Notepad++ page, which may be useful if you are building these commands into a recorded macro.

Lastly, using regex to fiddle with public HTML seems to be frowned on, so I suggest the following are useful for certain offline text cleaning and data-swivelling operations, not for mission-critical coding or live pages.

---

The list:

The list has to be posted here in plain text as a .PDF. I had a blog post all done and polished, but then found that WordPress.com blogs make an utter mess of posted regex code, even when wrapping it with the code tags which are supposed to protect snippets of code! So here are the working regexes in a handy four-page .PDF file…

regex.PDF

PanWriter – a free open source Markdown editor and HTML-Markdown converter

Pandoc is a useful universal document converter utility. Yet using the free Pandoc on Windows is often assumed to be about typing arcane command lines into a command prompt, and hoping for the best.

But now there’s a nice free and open source front-end for Windows, for those who just want the markdown. The 2019 PanWriter is a delightfully elegant and simple markdown writing pad for the rest of us.¹ It’s a speedy Windows desktop install, and it didn’t even need to be told where Pandoc was on my PC. It just ‘knew’.

Even if you don’t need another sweet text editor, PanWriter also serves as a document importer and converter utility. Just load up a HTML page you saved from the Web, and it instantly converts all the HTML code to markdown. I loaded a few really complex pages and it didn’t blink, instantly presenting a clean markdown conversion.

Why would one want to convert HTML to markdown, you ask? Because it places the HTML and other elements onto single lines, while retaining Web links in place. Such lines are far easier to extract data blocks from. Compared to fiendishly nested HTML code that sprawls across multiple lines.³

Picture: the above web page (just some random Met Museum image search results) in Notepad++ as markdown.

Once the markdown text is in Notepad++, then a macro can have its way with it. This can ‘Find and Mark’ the repeating line-blocks² containing the data you want to extract and clean (e.g. search-engine results). These can then be copied out to a new tab (Top Menu: Search | Bookmarks | Copy Bookmarked lines). Then the macro can run various operations that fix up the text a bit more,^4. before finally saving the cleaned list back to HTML via the user-friendly MarkdownViewerPlusPlus plugin for Notepad++.⁵

So the basic workflow here is:

1. Get your search results and save your Web page(s) as usual. There’s no need to painstakingly select-copy-paste just part of the page.

2. Use a joiner utility (TXTcollector) to join all the saved pages.⁶ Open the saved HTML with PanWriter and it instantly auto-converts into markdown.

3. Open the saved markdown with Notepad++ and run your cleaning and text-sorting custom macro on it.

4. Copy-paste the resulting linked data list as HTML to your blog etc. (Or to .CSV for Excel import and sorting).

The above is a more advanced and robust version of my recent home-brew workflow, which suggested a browser addon and manual copy-paste. That was more suitable for occasional use by bloggers and academics who can’t afford sophisticated data scrapers (and the proxies to run them).

This workflow has the advantage that: i) it’s all free software; ii) it doesn’t need you to pay for and burn through proxies in your paid scraping software; iii) as long as you have the HTML in your browser it can be grabbed, and it basically doesn’t matter how complex and nested the page code is, as it’s all going into Markdown; and iv) collection can be automated with Windows automation software (JitBit) etc, and processing can be automated with Notepad++ macros. But it is obviously not suited to automated scraping of millions of records from multiple shopping sites — if you’re into that game then you should have the cash to buy in those datasets.

---

---

Notes:

1. There are also two old GUIs for Pandoc, over on GitHub. One there is nice and simple, and has batch… but it crashes and fails on 64-bit Windows, as the developer admits in his readme. The other GUI at GitHub was tried and runs on 64-bit Windows, but seems far less user-friendly. There are also a half dozen Python scripts projects that do this.

2. Marking and exporting lines in Notepad++ can’t currently be done for multi-line nested HTML code, which is why a HTML-to-Markdown conversion is so useful. While multi-line block marking can be done between two keywords [ Find | Mark | Regex with Newline | then paste in…

(?<=STARTWORD)([\s\S]*?)(?=ENDWORD)

…this only places a single mark at the top of each marked and highlighted block. It does not run a line of marks down the entire block.

3. Yes, I know about XPath, but with a complex Web page it's: i) fiendishly tricky to do the initial puzzling out of what needs to be captured; ii) often fails to then grab what’s needed; iii) and has even more difficulty in aligning data fields when used as a browser addon.

4. Note that multiline search-replace needs to be done as \n commands not plugins, in macros. Also that Crtl + Home will get your cursor back to the top of the text.

5. Sadly one can’t yet use Notepad++ as the initial importer/converter, as it has no such plugin at present. I’ve looked. There are a couple of possible Python scripts but support for the Python plugin in the latest Notepad++ is a bit of a mess at present, with plugin structures being swopped around and then reverted. So that’s not really an option, unless you want to fall right back to version 5.9 or thereabouts to use a script.

6. Update: If you have no absolute need to keep the saved HTML pages as backup, then Clipboard Magic is lovely little Windows freeware that keeps copies of each clipboard, then when done you “Copy all clips to clipboard” and paste to Notepad++. Or if you still use an older 32-bit Notepad++ you can use the fine MultiClipboard plugin.

Freeware for cleaning and manipulation of text lists

These are all Windows PC freeware, with graphical user interfaces, tested and working on Windows 8.1.x. They may be useful for those who occasionally have to sort and clean and combine lists in text form, and who do not have access to paid tools such as the sophisticated TextPipe Pro or the Sobolsoft utilities, or to advanced training in Excel and regex commands.

---

---

The relatively simple:

1. Text Cleanup 2.0.

It “fixes” text automatically when you copy-paste it, according to various cleaning options you can save presets for. Its main use is to unwrap a chunk of text that has hard line-breaks, when copied to the clipboard. Or to place a new blank line between each line. This vital software only recently went freeware.

Can be used in combination with the free Clipboard Magic which keeps copies of all Clipboard text, and then allows you to “Copy all clips to clipboard”.

---

2. List Numberer.

This can do what Notepad++ can’t yet do, and does easily what Excel can only seem to with complex fiddly formulas and macros. Most useful for dealing with repeated blocks of data in a list (e.g. labelling them 1234, 1234, 1234), to enable mass deletion of certain lines in a text editor.

---

3. Text Magician 1.3.

Various operations including append text to the start and end of each line, delete multi-line blocks between X and Y, and more. (If you have ‘.DLL missing’ problems, either go find the required .DLL file, or use Version 1.0 which does not have that problem).

---

4. Duplicated Finder from AKS-Labs.

Easily find and extract the duplicates from a single list. Useful for checking for the presence of the a few duplicate URLs in long list or uniques, for instance. (See also the free Duplicate Master addin for Excel).

---

5. Excel example sheet: compare two lists and extract non-duplicates.

My free ready-made .XLS sheet for Excel, with formula. The second list is a jumbled up variant of the first, with some new additions in it. These additions are extracted and placed alongside. (Excel is not free, admittedly, but my guess is you could probably get the same formula working in whatever LibreOffice has as its Excel equivalent).

---

---

The potentially quite complex:

1. Notepad++.

The code programmer’s text editor. Column numbering (though it can’t do what List Numberer — see above — can do); sophisticated Regex (though the more sophisticated, the more difficult to remember and to get it working); Remove blanks lines (provided you can remember the menu sequence within its complex UI); and much more. Intensive research is often needed to learn how to do a particular bit of sophisticated text manipulation, and it’s also easy to overlook its most powerful features such as per-line list bookmarking. The devs have recently fumbled a move to a different plugin structure, and thus you may need to run the latest 64-bit version alongside an older 32-bit version in order to run PythonScript and older plugins such as Multiline Search/Replace (appears under ‘Tool Bucket’), Column Sorting and Line Filter 2.

---

2. WildGem 1.3.

A tool for building and testing ‘regular expression’ or ‘regex’ commands. Find and replace with these commands, and see the resulting changes (if any) in realtime. This software can hide some of the more common ‘regex’ snippets behind more user-friendly visual icons. Useful for instantly testing ‘regex’ command formulas you find, to see if they work, without having to wrestle with Notepad++. This is portable Windows software. In order to save your UI layout preferences, it must be run in Administrator mode.

---

3. CSVed.

A CSV file editor, an alternative you may prefer to behemoth software such as Excel. Move lists and sections around, split lists, add to lines. Appears to lack ability to do column numbering for lists (for which see List Numberer, above).

---

4. Openview’s Index Generator 7.0.

Dedicated to creating a back-of-the-book index for a book. This one is more about the creation of the list, admittedly, but it has various filtering options while doing this. The curious lack of a ‘filter for capitalised words only’ filter make it far less useful than its paid competitor. Asks for a donation on exit. (Note that Softpedia’s review states you “upload a document to the program”, and this wording may mislead the casual reader into thinking this is partly online cloud-linked software. It isn’t, it’s standalone Windows software).

---

5. There’s a free Selected HTML page-content to Markdown addon for chrome-based browsers, and also a Markdown to BBCode converter.

May be useful if either Markdown or BBCode is easier to work with, re: sorting and cleaning list-shaped content grabbed from a Web page. The latter is a self contained javascript-based Web page and can work offline, just save the page locally and re-open it.

---

Tutorial: simple web-scraping with freeware

This tutorial and workflow is for those who want to do fairly light and basic web-scraping, of text with live links, using freeware.

You wish to copy blocks of text from the Web to your Windows clipboard, while also retaining hyperlinks as they occurred in the original text. This is an example of what current happens…

On the Web page:
This is the text to be copied.

What usually gets copied to the clipboard:
This is the text to be copied.

Instead, it would be useful to have this copied to the clipboard:

---

Why is this needed?

Possibly your target text is a large set of linked search-results, tables-of-contents from journals, or similar dynamic content which contains HTML-coded Web links among the plain text. Your browser’s ‘View Source’ option for the Web page shows you only HTML code that is essentially impenetrable and/or unfathomable spaghetti — while this code can be copied to the clipboard, it is effectively un-usable.

Some possible tools to do this:

I liked the idea and introductory videos of the WebHarvy ($99) Web browser. Basically this is a Chrome browser, but completely geared up for easy data extraction from data-driven Web pages. It also assumes that the average SEO worker needs things kept relatively simple and fast. It’s desktop software with (apparently) no cloud or subscription shackle, but it is somewhat expensive if used only for the small and rare tasks of the sort bloggers and Web cataloguers might want to do. Possibly it would be even more expensive if you needed to regularly buy blocks of proxies to use with it.

At the other end of the spectrum is the free little Copycat browser addon, but I just could not get it to work reliably in Opera, Vivaldi or Chrome. Sometimes it works, sometimes it only gets a few links, sometimes it fails completely. But, if all you need to occasionally capture text with five or six links in it then you might want to take a look. Copycat has the very useful ability to force absolute URL paths in the copied links.

I could find no Windows-native ‘clipboard extender’ that can do this, although Word can paste live ‘blue underlined’ links from the clipboard — so it should be technically possible to code a hypothetical ‘LinkPad’ that does the same but then converts to plain text with HTML coded links.

My selected free tool:

I eventually found something similar to Copycat, but it works. It’s Yoo’s free Copy Markup Markdown. This is a 2019 Chrome browser addon (also works in Opera and presumably in other Chrome-based browsers). I find it can reliably and instantly capture 100 search results to the clipboard in either HTML or Markdown with URLs in place. You may want to tick “allow access to search engine results”, if you plan to run it on major engines etc. Update: it can also just copy the entire page to the clipboard in markdown, no selection needed!

---

Cleaning the results:

Unlike the Copycat addon, it seems the ‘Copy Markup Markdown’ addon can’t force absolute URL paths. Thus, the first thing to check on your clipboard is the link format. If it’s ../data/entry0001.html then you need to add back the absolute Web address. Any text editor like Notepad or Notepad++ can do this. In practice, this problem should only happen on a few sites.

You then need to filter the clipboard text, to retain only the lines you want. For instance…

Each data block looks like:

    Unwanted header text.
    This is [the hyperlinked] article title.
    Author name.
    [The hyperlinked] journal title, and date.
    Some extra unwanted text.
    Snippet.
    Oooh, look… social-media buttons! [Link] [Link] [Link] [Link]
    Even more unwanted text!

You want this snipped and cleaned to:

    Author name.
    [The hyperlinked] article title.
    [The hyperlinked] journal title, and date.

Notepad++ can do this cleaning, with a set of very complex ‘regex’ queries. But I just couldn’t get even a single one of these to work in any way, either in Replace, Search or Mark mode, with various Search Modes either enabled or disabled. The only one that worked was a really simple one — .*spam.* — which when used in Replace | Replace All, removed all lines containing the knockout keyword. Possibly this simple ‘regex’ could be extended to include more than one keyword.

The fallback, for mere mortals who are not Regex Gods, is a Notepad++ plugin and a script. This takes the opposite approach — marking only the lines you want to copy out, rather than deleting lines. The script is Scott Sumner’s new PythonScript BookmarkHitLineWithLinesBeforeAndAfter.py script. (my backup screenshot). This script hides certain useful but complex ‘regex’ commands, presenting them as a simple user-friendly panel.

This script does work… but does not work in the current version of Notepad++. Unfortunately the new Notepad++ developers have recently changed the plugin folder structure around, for no great reason that I can see, and in a way that breaks a whole host of former plugins or which make attempted installs of these confusing and frustrating when they fail to show up. The easiest thing to do is bypass all that tedious confusion and fiddly workarounds, and simply install the old 32-bit Notepad++ v5.9 alongside your shiny new 64-bit version. On install of the older 32-bit version, be sure to check ‘Do not use the Appdata folder’ for plugins. Then install the Notepad++ Python Script 1.0.6 32-bit plugin (which works with 5.9, tested), so that you can run scripts in v5.9. Then install Scott Sumner’s new PythonScript BookmarkHitLineWithLinesBeforeAndAfter.py script. Install it to C:\Program Files (x86)\Notepad++\plugins\PythonScript\scripts.

OK, that technical workaround diversion was all very tedious… but now that you have a working useful version of Notepad++ installed and set up, in Notepad++ line filtering is then a simple process.

First, in Notepad++…

    Search | Find | ‘Mark’ tab | Tick ‘Bookmark line’.

This adds a temporary placeholder mark alongside lines in the list that contain keyword X…

In the case of clipboard text with HTML links, you might want to bookmark lines of text containing ‘href‘. Or lines containing ‘Journal:‘ or ‘Author:‘. Marking these lines can be done cumulatively until you have all your needed lines bookmarked, ready to be auto-extracted into a new list.

Ah… but what if you also need to bookmark lines above and below the hyperlinks? Lines which are unique and have nothing to ‘grab onto’ in terms of keywords? Such as an article’s author name which has no author: marker? You almost certainly have captured such lines in the copy process, and thus the easiest way to mark these is with Scott Sumner’s new PythonScript (linked above). This has the advantage that you can also specify a set number of lines above/below the search hit, lines that also need to be marked. Once installed, Scott’s script is found under Scripts | Python Scripts | Scripts, and works very simply and like any other dialogue. Using it we can mark one line above href, and two lines below…

Once you have all your desired lines bookmarked, which should only take a minute, you can then extract these lines natively in Notepad++ via…

    Search | Bookmark | ‘Copy Bookmarked lines’ (to the Clipboard).

This whole process can potentially be encapsulated in a macro, if you’re going to be doing it a lot. Perhaps not necessarily with Notepad++’s own macros, which have problems with recording plugins, but perhaps with JitBit or a similar automator. The above has the great advantage that you don’t have to enter or see any regex commands. It all sounds fiendishly complicated, but once everything’s installed and running it’s a relatively simple and fast process.

---

Re-order and delete lines in data-blocks, in a list?

Scott Sumner’s script can’t skip a line and then mark a slightly later line. Thus the general capture process has likely grabbed some extra lines within the blocks, that you now want to delete. But there may be no keyword in them to the grab onto. For instance…

    [The hyperlinked] article title
    Random author name
    Gibbery wibble
    Random journal title, random date

The Gibbery wibble line in each data block needs to be deleted, and yet each instance of Gibbery wibble has different wording. In this case you need either: the freeware List Numberer (quick) to add extra data to enable you to then delete only certain lines; or my recent tutorial on how to use Excel to delete every nth line in a list of data-blocks (slower). The advantage of using Excel is that you can also use this method to re-sort lines within blocks in a long list, for instance to:

    Random author name
    [The hyperlinked] article title
    Random journal title, random date

---

Alternatives?:

Microsoft Word can, of course, happily retain embedded Web links when copy-pasting from the Web (hyperlinks are underlined in blue, and still clickable, a process familiar to many). But who wants to wrestle with that behemoth and then save to and comb through Microsoft’s bloated HTML output, just to copy a block of text while retaining its embedded links?

Notepad++ will allow you to ‘paste special’ | ‘paste HTML content’, it’s true. But even one simple link gets wrapped in 25 lines of billowing code, and there appears to be no way to tame this. Doing the same with a set of search engine results just gives you a solid wall of impenetrable gibberish.

There are also various ‘HTML table to CSV / Excel’ browser addons, but they require the data to be in an old-school table form on the Web page. Which search and similar dynamic results may not be.

There are plenty of plain ‘link grabber’ addons (LinkClump is probably the best, though slightly tricky to configure for the first time), but all they can grab is the link(s) and title. Not the link + surrounding plain-text lines of content.

There were a couple of ‘xpath based’ extractors (extract page parts based on HTML classes and tags), but in practice I found it’s almost impossible to grab and align page elements within highly complex code. Even with the help of ‘picker’ assistants. I also found an addon that would run regex on pages, Regex Scraper. But for repeating data it’s probably easier to take it to per-line Markdown then run a regex macro on it in Notepad++.

The free ‘data scraper’ SEO addons all look very dodgy to me, and I didn’t trust a single one of them (there are about ten likely-looking ones for Chrome), even when they didn’t try to grab a huge amount of access rights. I also prefer a solution that will still go on working on a desktop PC long after such companies vanish. Using a simple browser addon, Notepad++ and Excel fits that bill. If I had the cash and the regular need, I would look at the $99 WebHarvy (there’s a 14-day free trial). The only problem there seems to be that it would need to be run with proxies, whereas the above solution doesn’t as the content is simply grabbed via copy-paste.

How to remove every nth line in a list

How to remove every nth line in a list. The list is made up of repeating four-line blocks of text.

The situation:

You have a long and mostly cleaned text list that looks like this…

Random article title
Random author name
Gibbery wibble
Random journal title

Random article title
Random author name
Gibbery wibble
Random journal title

Random article title
Random author name
Gibbery wibble
Random journal title

… and you of course wish to delete all the unwanted Gibbery wibble lines. All the Gibbery wibble text is different. Indeed, there’s no keyword or repeating element in each four-line data block for a search-replace operation to the grab onto. The only repeating element is the blank line that separates each data block of four lines.

So far as I can see, after very extensive searching, there’s as yet no way to deal with this in Notepad++, even with plugins and Python scripts.

The slower solution:

The more flexible but longer solution is Excel. However, the latest version of Notepad++ (not the older, 32-bit version) will let you quickly take the first and vital step. You first delete the blank lines with…

Edit | Line Operations | Remove Empty Line

It’s far easier to delete blanks in a long list in Notepad++, rather than wrestling with complex ten-step workflows in Excel, just to do such a simple thing.

Then you copy-paste the list into a new Excel sheet. You then add these two Excel macros and run the first. Both run fine in Excel 2007. The first splits the column into chunks of 4 (if you have three lines per block, change all the 4s in the macro to 3s, if six lines then change to 6s, and so on). Each chunk is placed into a new column on the same sheet.

You can then delete the offending Gibbery wibble row, which will run uniformly across the spreadsheet. In this example, it all runs across row 3.

The second macro is then run and this recombines all the columns back into a long list, and places the recombined list onto a new sheet.

The free ASAP Utilities for Excel can then ‘chunk’ this list back into blocks of four, enabling you to add a blank line between each block. Optionally, you can add the HTML tag for a horizontal rule.

The same core method can be used to re-sort the lines each block, or to add numbering to each line as: | 1. 2. 3. 4. | 1. 2. 3. 4. | These operations are something Notepad++ can’t yet do.

The quicker solution:

If you need a quicker option, and don’t need to re-sort the lines in each data block in Excel, then try the Windows freeware List Numberer. As you can see below, once you’ve used this utility to run a simple operation, then a regex search back in Notepad++ (.*line3.* — used in Replace | Replace All) will clear all the unwanted lines.

How to save a multipage Web-book, or full set of journal articles, to a single PDF file.

How to save a multipage Web-book or full set of journal articles to a single PDF file.

Situation: You sometimes encounter full books online, split up into perhaps 300 or more separate HTML Web pages, each containing a bit of text from the book. You wish to re-combine this chopped-up book into a single offline PDF or ebook file, with the bits assembled in the correct order. You might want to do the same with a large journal issue. You need some Windows freeware to solve this, and don’t wish to use cloud/upload services.

Solution: A free Chrome browser plugin, and a Windows freeware utility.

Test book: Minsky’s The Society of Mind from Aurellem, with nearly 300 HTML pages. These are all linked from the main front page, which shows a linked table-of-contents.

Clicking heck!

---

Step 1. Install Browsec’s Link Klipper – Extract all links browser addon (for Chrome based browsers, inc. Opera) or similar. Run it on your target Web page. Open the resulting plain-text list of extracted URL links, re-order these as needed, and then copy the list of the links you want to the Windows clipboard.

One problem you may encounter here is that the filenames may be obfuscated, as perhaps jj8er4477-j.html rather than Chapter-1.html. But it seems that Link Klipper follows the URLs down in in-HTML sequence, and thus presents them in a list in the same manner. Linkclump is a good alternative browser addon, for those who need precise and manual control of the URL capture, though it is probably a bit fiddly for the first-time user to get working in that manner.

Note that Link Klipper is meant for the SEO crowd, so it can also do Regex and can save to .CSV for sophisticated link-sorting with Microsoft Office Excel.

Step 2. The genuine Windows freeware Weeny Free HTML to PDF Converter 2.0 can then accept Link Klipper’s simple URL list. Just paste it in…

Weeny is very simple to get running and will then go fetch and save each URL in order, outputting a clean PDF for each (as if it had been saved from a good Web browser). There’s no option to select repeating parts of each page to omit, it saves all-or-nothing. It can’t process embedded videos and similar interactive/multimedia elements.

During the saving process Weeny may appear to freeze, showing ‘Not Responding’, if fed hundreds of HTML pages. However, an inspection of the output folder will show that PDFs continue to be converted and dropped into it one-by-one. Thus, even if Weeny seems to choke and crash on 300+ files, it hasn’t done so. Just let it run until it completes.

If the Link Klipper URL list was in the correct sequence, then a sort ‘By Date’ of the resulting PDF files should place the book parts in their correct order, even if the filenames were obfuscated.

We could have downloaded the pages as HTML, but in practice it’s not viable to then join them up. Inevitably, there’s some broken HTML tag somewhere in the combined file, and that causes problems in the text which start to cascade down. PDF is the more robust format.

Step 3. OK, so that’s fairly quickly and easily done. But, oh joy… you’ll now have nearly 300 PDF files, all very nice-looking… but separate! Weeny is sweet software, but not very powerful and thus it doesn’t also join the PDFs together.

If you have the full paid Adobe Acrobat (not the Reader) then you can combine these PDFs very easily (or ‘bind’ them, in Adobe-speak). Acrobat also offers the great benefit of file re-ordering by dragging.

You’re done, and the whole process should have taken ten minutes at most. If the font is not ideal for lengthy reading, the free Calibre can convert the PDF to .RTF or .DOCX for Word, HTML, and various eBook formats.

---

“But I don’t have the full Adobe Acrobat”:

For those who need freeware for this last step of combining the PDFs, you need to find one that offers a similar ‘re-ordering by dragging’ to Adobe Acrobat. Such freeware is not at all easy to find. Most such Windows utilities are old and use very clunky up/down buttons for re-ordering. That’s not so useful if you have a file number 298 that needs to be moved up to become file 1 — you’re only going to want to do that by dragging, not by clicking a button 297 times. Why might you need to re-order? Because with a big book, you almost certainly got the file order a little wrong, when glancing down and editing the initial URL list in Link Klipper.

Eventually, I found the right sort of free software to do the job. DocuFreezer 3.0 is free for non-commercial use, only adding a non-obtrusive watermark “Created by free version of Docufreezer”. It’s robust and good-looking 2019 software, and it needs Microsoft .NET Framework 4.0 or higher to run (which many Windows users already have).

DocuFreezer can re-sort the imported PDF files ‘By Date’, or by some slightly fiddly dragging (a feature which seems unique among such freeware). It can even OCR the resulting PDF. You just need to remember to tell it to combine and save as a single PDF, and to do the OCR…

It’s reasonably fast, if you don’t OCR. Removing the watermark, by getting the paid Commercial version, costs $50. Even so, Docufreezer’s free version is no problem if all you want is a personal offline PDF of a ebook for reading — the watermark is quite discreetly placed on the side edge of each page in plain black lettering…

You can also see here that the embedded video, from the original HTML page, was elegantly worked around by Weeny while retaining the page’s images and font styling.

---

To .CBZ format:

Theoretically one could use this process to then get .JPG files, to compile offline versions of webcomics like Stand Still. Stay Silent., and other primarily visual sequential content. If you have the full Adobe Acrobat then it’s easy to save out the PDFs as big page-image .JPGs in sequence, bundle them into a .ZIP, rename the .ZIP to .CBZ file… and you’re done.

Though you may then encounter a problem in the layout. Unlike mostly-text books, webcomics and other visuals may not fit well on a single portrait-oriented PDF page, without running over. In other words, if you need to scroll down the Web page to see the whole image, then your final PDF page-flow may not be ideal.

In practice, most PDF-to-JPG freeware utilities are not viable in this workflow. I found only a few, and they either contain third-party ‘toolbars’ or just don’t install on modern Windows, or the JPGs they produce are watermarked. They would also need to offer file re-sorting by-dragging, and robust batch processing, and a file mask to rename the .JPG files sequentially — because it’s important for a CBZ to have its filenaming of pages be properly sequential (0001.jpg, 0002.jpg). I’d welcome hearing of such a freeware, but I don’t think it currently exists.

The better option then is simply to read the material online. Or if you really need it offline, then use a free open source website ripper such as HTTrack Website Copier to make a mirror of the website and set to only save the .JPGs to your PC. This assumes that the website doesn’t have traffic surge control or anti-ripper measures in place. But you should really be supporting the comic maker and buying their paid Kindle ebook editions.

---

“Ooh, does the workflow work on open access journal TOCs?”:

Yes, indeed it does. Not all open/free journals also offer PDF version of their articles, especially those in a more magazine-like, trade journal, or blog-like format. In such a case, one can run the above quick workflow on the issue’s TOC page, thus quickly providing yourself with a per-issue portable offline single-PDF for your favourite journal in the garden or at the beach. You can then run it through the free Calibre to get it to various ebook formats such as .MOBI (Kindle ereader) and .ePUB.

For a journal issue where PDFs links are already present beside the HTML article links, but there are a great many PDFs, then the browser addon Linkclump is your best option to grab them all.

Clicking heck, that’s a lot of PDFs in an issue! And there’s no single-volume PDF.

You can set up LinkClump to select / open / download all the PDF links (this works even with repository and OJS redirects which use /cgi/), to grab the PDFs for joining with DocuFreezer or some other free desktop PDF joiner. This method is a lot easier than fiddling around with a bulk downloader browser-addon, and picking out the PDFs from a long jumbled list of files. Or you can have LinkClump grab a list of the HTML article URLs for processing to a PDF book, with the above Klipper – Weeny – DocuFreezer workflow.

---

“My super-mega-combo PDF is too big”:

If your resulting PDF is too large to Send to Kindle (Amazon has as a 50Mb per-file transfer limit, and many people also have very slow uplinks), then there are a couple of PDF shrinkers worth having, from the freeware but rather clunky Free PDF Compressor to the slick and easy $20 PDF Compressor V3 (I like and use the latter a lot).

How to resize a huge image without opening it

This mini-tutorial may be useful for those working in museums and archives, or doing print work with big maps, who encounter image files so huge they can’t be opened.

Problem: You have been sent a huge image file, and you need to shrink and resize it. It’s very nice that the sendee only sent a 80Mb .JPG rather than their 8Gb original map sheet. But even their 28,000 pixel .JPG can’t be opened on your puny PC. Even Photoshop and IrfanView balk at loading/viewing it. You can’t use the free GigaPXTools (for re-sizing gigapixel images without opening them) because… it prefers .TIFs and can’t work with .JPG files.

Solution: the free IrafanView can shrink it, if used in batch mode.

Method: First close down everything on your PC that’s not essential and might be eating your PC’s memory. Make sure you have the latest 64-bit IrfanView.

1. Then place your target image, on its own, in a new folder called ‘batch’.

2. Open any small image in IrfanView, to launch the software.

3. Press ‘B’ to open Batch conversion. In “Look in:” navigate to your target folder. You have have to do this a few times until your target image appears as a preview thumbnail. Do not click on the preview thumbnail (as this will almost certainly crash IrfanView)! This step will at least show you that file is viable as an image, and not corrupted.

Now you can see why we needed the image on its own in its own folder — IrfanView has enough to do with just showing a preview of this file, let alone previewing any other images in the same folder at the same time. If even this step proves too much for IrfanView, try switching to file-name view only, and then re-starting the process.

4. Then tick “Use Advanced Options” and set “Resize” to 3000 pixels, or whatever reduced size you want. Click OK.

5. Set your target output folder. Without clicking on the Preview thumbnail, click the “Add All” button to add the target file to the list of files to be processed.

6. Click “Start Batch”. Then you go off and have a coffee, as it’s likely to take 20 minutes. A Windows message that the software is “Not responding” is normal. Eventually there will be a message from IrfanView that the batch process has completed.

Lightkey – the free version tested

Lightkey – Free Basic Edition. This offers the LightkeyPad text editor with smooth predictive (autocomplete) text, and the editor learns rapidly as you type. The paid version works with Microsoft Office 2010 (and higher) and the Google Chrome Web browser.

But the free version of Lightkey seems fine, albeit after a download and install that seemed to take aeons. There’s also a direct download link for the .exe here, for those who want offline installs. Assume you’ll be spending a while on getting it down, and then installed and up and running. But once you finally get out the other side of that slough, and the profile-building, LightkeyPad turns out to be a pleasing simple text editor with fluid predictive auto-typing and some light-touch spelling/grammar correction.

Initially I thought it was not very British, as it wanted to offer Stanford for “Stoke-on-Trent”, but it can do the county name “Staffordshire” out of the box. And type “Stoke-on-Trent” a few times and it even gets the hang of that. As such, it’s not necessary to manually set up a personal configuration file of words. It seems the software will learn those as it goes along.

It can even cope with “Lovecraft” after you type his name a few times. “Cthulhu”, too. Four times seems to be the usual times you type a new word before the software “knows” it. It can’t correct “If can’t” to “It can’t”. Nor can it autoclose HTML tags, leaving you to add the URL and title in the middle.

If you find this freeware useful will rather depend on what sort of typist you are. Do you look down at the keyboard as you type with two fingers, or look at the screen while ‘typing blind’? That will also partly depend on when you type, as typing in low light is not so easy either way (unless perhaps you have a snazzy gamer keyboard where the letters glow-in-the-dark).

The user presses the Tab key on the keyboard to confirm a suggested word, and doing this rapidly becomes easy and reflexive.

There’s no post-typing spell-check or grammar checker to run over the entire finished text, and such things happen as you type. There is a word-count for the finished text, which is handy, but the final pass of spellchecking then needs to be done on pasting the text into Word or WordPress. I’m guessing there may be a one-click “send to Word” button in the Pro version of LightkeyPad.

There’s a ‘dark mode’ done in nice midnight-blues, that’s easily accessed via a big button. The icons on the top bar are neat and pleasing.

The software saves to .TXT or .RTF format.

What would improve it?

* A simple search-replace would be welcome, but I guess that would mess with the usefulness of the typing and word analysis. A “Select all” menu item is also curiously missing.

* It needs to save its configuration. On launch it doesn’t remember the window height, or the chosen font from the previous session. (It’s possible to work around the font problem by saving a “template.RTF” with one’s chosen font set up, and then having a Windows Startup link to that file and thus launch Lightkey. That’s not ideal, but it works. Lightkey also insists on saving to My Documents every time, instead of to the last save location.

* Loading time could be a touch quicker, as it’s not as instant as Notepad. But if you launch it at Windows Startup, as suggested above, then it’s already open and is as quick to spring into action as Notepad is.

* I disliked its Taskbar ‘hidden icons’ panel icon, which — being slightly slanted — looks ugly and jaggy (because it’s not being aliased). On closer inspection, however, this turns out to be the LightKey Control Center .exe, this having been launched as a silent Startup launch, and it’s not LightkeyPad. As such, simply Ctrl-Alt-Del to get to the Windows Startup services tab and from there disable the LightKey Control Center permanently. The LightkeyPad editor doesn’t appear to need it in order to work.

What about privacy? It does create a profile based on your “recent documents”, but there is an optional scan of these at the install point, and…

“the user’s documents and emails, along with their typing data, will NOT be sent or collected by Lightkey’s servers in any way, as they are the user’s private property.”

But some sort of unique cumulative hash may be being sent, and as such you’ll want to install offline and then uncheck this item ASAP…

It also learns from your typing, so that unique word/keystroke record is presumably being stored somewhere on your PC. That’s potentially a valuable ‘fingerprint’, from which things like business secrets might be somehow reconstructed. In which case a firm won’t want such data slipping out of the user’s PC and being sent off to Whereizitagin. I’m not suggesting any malfeasance here on the part of the makers, but just that you need to be sure that such a local file — if it exists — is truly secure.

Overall LightKey is an interesting development in genuine freeware, and even out-of-the-box it’s not as annoying as you might think an auto-complete text editor would be. I loathe always-wrong Web search auto-complete as much as the next user, but this software does auto-complete and substitutions quite nicely and gets in the way as little as possible. As such I think I have a new first-draft composition Notepad replacement here, and will try it as a replacement for a few weeks.

---

Desktop alternatives?

* The free Notepad++ can do some Auto Completion natively, albeit via a method that’s not changed in over a decade and in a fiddly-looking way that’s aimed at coders. It can even do autoclose, with the help of several coder-focussed plugins. There’s also the Presage plugin for Notepad++, though that was last updated in 2015. There is what looks like a Presage Windows standalone, but it actually seems like it just runs a service at Windows startup that’s required by the Notepad++ plugin.

* If you really need the search-replace in similar free software, PredictEd 1.1 is open source freeware from 2018 and similar to Lightkey. But it’s very basic in design and far more clunky in its method.

* Predictive Tab Key Auto Complete for Chrome browsers. Again, free, and the only somewhat-recent one on the Chrome store. Not updated since 2015, and it’s about as annoying as you’d expect when composing a WordPress blog post. Very very fast, but not accurate and doesn’t appear to learn in any way I could discern. It would be great to see something like this that could be constrained only to a list of user-defined words and phrases (see Auto Text Expander below), but this doesn’t appear to be it.

* Windows? Well, Apparently Windows 10 has a “Show text suggestions as I type” setting in Windows, but I have no idea if it’s more than ‘Microsoft Clippy reborn’.

* In Word, one can set up custom autocorrects: select Word Options | ‘Proofing’ | ‘AutoCorrect Options’ | ‘Replace Text as you type’. #h can be set to autocorrect to href, for instance. The free LibreOffice apparently offers AutoCorrect that works with words longer than eight letters, and can be similarly customised. But who wants to launch either of these lumbering behemoths, just to do what should be done in Notepad?

* Paid? the $130 Typing Assistant 8.x works with any Windows software and Web browsers, does what is says, is 64-bit and developed. Which is presumably why it’s so expensive. The old 32-bit abandonware Smart Type Assistant for Windows is similar and is now free.

---

There are also paid ‘abbreviation managers/expanders’ like PhraseExpress and Breevy and FastFox, but they are: i) expensive and mostly way too complicated for such a simple task; and ii) rely on you being able to remember the abbreviation, or to remember to place a # in front of the start of a word, or which letter to stop at to trigger the expansion without overtyping. ShortKeys Lite is a rather ancient and very clunky free choice here, and there are a couple of equally ancient Notepad++ plugins whose makers used descriptions such as ‘snippet’ and ‘substitution’.

Similar Web browser addons are ProKeys, Auto Text Expander and Text Blaze (beta). Also the right-click Paste Email.

All these are useful for adding your signature, email addresses and boilerplate text, but in practice are not so useful for composing ‘in the flow’.