Use Search Regex to delete part of a URL, in WordPress

Problem: You need a working regex to delete an /unwanted section/ of a URL, on a self-hosted installation of WordPress. The /unwanted section/ is different in each URL, and you have hundreds or even thousands of such URLs to deal with.

Solution: If your /unwanted section/ string has a repeating section of the URL in front of it, then you’re in luck. The popular free WordPress plugin ‘Search Regex’ and this small regex, will do it. The regex is seen here working as expected in the RegularExpressions101 sandbox…

What it’s doing: The desired repeating URLpart2 in the URL path is being found and marked, along with its following / slash. This and whatever follows the found section is also marked for deletion, even if what follows is different in every URL. The ‘marking for deletion’ stops at the next / in the URL.

In WordPress: Here’s how it’s applied in Search Regex in WordPress. Because this example is also deleting the repeating URLpart2/, that gets added back via the “Replace” box.

Warnings: Always preview first, using “Search”. Only if you are absolutely happy with how the URL now looks in the previews, should you then press “Replace All”. Always make a backup of the WordPress install before attempting such changes.

Put a Python in your Web browser

Embed Python scripts in HTML with PyScript…

embed Python programs directly in HTML pages and execute them within the browser without any server-based requirements

What could possibly go wrong?

New category for posts: ‘Regex’

A new category for posts on this blog, Regex. I’ve gone back and retrospectively tagged old posts with it.

Here’s a regex to handle deleting image thumbnail suffixes in WordPress

When placing an image in a blog post, on a server installation of WordPress you often get something like this code…

The main image is only linked to, and an ersatz auto-generated thumbnail is what’s shown on the page. One may wish, for various reasons, to change an .XML archive of a WordPress blog so as to only have the filename. If one could remove the suffix seen here in yellow…

… then the blog post will only need to call the original image. The HTML code will take care of the resizing on the blog post.

Such snipping is useful if you only archived the .XML and the original images. You may no longer have or never harvested the multifarious thumbnails that were auto-generated by your server installation of WordPress.

Such filename snipping can be done, and with a simple regex formula, in Notepad++. But first… check what images you have in your local archive for the blog, since it’s just possible that the harvesting collected the thumbnails and not the originals.

Definitely have the originals, with no suffixes? Ok, then let’s proceed. Thankfully all the added filename suffixes have certain repeating elements, even if they have differing pixel dimensions. Thus they can be handled by a regex.

In Notepad++ the following working and tested regex will search and delete the thumbnail extensions, even if they all have different pixel dimensions:

FIND: \|*-([0-9]+)x([0-9]+)\.jpg\|*

REPLACE: .jpg

In plain English: find everything between any – and a following .jpg and if it has the general form of numberxnumber then delete it along with the – and .jpg. Replace the deleted string with .jpg and then repeat the process down through the whole .XML file.

Possibly regex gurus will shriek and swoon at my formulation seen above, but… it works for me.

---

If you want this regex to run automatically on any post you make on WordPress, as you blog, then Regex Replace extension for the Chrome browser will do the job. It automatically removes the suffix after you “Update” (i.e. save) the post, not when you first insert the image. Thus you’ll need to press “Update” twice on your blog post.

Tested and working. You probably also want to tell Regex Replace to run only on your target blog or website.

You don’t usually need this on free WordPress.com blogs, because they do things differently when placing an image on a blog post.

If you have a fixed width on your blog posts, you can also prevent WordPress from generating unwanted thumbnails thus…

… here I have gone to Media | Settings in the Dashboard, and set 0 to prevent all except “medium” from generating. This is set to the width of the blog post, so I get the HTML sizing code, but the regex ensures all my posts both display and link to the source image. Obviously in this use-case you’ll try not to inflict file sizes above 500kb on your readers.

How to search-replace all image paths in a .XML WordPress export

Here’s how to search-replace all image paths in a .XML WordPress blog export, using the freeware Notepad++.

The idea here is to point them all to a folder of /oldimages/ with your cache of old blog images it it. Otherwise, if there’s no live blog for the new WordPress install to go and fetch them from, they won’t show on the blog posts. So you ideally want all your images pointing to mysite/blog/oldimages and that’s where you upload all your archived blog images.

The \d+ bit stands for ‘any date number’ e.g. /01/02/ It’s a sort of ‘wildcard for dates’. This does not work inside of WordPress, for instance when you have a .XML already imported and a regex plugin to call on. It appears to be Notepad++ specific.

This method assumes you haven’t been letting WordPress rename your image files by size. e.g. upload fluffybunny.jpg and have WordPress insert it as a shrunken fluffybunny-500×350.jpg along with a link to the main and properly name image file. In which case deeper surgery is required.

With a free WordPress blog you may not have this problem. For instance, the above image is placed in the post in shrunken form. But this is done by a bit of code that gets appended to the file path…

… and the image itself is not duplicated, shrunken and renamed. Even when, as a 585 pixel wide image, it only appears in the blog post at 529 pixels.

---

Also useful here is the old Windows freeware WXRsplitter which will split a single WordPress .XML archive into smaller but valid chunks. Often, a large single .XML (known to WordPress as a WXR file) will not upload once it gets beyond about 18Mb or so. The freeware still works fine. Once the .XML is chunked, you just upload and import each piece in the numbered sequence.

Freeware to convert a WordPress blog .XML export to Word

Hurrah. Desktop software has been found that will robustly convert all of a WordPress blog .XML export to Word. Specifically, a free blog hosted at the .com WordPress, which doesn’t permit or offer any fancy ebook conversion plugins. The solution is good old Windows freeware, as usual. Though the various freeware directories know nothing of it, and it was only found after hours of digging.

The XML to Doc software is the snappily titled wpxslgui…

wpxslgui is a Windows application which converts an XML File generated by the WordPress Export function into an HTML or Word HTML document.

It’s Windows freeware in v.1.04 (June 2020) from Devio IT Services, being the worthy and generous Herbert Oppolzer of Austria. Tested and working here. Very simple usage, with a Windows GUI.

Also includes the option to… “Convert WordPress XML to a single HTML file allowing filter by category (JavaScript)” but the “Word HTML” output saves as a .DOC file.

However if you cleverly just re-name the .DOC to .HTML it then works fine as a Web page in a Web browser, and thus calls in the images. I’m assuming here that your browser is allowed to go online, but Word is not.

The overall aim here is to get the blog to a clean ebook format for Kindle, removing as much gunky code and blog-cruft as possible. Archivists may also be interested here. As such there are some initial changes you may want to make…

1). First you may want to delete the superfluous 24-hour timestamp by editing the WordPress .XML output itself. What you’re targeting looks like this…

Since datestamps have a unique pattern, a regex can deal with them. Simply deleting all timestamps in Notepad++ with a regex is…

 \d{2}:\d{2}:\d{2}

…and note the single space at the start of the regex.

Running this removes all timestamps but leaves the date intact. Possibly a regex could also re-work the main date to something nicer (e.g. 12th July 2015), but it would likely be a very complex regex.

2). After conversion wpxslgui adds italics on post titles. The CSS stylesheet is embedded at the top of the output HTML, and so changing italics is just a matter of tweaking font-style:italic on H2. Bold might be better.

3). While you’re in the CSS you may want to have external links be something other than blue and purple. In which case edit the colours of a:link and a:visited in the header CSS.

4). wpxslgui adds numbering of posts, in front of each post title. The code spans multiple lines and looks like this…

Awkward, but not impossible for a regex to fix. This Search-Replace regex for Notepad++ will replace all of these with a tilde or whatever other elegant typographer’s HTML mark you might want.

5). Once you have output in semi-cleaned HTML, Ctrl + A should “Select all…” from the browser and then you paste to whatever you’re using. If you were keen on YouTube embeds you will then need to manually go through and delete the WordPress code for these. The WordPress YouTube ‘slug’ insert is presented raw in the post. Without a WordPress installation the code can’t call the video. The same will likely be true for any other fancy embeds of maps, charts, podcasts etc.

Finally, note also that wpxslgui only deals with posts, not pages.

One-click to remove a verbose site from Google

One-click to remove a verbose site from Google Search results, a new UserScript. Preset for Wikipedia, but the URL can be easily changed to be any verbose website. It should ideally be a website that you usually regularly want to remove from search results, but sometimes want to keep. The script is thus more flexible than a regular list-based site blocker.

It works by re-running the current search, but only an instant after some regex has cunningly inserted the command    into the URL.

---

Also, yes, I’m aware that my ‘add JURN as a link to Google Search’ UserScript has stopped working. Google has re-labelled the divs on the text links just below the search box. A similar script that allows the current search to be passed to Scholar has also stopped working, as have several similar menu scripts. I’m waiting for one of these scripts to update, and thus to show me how it needs to be fixed.

Some free tools to extract data from fetched HTML

Here are some relatively simple free Windows desktop tools to ‘extract an item of data from fetched HTML’. They were found while considering if it might be possible to append ISSNs to the JURN directories in a semi-automatic manner.

My target task was: you have a big list of URLs, and the HTML pages for these are to be automatically fetched. Their text is then regex-ed (or the Excel equivalent) to extract a tiny snippet of text data from each page. In this case, any line following the first instance of the word “ISSN” on each home-page. Ideally, each extracted text snippet is then automatically appended to its source URL.

1. Excel Scrape HTML Add-In, free from Analyst Cave. I can’t do anything with it in Excel 2007, so I assume it needs Excel 2016 or higher (2016 introduced the new features Power Query and Get & Transform).

2. Download WebExtractor360 1.0. Simple Windows abandonware from 2009, and lacking any Help in terms of… how do you format your big list of URLs so they can be automatically processed? It also looks like it cannot be limited to just the first-encountered home-page. Still, someone might figure out that bit of the WebExtractor360 puzzle, or pick up the open source code at SourceForge and develop it for easier batch processing and expanded output options.

3. DEiXTo. Genuine Windows freeware from Greece, for “Web data extraction made easy!” The baffling interface and example-free techie manual strongly suggest otherwise though, and you’ll likely need to read the manual very carefully to get it working. There’s also a 2013 academic paper on DEiXTo from the authors.

4. Update: the open source freeware Web-Harvest 2.x from 2010, Java with a clean Windows GUI and a good manual. Seems like a good alternative to DEiXTo. Still works and has many examples and templates, but no template to run through a list of URLs and grab a fragment of data from each home-page. Despite the name it’s a data extractor, not a site harvester.

5. Update: I made one for Excel 2007, and it’s free. Take a list of home-page URLs, harvest the HTML, extract a snippet of data from each.

For paid Windows desktop software, that doesn’t require a PhD in Spreadsheet Wrangling and which indeed assumes you’re not working in Excel, look at Sobolsoft’s $20 Extract Data & Text From Multiple Web Sites Software and BotSol’s Web Extractor. The first from Sobolsoft requires Internet Explorer and that you delve into two of Explorer’s settings to make it not be verbose, in terms of IE not freaking out with process-stopping alerts every time it meets a Twitter button etc. Search is not ideal, as it cannot be limited to just the first-encountered ‘home’ page. Output it not ideal either, as it cannot offer Source URL = no result as a line in the results. The latter software from BotSol has the great advantage that it can limited itself to the home-page and will also try another 2 nearby pages (“About” etc), if it can’t find the target data on the home-page. It’s designed to extract phone numbers, but can be configured to get anything. It’s free for a version that processes a list of 10 URLs at a time, and is $50 for an ‘unlimited URLs’ version (that is regrettably time-bombed).

There are browser-based tools like the long-standing OutWitHub and new free Cloud services such as Octoparse, but they appear focused on ripping competitor ecommerce listings and plugging them into your boss’s database. Also, apparently Octoparse’s “List of URLs” feature requires all the pages to have exactly the same HTML elements.

Free: My Little Regex Cookbook, for Notepad++

New, My Little Regex Cookbook as a printable eight-page PDF. It has numerous working examples of useful regex for Notepad++ users working with data extraction and text lists. All tested and working in Notepad++.

This is my expanded and now prettified 1.3 PDF version of what first appeared here as the post “Some useful regex commands for Notepad++” in May 2019.

Download: little_regex_cookbook_2020.pdf

Freeware: TextWorx

There’s a relatively new entry in genuine Windows freeware for complex text-manipulation, and this hadn’t been found when I made my summer 2019 survey of Freeware for cleaning and manipulation of text lists.

It’s TextWorx by bgmCoder, a “Universal Text Manipulator”. It lives up to the name, in terms of being able to use it with any text-editor. Highlight the text block you want to work with. Press a keyboard shortcut. Up pops a well-organised tool offering a huge range of “advanced text-manipulation routines”.

The default keyboard shortcut required to trigger the menu is a bit of a contortionist show-stopper, or else it requires you to remove your hand from the mouse:

Win key + K or Win key + shift + K.

But the shortcut is not hardwired and can be changed in the .INI file. And it’s easy enough to trigger a keyboard shortcut with a mouse-gesture. Choose a gesture that ends up somewhere suitable on the screen, since your mouse-cursor position is where the TextWorx interface will appear.

What it doesn’t seem to have is regex functions. It can’t thus function as a handy regex ‘key-ring’. For instance it can’t do things like “Extract all text found between KEYWORD1 and KEYWORD2 to a new List”. For that you’d want regex or Sobolsoft’s £20 “Extract Data Between Two Strings” utility software, which saves the extracted substrings as a list. Or you could save £20 by doing the same with this tested-and-working regex and a copy of the free Notepad++…

FIND:         .*?KEYWORD1(.*?)KEYWORD2|.+
REPLACE:    \1\r\n