HTML Correctness and Validators
• http://xahlee.org/js/html_correctness.html

plain text version follows.

---------------------------

HTML Correctness and Validators

Xah Lee, 2008-12-28

Some notes about html correctness and html validator.

Condition Of Website Correctness

My website “xahlee.org” has close to 4000 html files. All are valid html files. “Valid” here means passing the w3c's validator at http://validator.w3.org/. Being a programing and correctness nerd, correct html is important to me. (correct markup has important, practical, benefits, such as machine parsing and transformation, as picked up by the XML movement. Ultimately, it is a foundation of semantic web↗.)

In programing lang communities, the programer tech geekers are fanatical about their fav lang's superiority, and in the case of functional langs, they are often proud of their correctness features. However, a look at their official docs or websites, they are ALL invalid html, with errors in just about every 10 lines of html source code. It is fucking ridiculous.



In the web development geeker communities, you can see how they are tight-assed about correct use of HTML/CSS, etc, where there are often frequent and heated debates about propriety of semantic markup, and they don't hesitate to ridicule Microsoft Internet Explorer browser, or the average HTML content producer. However, a look at the html they produced, also almost none are valid.

The bad html also happens in vast majority of docs produced by organization of standards, such as the Unicode Consortium↗, IETF↗. For example, if you run w3c's validator on their IETF's home page, there are 32 errors, including “no doctype found”, and if you validate unicode's http://www.unicode.org/faq/utf_bom.html, there are 2 errors. (few years ago, they are much worse. I don't think even “w3.org”'s pages are valid back then.)

In about 2006, i spent few hours research on what major websites produces valid html. To this date, I know of only one major site that produces valid html, and that is Wikipedia. This is fantastic. Wikipedia is produced by MediaWiki↗ engine, written in PHP. Many other wiki sites also run MediaWiki, so they undoubtfully are also valid. As far as i know, few other wiki or forum software also produces valid html, though they are more the exceptions than norm. (did try to check 7 random pages from “w3.org”, looks like they are all valid today.) Personal Need For Validator

My personal need is to validate typically hundreds of files on my local drive. Every month or so, i do systematic regex find-replace operation on a dir. This often results over a hundred changed files. Every now and then, i improve my css or html markup semantics site wide, so the find-replace is on all 4000 files. Usually the find-replace is carefully crafted with attention to correctenss, or done in emacs interactively, so possible regex screwup is minimal, but still i wish to validate by batch after the operation.

Batch validation is useful because, if i screwed up in my regex, usually it ends up with badly formed html, so html validation can catch the result.

In 2008, i converted most my sites from html 4 transitional to html 4 strict. The process is quite a manual pain, even the files i start with are valid.

Here are some examples. In html4strict:

* “‹br›” must be inside block level tags. Image tag “‹img ...›” needs to be enclosed in a block level tag such as “‹div›”. Content inside blockquote must be wrapped with a block level tag. e.g. “‹blockquote›Time Flies‹/blockquote›” would be invalid in html4strict; you must have “‹blockquote›‹p›Time Flies‹/p›‹/blockquote›”

Lets look at the image tag example. You might think it is trivial to transform because you can simply use regex to wrap a “‹div›” to image tags. However, it's not that simple. Because, for example, often i have this form:

‹img src="pretty.jpg" alt="pretty girl" width="565" height="809"› ‹p›above: A pretty girl.‹/p›

The “p” tag immediately below a “img” tag, functions as the image's caption. I have css setup so that this caption has no gap to the image above it, like this:

img + p {margin-top:0px;width:100%} /* img caption */

I have the “width:100%” because normally “p” has “width:80ex” for normal paragraph.

Now, if i simply wrap a “div” tag to all my “img” tags, i will end up with this form:

‹div›‹img src="pretty.jpg" alt="pretty girl" width="565" height="809"›‹/div› ‹p›above: A pretty girl.‹/p›

Now this screws up with my caption css, and there's no way to match “p” that comes after a “div › img”.

Also, sometimes i have a sequence of images. Wrapping each with a “div” would introduce gaps between them.

This is just a simplified example. In short, converting from html4transitional to html4strict while hoping to retain appearance or markup semantics in practical ways is pretty much a manual pain. (the ultimate reason is because html4transitional is far from being a good semantic markup. (html4strict is a bit better)) Validators

In my work i need a batch validator. What i want is a command line utility, that can batch validate all files in a dir. Here are some solutions related to html validation.

* The standard validator service by w3c: http://validator.w3.org/ (see also: W3C Markup Validation Service↗ ). The problem with this is that it can't validate local files, and can't do in batch. Using it to validate 4000 files thru network (with a help of perl script) would not be acceptable, since each job means massive web traffic. (my site is near 754 Mebibyte↗.)

* FireFox has a “Html Validator” add-on by Marc Gueury. https://addons.mozilla.org/en-US/firefox/addon/249. This is based on the same code as w3c validator, can work on local files, is extremely fast. When browsing any page, it shows a green check mark on the window corner when the file is valid. I heavily rely on this tool.

* FireFox has a “Web Developer” add-on by Chris Pederick. https://addons.mozilla.org/en-US/firefox/addon/60 Since Firefox “v.3”, it has a icon that indicates if a page's css and javascript are invalid (has errors), and also indicates whether the file is using Quirks mode↗. I heavily rely on this tool.

I heavily relie on the above 2 FireFox tools. However, the FireFox tools do not let me do batch validation. Over the years i've searched for batch validation tools. Here's some list:

* HTML Tidy↗ A batch tool primarily for cleanup html markup. I didn't find it useful for batch validation purposes, nor for html conversion jobs. It doesn't do well for my html conversion needs because the tool is incapable of retaining your html formatting (i.e. retain your newlines locations). I do a lot regex based text procesing on my html files, so i need assumptions about how newlines are in my html files. If i use tidy on my site, that means i have to abandon regex based text processing, and instead, have to treat my files using html and dom parsers, which makes most practical text processing needs quite more complex and cumbersome.

* A perl module “HTML::Lint”, at http://search.cpan.org/~petdance/HTML-Lint-2.06/lib/HTML/Lint.pm. Seems pretty much like HTML Tidy.

* http://htmlhelp.com/tools/validator/offline/index.html.en is another validation tool. I haven't looked into yet. Their doc about differences to other validator: http://htmlhelp.com/tools/validator/differences.html.en, is quite interesting, and seems a advantage for my needs.

* OpenJade and OpenSP. http://openjade.sourceforge.net/ Seems a good tool. Haven't looked into.

* Emacs's nxml mode http://www.thaiopensource.com/nxml-mode/, by the xml expert James Clark↗. This is written in elisp with over 10 thousand lines of code. It indicates whether your xml file is valid as you type. This package is very well received, reputed to make emacs the best xml editor. This is fantastic, but since my files are currently html not xhtml, so i haven't used this much. There are emacs html modes based on this package, called nxhtml mode, but the code is still pretty alpha and i find it having a lot problems.

One semi solution for batch validation i found is: “Validator S.A.C.”, at http://habilis.net/validator-sac/. It is basically w3c's validator compiled for OS X with a GUI interface. However, this is not designed for batch operation. If you want to do batch, i run it like this: “/Applications/Validator-SAC.app/Contents/Resources/weblet ‹html file path›”. However, it output a whole report in html on the validation result (same as the page you see in w3c validation). This is not what i want. What i want is simply for it to tell me if a file is valid or not. For any error detail, i can simply load the page in FireFox myself, since if i need to edit it i need to view it in FireFox anyway. So, to fix this problem, you can wrap a perl script, which takes a dir and simply print any file path that's invalid.

Here's the perl script:

# perl

# 2008-06-20 validates a given dir's html files recursively requires the mac os x app Validator-SAC.app at http://habilis.net/validator-sac/ as of 2008-06

use strict; use File::Find;

my $dirPath = q(/Users/xah/web/emacs); my $validator = q(/Applications/Validator-SAC.app/Contents/Resources/weblet);

sub wanted { if ($_ =~ m{\.html$} && not -d $File::Find::name) {

my $output = qx{$validator "$File::Find::name" | head -n 11 | grep 'X-W3C-Validator-Status:'};

if ($output ne qq(X-W3C-Validator-Status: Valid\n)) { print q(Problem: ), $File::Find::name, "\n";

} else {

print qq(Good: $_) ,"\n";

}

}

}

find(\&wanted, $dirPath);

print q(Done.)

However, for some reason, “Validator S.A.C.” took nearly 2 seconds to check each file, in contrast, the FireFox html validator add-on took a fraction of a second while also render the whole page completely. For example, suppose i have 20 files in a dir i need to validate. It's faster, if i just open all of them in FireFox and eyeball the validity indicator, then running the “Validator SAC” on them.

I wrote to its author Chuck Houpt about this. It seems that the validator uses Perl and loads about 20 heavy duty web related perl modules to do its job, and over all is wrapped as a Common Gateway Interface↗. Perhaps there is a way to avoid these wrappers and call the parser or validator directly.

I'm still looking for a fast, batch, html validation tool.