Andornot Consulting
Wednesday, February 24, 2010 3:18 PM

Replace MS Word special characters in javascript and C#

by Peter Tyrrell

MS Word uses characters from the Windows-1252 character encoding set which are not represented in ASCII or ISO-8859-1. This is often a pain in the butt. Special characters include:

  • the… ellipsis
  • ‘smart’ “quotes”
  • en – dash and em — dash
  • dagger † and double dagger ‡
  • and more, but these are most common.

If you want to replace them with ASCII cognates, here's a function to do that. (Daggers don't have cognates as far as I know.)

Javascript

/// Replaces commonly-used Windows 1252 encoded chars that do not exist in ASCII or ISO-8859-1 with ISO-8859-1 cognates.
var replaceWordChars = function(text) {
    var s = text;
    // smart single quotes and apostrophe
    s = s.replace(/[\u2018|\u2019|\u201A]/g, "\'");
    // smart double quotes
    s = s.replace(/[\u201C|\u201D|\u201E]/g, "\"");
    // ellipsis
    s = s.replace(/\u2026/g, "...");
    // dashes
    s = s.replace(/[\u2013|\u2014]/g, "-");
    // circumflex
    s = s.replace(/\u02C6/g, "^");
    // open angle bracket
    s = s.replace(/\u2039/g, "<");
    // close angle bracket
    s = s.replace(/\u203A/g, ">");
    // spaces
    s = s.replace(/[\u02DC|\u00A0]/g, " ");
    
    return s;
}

C# extension method

public static string ReplaceWordChars(this string text)
        {
            var s = text;
            // smart single quotes and apostrophe
            s = Regex.Replace(s, "[\u2018|\u2019|\u201A]", "'");
            // smart double quotes
            s = Regex.Replace(s, "[\u201C|\u201D|\u201E]", "\"");
            // ellipsis
            s = Regex.Replace(s, "\u2026", "...");
            // dashes
            s = Regex.Replace(s, "[\u2013|\u2014]", "-");
            // circumflex
            s = Regex.Replace(s, "\u02C6", "^");
            // open angle bracket
            s = Regex.Replace(s, "\u2039", "<");
            // close angle bracket
            s = Regex.Replace(s, "\u203A", ">");
            // spaces
            s = Regex.Replace(s, "[\u02DC|\u00A0]", " ");
            
            return s;
        }
Friday, February 19, 2010 10:19 AM

ClamAV on Windows 2008 x64 update

by Peter Tyrrell

Further to my previous post which describes how to install and run ClamAV as a service on Windows, here is extra info on what is required for Windows Server 2008 x64, discovered the hard way, through trial and error, by the persistent IT brains at SET-BC (Special Education Technology British Columbia). Particular thanks to IT Manager Gordon Eddy.

Configuring ClamD and FreshClam services

No special configuration is needed for the services. Clam and FreshClam both run fine under Local System, with automatic start and defaults for other service settings. The service account needs read/write access to C:\ClamAV, which it would have by default if Local System is used.

Starting the services

If the ClamD service is started from the services UI and there is no database, or the database is no good, it will fail silently. This could lead one off on a long goose chase looking at service accounts, permissions, service settings, etc. trying to figure out why the service won’t run, when in fact all you need is to run FreshClam once to get a good database, and then ClamD will run fine.

Run FreshClam immediately after installing ClamAV, before trying to run the ClamD service the first time. If FreshClam reports corruption or other problems with the existing database files, delete everything in ..\data and run FreshClam again.

There is no useful error info returned when trying to start ClamD from the services UI. On the other hand, starting it from a command prompt using "NET START CLAMD" does return a bit of useful info. Executing it directly as "clamd.exe --daemon" also returns some (different) info.

Registry setting required

There is a registry file in the ..\docs folder called clamav.reg that contains some path information. The docs seem to be saying that this is not necessary unless you are running Clam from somewhere other than the default location. We found this is not true. Even if installed to C:\ClamAV, Clam will not run without this info in the registry.

TCPAddr setting required

In clamd.config, the comments suggest that the TCPAddr setting is optional. It is not. Set it to the address at which ClamD should respond. If this is missing, the Clamd service will run, but will not respond to requests.

Wednesday, February 17, 2010 11:14 AM

Charting Change Atlas: the technical details

by Peter Tyrrell

The City of Burnaby just launched their Charting Change Atlas, a series of interactive maps showing historical points of interest for the past 100 years. See the news post here. Since I was responsible for its development, I thought I’d talk about some of the technical aspects.

Over 350 points of interest, held as records in an Inmagic database, are associated with four map images. Making those images look and act like current web-based map applications was a bit of a challenge, and it was critical to keep page size down and performance snappy.

Map images

Each map image originated as a 2700 x 3500 TIFF, and each point of interest was located as an X/Y coordinate in pixels relative to the image. These images were not maps, but pictures that looked like maps. That meant we couldn't place points of interest on them via address or latitude/longitude.

The large images were then sliced into PNG tiles by excitable chefs wielding ginsu knives. The tiles are loaded dynamically when they are pulled into the visible viewport to cut down on initial load time. A modified version of the jQuery plugin Lazy Load is used to determine when an unloaded tile has appeared within the viewport.

A map framework

The viewport resizes to fit the browser resolution, whatever it is, regardless of the size of the map within, and reveals only a portion of the map at a time. It contains a very large outer div tag that marks the boundaries of the drag area, and an inner div that contains the map tiles.

The map can be dragged or scrolled in any direction, but the majority of it is hidden outside the viewport bounds. jQuery UI Draggable powers the drag mechanism. The snap-back feature, which pops the map back to the viewport boundary when the edges are pulled in too far, was inspired by my iPhone.

The rest of the map UI is a nod to current web-based map applications - i.e. Google Maps and Bing Maps - in an effort to create an intuitive and comfortable experience for a visitor. We found the more the application looked and acted like these map applications, the less we had to explain to a first-time user. On the other hand, it perhaps worked too well, because users keep expecting the map to zoom! It doesn't zoom because it consists of just the one layer, while Google, Bing and Co. boast multiple vertical layers. Something to work on in future...

Points of interest

The point data are recursively loaded sector by sector upon page load via jQuery AJAX to a .NET web service, which performs the database query and returns the results as a JSON object. This load sequence considerably improves the perception of performance: large amounts of data are pushed over the wire, but they are spread over the first few seconds following page load.

Pins are cloned from a pin template, stuffed with point of interest data and positioned relative to the map container. The dialog that appears when a pin is hovered over or clicked is built from a dialog template bound to pin data on demand. A modified version of John Resig's Micro-Templating function is used for client-side templating and databinding.

All points of interest are held inside a Heritage Landmarks Inmagic CS/Textworks database, already in use at http://heritageburnaby.com/research/Landmarks/, which was extended to hold information specific to the Charting Change Atlas project: query URLs out to other databases, a blurb and primary image, coordinates, and a Google Street View URL.

Conclusion

It was a blast designing and building the Charting Change Atlas, and a privilege to work together with the far-seeing heritage team at the City of Burnaby. Kudos to them for leveraging their existing toolset and informational treasure trove to make history so alive and accessible.

 

Thursday, December 10, 2009 12:49 PM

How to export only files modified since revision x with TortoiseSVN

by Peter Tyrrell

It's not obvious how to get *just* files modified since a given revision when using TortoiseSVN against a Subversion source control repository, and it's often handy to do so. I do it to send client updates, for instance, since there may have been numerous commits since it was last deployed and sometimes I just want the diffs, not the entire project.

So here is how to get only the files modified since a given revision, with folder structure intact.

  1. On top-level folder, right-click and TortoiseSVN --> Show log.
  2. From the list of revisions, select all back to, but not including, the last revision that was deployed (to ensure you get just files changed *since* that revision).
  3. Right-click on the selections and Compare revisions.
  4. Now you're looking at a list of files modified since that last deployment revision. Select all, right-click and Export selection to...
  5. Choose a folder to export to in the folder browser dialog.

Now you have just the files you want, exported and preserved in their folder hierarchy.

Wednesday, October 21, 2009 3:05 PM

Fix IE8 unable to authenticate on a local website using Windows authentication

by Peter Tyrrell

Symptom

IE8 refuses to authenticate on a local website (website and browser are on the same machine) even when valid credentials are supplied, when the website is reached using a host header bound to the machine's loopback address. After a few attempts, the website reports an HTTP 401.1 Access Denied error. A different browser may authenticate successfully. Browsing the website with IE8 from an external client computer authenticates as expected.

Cure

A Windows security update is responsible for a loopback check security feature that is meant to prevent reflection attacks. Authentication fails if the host header does not match the local computer name. Disable the loopback check in the registry:

  1. Run regedit.
  2. Find HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa.
  3. Add a new DWORD value called DisableLoopbackCheck.
  4. Modify the new value data to 1.
  5. Reboot.

Related