Charting Change Atlas: the technical details

by Peter Tyrrell Wednesday, February 17, 2010 11:14 AM

The City of Burnaby just launched their Charting Change Atlas, a series of interactive maps showing historical points of interest for the past 100 years. See the news post here. Since I was responsible for its development, I thought I’d talk about some of the technical aspects.

Over 350 points of interest, held as records in an Inmagic database, are associated with four map images. Making those images look and act like current web-based map applications was a bit of a challenge, and it was critical to keep page size down and performance snappy.

Map images

Each map image originated as a 2700 x 3500 TIFF, and each point of interest was located as an X/Y coordinate in pixels relative to the image. These images were not maps, but pictures that looked like maps. That meant we couldn't place points of interest on them via address or latitude/longitude.

The large images were then sliced into PNG tiles by excitable chefs wielding ginsu knives. The tiles are loaded dynamically when they are pulled into the visible viewport to cut down on initial load time. A modified version of the jQuery plugin Lazy Load is used to determine when an unloaded tile has appeared within the viewport.

A map framework

The viewport resizes to fit the browser resolution, whatever it is, regardless of the size of the map within, and reveals only a portion of the map at a time. It contains a very large outer div tag that marks the boundaries of the drag area, and an inner div that contains the map tiles.

The map can be dragged or scrolled in any direction, but the majority of it is hidden outside the viewport bounds. jQuery UI Draggable powers the drag mechanism. The snap-back feature, which pops the map back to the viewport boundary when the edges are pulled in too far, was inspired by my iPhone.

The rest of the map UI is a nod to current web-based map applications - i.e. Google Maps and Bing Maps - in an effort to create an intuitive and comfortable experience for a visitor. We found the more the application looked and acted like these map applications, the less we had to explain to a first-time user. On the other hand, it perhaps worked too well, because users keep expecting the map to zoom! It doesn't zoom because it consists of just the one layer, while Google, Bing and Co. boast multiple vertical layers. Something to work on in future...

Points of interest

The point data are recursively loaded sector by sector upon page load via jQuery AJAX to a .NET web service, which performs the database query and returns the results as a JSON object. This load sequence considerably improves the perception of performance: large amounts of data are pushed over the wire, but they are spread over the first few seconds following page load.

Pins are cloned from a pin template, stuffed with point of interest data and positioned relative to the map container. The dialog that appears when a pin is hovered over or clicked is built from a dialog template bound to pin data on demand. A modified version of John Resig's Micro-Templating function is used for client-side templating and databinding.

All points of interest are held inside a Heritage Landmarks Inmagic CS/Textworks database, already in use at http://heritageburnaby.com/research/Landmarks/, which was extended to hold information specific to the Charting Change Atlas project: query URLs out to other databases, a blurb and primary image, coordinates, and a Google Street View URL.

Conclusion

It was a blast designing and building the Charting Change Atlas, and a privilege to work together with the far-seeing heritage team at the City of Burnaby. Kudos to them for leveraging their existing toolset and informational treasure trove to make history so alive and accessible.

 

How to export only files modified since revision x with TortoiseSVN

by Peter Tyrrell Thursday, December 10, 2009 12:49 PM

It's not obvious how to get *just* files modified since a given revision when using TortoiseSVN against a Subversion source control repository, and it's often handy to do so. I do it to send client updates, for instance, since there may have been numerous commits since it was last deployed and sometimes I just want the diffs, not the entire project.

So here is how to get only the files modified since a given revision, with folder structure intact.

  1. On top-level folder, right-click and TortoiseSVN --> Show log.
  2. From the list of revisions, select all back to, but not including, the last revision that was deployed (to ensure you get just files changed *since* that revision).
  3. Right-click on the selections and Compare revisions.
  4. Now you're looking at a list of files modified since that last deployment revision. Select all, right-click and Export selection to...
  5. Choose a folder to export to in the folder browser dialog.

Now you have just the files you want, exported and preserved in their folder hierarchy.

Tags: svn | TortoiseSVN

Fix IE8 unable to authenticate on a local website using Windows authentication

by Peter Tyrrell Wednesday, October 21, 2009 3:05 PM

Symptom

IE8 refuses to authenticate on a local website (website and browser are on the same machine) even when valid credentials are supplied, when the website is reached using a host header bound to the machine's loopback address. After a few attempts, the website reports an HTTP 401.1 Access Denied error. A different browser may authenticate successfully. Browsing the website with IE8 from an external client computer authenticates as expected.

Cure

A Windows security update is responsible for a loopback check security feature that is meant to prevent reflection attacks. Authentication fails if the host header does not match the local computer name. Disable the loopback check in the registry:

  1. Run regedit.
  2. Find HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa.
  3. Add a new DWORD value called DisableLoopbackCheck.
  4. Modify the new value data to 1.
  5. Reboot.

Related

Tags:

How to enable Windows 7 single sign-on for a website using Windows authentication

by Peter Tyrrell Wednesday, October 21, 2009 2:43 PM

Symptom

You visit an intranet website that uses Windows authentication with an IE browser on Windows 7 and expect to be logged in automatically with your current domain credentials. Instead, you are challenged for credentials. Once you enter credentials everything acts normally, but you expect a single sign-in scenario and it doesn't happen. If you visit the same URL with IE on another Windows OS within the domain, single sign-on works as expected.

Cure

A default local security policy in Windows 7 prevents LM and NTLM responses.

  1. Go to Local Security Policy > Security Settings > Local Policies > Security Options
  2. Select Network security: LAN Manager authentication level
  3. Change security setting to Send LM & NTLM responses

win7_secpol

Related

Tags: Windows 7

How to set up ClamAV as a Windows Service to scan file streams on demand

by Peter Tyrrell Thursday, October 15, 2009 9:44 PM

Overview

Clam Antivirus, or ClamAV for short, is an open-source antivirus solution for UNIX. It's also the ONLY open-source antivirus solution. Naturally it would nice to have it for Windows, too, and it's absolutely possible; the hard part is finding out how to go about it.

Documentation on installing ClamAV for Windows and running its daemon clamd as a Windows Service is as scarce as hen's teeth, and the best information I could find was scattered across various unrelated forums and articles and none of it was fully up to date. So here it is, laid out as best as I can tell it, in one place: where to find a current native Win32 port of ClamAV, how to install it, how to set up clamd to run as a service, and the settings relevant to scanning file streams sent from an external client.

To give some context: I am using ClamAV to scan user-uploaded files for viruses in an ASP.NET web application, before writing them to disk. Each upload is sent as a stream from the webserver to another Windows 2003 server running the clamd service, which scans them and replies with some basic info on the outcome of the scan.

UPDATE Feb 19 2010: Extra info for installing on Windows Server 2008 x64.

Get the current version of ClamAV for Windows

Get ClamAV for Windows 0.95.2 at http://hideout.ath.cx/clamav/.

I am writing on October 15, 2009 and the current version of Clam AntiVirus is 0.95.2. The best 0.95.2 native Windows port available, in my opinion - because it has an installer - is at http://hideout.ath.cx/clamav/.

Another 0.95.2 alternative is at http://oss.netfarm.it/clamav/, but does not have an installer and requires you grab an additional assembly from Microsoft. On the other hand, it offers a 64 bit build.

You don't want ClamWin, which is a GUI aimed at the desktop crowd, and doesn't include clamd. Or it doesn't appear to right now. It might later. Maybe. It's hard to tell. See hen's teeth, above. Nor do you want the former native Win32 port at http://w32.clamav.net/ which is dead, dead, dead at version 0.92.1. Nor the Cygwin version, nor the Interix version. No.

UPDATE Dec 14 2010: Commenter J. Moore reports that MS Security Essentials reckons the hideout.ath.cx version contains a hacking tool Hideproc.c within chp.exe. The distribution from oss.netfarm.it does not contain chp.exe and doesn't throw any virus alerts.

Install ClamAV for Windows

Run the installer, ClamAV-095-2.exe. It is best to let the installer deploy to the default location, which is c:\clamav\, or you'll have to manually modify a bunch of clamav settings files afterwards.

Install clamd as a service

  1. Open a command prompt.
  2. Change directory to c:\clamav\ or wherever you installed clamav.
  3. Run clamd.exe --install
  4. Open services.msc and edit the newly installed "ClamWin Free Antivirus Scanner Service" to start Automatically and/or use credentials other than the local system account, etc.

clamav_cmd

Um, yes, that was easy.

The astute will note that the service names itself "ClamWin yada yada." Again, clamd is NOT included with ClamWin, so your guess is as good as mine as to the connection here, but there clearly *is* some connection, the true nature of which I have been unable to fathom.

You may find, if you Google for "clamd windows service", various helpful threads or articles describing how to forcibly set up clamd as a service with instsrv.exe and srvany.exe from the Windows 2003 Resource Kit Tools. And that works - but it's not necessary. I can only assume that the ClamAV Windows ports came out with a built-in "install as service" feature at some point fairly recently, but the word just hasn't got around.

I'll get to the settings for clamd next, but you need an up-to-date antivirus library first.

Schedule freshclam to update frequently

Freshclam fetches antivirus library updates. You need to run it right away to get the initial database up and going - just double-click freshclam.exe. After that, you can install it as a service in the same manner as clamd above. Or create a batch file that runs freshclam.exe on a schedule with Windows Task Scheduler. It has its own settings file: freshclam.conf.

Settings for on-demand stream scanning

Settings for clamd are in clamd.conf, and there are quite a few, but the ones most relevant for on-demand stream scanning are:

  • TCPAddr - server IP address
  • TCPSocket - port clamd will listen on, default 3310
  • StreamMaxLength - maximum size of stream to be scanned, in megabytes

Create a Windows firewall exception

The lazy way out here is to add clamd.exe as an exception to Windows firewall. Read on if you like.

You can exception just the port clamd listens on (default 3310), but a call to its STREAM method generates a reply on a random port with a range you can set in clamd.conf, which would mean you would have to add exceptions for the entire range. To problematize still further, STREAM has been deprecated in favour of INSTREAM, which sticks to the port you called it on in the first place. For now I'm just going to exception clamd.exe, though I did restrict its scope to the local subnet.

Call clamd from ASP.NET

I've found just one available .NET library that will call a clamd service, called WRAVLib: http://www.wolfereiter.com/antivirus.aspx. Unfortunately, it's somewhat out of date and written for .NET 1.1, but happily the source code is freely available, so you can compile for .NET 3.5 if you like. Direct link to source code is here: http://www.wolfereiter.com/Downloads/wravlib/wravlib-1.1-src.zip.

It does target the deprecated STREAM method instead of INSTREAM, which I touched on above, but it's still the fastest way to get up and running. Here's a bit of pseudocode to give an idea:

// create scan agent
IVirusScanAgent agent = new ClamdStreamAgent("127.0.0.1", 3310, false);
 
// create unique scan id
string scanId = Guid.NewGuid().ToString();
 
agent.VirusFound += ((sender, args) => {
   // do something
});
 
agent.ItemScanCompleted += ((sender, args) =>  {
   // do something
});                                  
                                    
// scan filestream
agent.Scan(scanId, file.InputStream);

Conclusion

I wish I had more time to fill in the gaps and provide more detail, but I just don't. Even this amount of information took way too long to gather in the first place! It should provide a good foundation nevertheless.

Acknowledgements 

  • This how-to on setting up ClamAV for Kerio MailServer on Windows convinced me I could run clamd as a service in the first place.
  • This forum thread on installing ClamAV Win32 with hMailserver was instrumental, guiding me with sample settings files and good pointers.

Tags: ClamAV

Month List