Cablechip Solutions

OS Maps and GPS Data

2011-09-09T15:06:00.000-07:00

GPS - the global open standard - use latitude and longitude. to identify a point.

Ordnance Survey Maps use their own systen. The points (e.g. TZ123456), or sometimes given at 6 figure easting and northings (e.g. 123456,123456) while user friendly, and easy to locate on the grid superimposed on OS maps, can't easily be be used by satnavs.

The maps to convert between the 2 systems is rather comples, but we can do it for you..

Character Sets

2008-10-06T07:45:00.000-07:00

Definitions

Character Sets

A character set in a sequence of characters, each character is represented by a number.
e.g. 65=A, 66=B, 67=C, ... 1234=£, ...
Examples of character sets are:
ASCII, ISO 8859-1, Windows 1252, ANSI

Character Encoding

A character encoding is a means of representing a character set in a file.
For ASCII and Windows 1252 (or ANSI), its easy, 1 byte = 1 character.
For large character sets, with more than 256 characters, it is more complex, as more than 1 byte per character is used.
"UTF-8" uses 1, 2, 3 or 4 bytes per character.

Character References in HTML and XML

An entity reference is of the form:
" € ©
A numeric reference (in HTML and XML) for character 256 is of the form:
ÿ (decimal) or ÿ (hexidecimal)

Character Sets

ASCII

ASCII is the original Character Set, with 128 characters defined.
1 byte = 1 character.

ISO 8859-1

This is the ISO "Western European" character set.
It is the original "web" character set, and used as the default by older browsers.
ISO 8859-1 is a subset of the larger UCS/Unicode character set (not quite true, but almost)
It is now depreciated (obsolete) - use UTF-8 instead.

Unicode and UCS (Universal Character Set)

This is a very large character set. It is a combination of the ISO 8859-1 characters,
plus mathematical and other symbols,
plus the Chinese, Hebrew, Japanese, Greek, Thai, Persian and other alphabets.
Some special cases :
- there are spaces reserved for 'user defined' characters,
- some characters can be combined to make composite characters (e.g. Thai)
Unicode/UCS is a character set. It is encoded using UTF-8

Windows 1252 / ANSI Character Set

This is the Windows character set.
It is encoded using 1 bytes (0-255) per character.
From 0-127, its the same as ASCII
Between 0x80 and 0x9F there are differences. This is the problem area, as these character positions are 'not defined' in ISO 8859-1 and UTF-8
From 0xA0 and above, its "the same" as ISO 8859-1.

Character Encoding

UTF-8

UTF-8 is actually a character encoding, not a character set. Colloquially, it is now used to mean "Unicode/UCS with the UTF-8 encoding"
Its a means of using 1, 2 , 3 or 4 bytes to store a very large character set.
ASCII characters (0-127) take up 1 byte, so its backwards compatible.
£, maths symbols, Chinese and Japanese characters take up 2, 3 or 4 bytes
Some Windows editors use a 'BOM', a character at the front of a file to indicate that the file contains UTF-8 encoded characters. (Actually, its a 2 byte character that's illegal in UTF-8). Not part of the spec.

Character Set Conversion Problems

Whatever you've used in the past, UTF-8 (Unicode/UCS), is the thing to aim for. Google, Blogger etc all use it.

From Windows 1252 / ANSI to the ISO character sets

If converting text from a Windows file to a web page in ISO format, you may have to map some 'hi byte' characters, e.g. the euro symbol, as the character numbers will not be the same.
If copy-and-past'ing, windows will take care of the conversion for you.

From ISO 8859-1 to UTF-8/UCS/Unicode

Viewing a ISO 885901 file in a web browser page set to UTF-8 will display any characters greater than 128 as illegal characters
In terms of character sets, the conversion is straight forward, as there are "no" differences you are likely to encounter.
The character encoding is the problem. Example: In ISO 8859-1, character(165) is stored as binary 165. In UTF-8, it should be 2 bytes. The single byte will be an illegal UTF-8 character.
The solution is programing language dependent or editor dependant.
- For example, in the Notepad++ editor, there is a 'convert ANSI to UTF-8' option.
- In perl: $string =~ s/ ( [\x80-\xff] ) / chr($1) /gxe;

UTF-8 to ISO 8859-1

Viewing a UTF-8 file in a web browser page set to ISO 8859-1 will display 2 (or more)characters for each UTF-8 'hi byte' character.
e.g. For 2 byte UTF-8 characters, it will display an illegal character, followed by the character you want.
The solution: First, identify all characters in your input stream, that don't have ISO 8850-1 equivalents
Maybe convert
- all the exotic utf-8 bullet points to &#nn;
- the exotic hyphens to - (minus sign)
- the various 6, 66, 9, 99 style quotes to ' and "
For XML feeds with character codes greater than 255, consider &#nn; escape sequences(rather than &name; or the binary code, both of which will cause problems)

Case Study

A well known international newspaper has a publishing system that uses UTF-8, and a series of XML feeds that use ISO 8859-1

Analyse an entire year's worth or newspaper articles.
- Make a list of every unique characters used.
- Cater for &#nn; and &name; style characters.
Map all the UTF-8 characters found (with character code greater than 128) to ISO 8859-1 equivalents.
Flag up any UTF-8 characters encountered in the conversion process which are not covered by this mapping.
- Again, cater for &#name; and &name; characters.
Escape all characters greater than 128 with the XML &#nn; escape sequence, so the output file is pure ASCII

Appendix : Differences between Windows 1252 and the ISO Character Sets

In practise, you may wish to map characters to ', ", and - rather then "left single quote" etc


0x80 0x20ac ;Euro Sign
0x81 0x0081
0x82 0x201a ;Single Low-9 Quotation Mark
0x83 0x0192 ;Latin Small Letter F With Hook
0x84 0x201e ;Double Low-9 Quotation Mark
0x85 0x2026 ;Horizontal Ellipsis
0x86 0x2020 ;Dagger
0x87 0x2021 ;Double Dagger
0x88 0x02c6 ;Modifier Letter Circumflex Accent
0x89 0x2030 ;Per Mille Sign
0x8a 0x0160 ;Latin Capital Letter S With Caron
0x8b 0x2039 ;Single Left-Pointing Angle Quotation Mark
0x8c 0x0152 ;Latin Capital Ligature Oe
0x8d 0x008d
0x8e 0x017d ;Latin Capital Letter Z With Caron
0x8f 0x008f
0x90 0x0090
0x91 0x2018 ;Left Single Quotation Mark
0x92 0x2019 ;Right Single Quotation Mark
0x93 0x201c ;Left Double Quotation Mark
0x94 0x201d ;Right Double Quotation Mark
0x95 0x2022 ;Bullet
0x96 0x2013 ;En Dash
0x97 0x2014 ;Em Dash
0x98 0x02dc ;Small Tilde
0x99 0x2122 ;Trade Mark Sign
0x9a 0x0161 ;Latin Small Letter S With Caron
0x9b 0x203a ;Single Right-Pointing Angle Quotation Mark
0x9c 0x0153 ;Latin Small Ligature Oe
0x9d 0x009d
0x9e 0x017e ;Latin Small Letter Z With Caron
0x9f 0x0178 ;Latin Capital Letter Y With Diaeresis

How to Install Template Toolkit on Windows

2008-10-06T07:26:00.000-07:00

If using Activestate Perl on Windows XP, Windows Vista etc.:

Have a look at : http://cpan.uwinnipeg.ca/PPMPackages/10xx/
These modules are for Perl 5.10 (only). Use perl -v to check you're using 5.10
run ppm (activestate's perl packet manager)
- can just type ppm at a dos prompt (in vista at least)
In ppm: Edit > Preferences > Add Repository
- name = university of winnipeg
- url = http://cpan.uwinnipeg.ca/PPMPackages/10xx/
- this will then sync with the repository
In ppm: Select template toolkit (may have to do "view > all packages" to see it)
In ppm: File > Run Marked Actions

As a bonus, there's loads of other perl modules as well.

Regex : Tutorial

2008-09-30T06:24:00.000-07:00

Regular expressions, or regex, are one of the best things about Perl. So good infact, that they been copied by most other programming languages.

They are sadly let down however by Perl "man pages" and the chapter in the 'Programming Perl' camel book which in no way does them justice. It explains them, but only in a way a Computer Science PHD student would understand.

So, here's my tutorial with lots of examples.

NB 1: this is based on Perl, but you can use them in Javascript, Python etc.

NB 2: once you understand this, you should be ready for the reference material

Regex : Using a regular expression (perl specific).

2008-09-29T07:24:00.000-07:00

There are 3 ways to call a regex:

Use: $x =~ //
Use: $x =~ m//
or m## or m"" or m{} or anything else you care to choose
Use: qr()
or qr## or qr"" or qr{}

Why so many ways. First, that's Perl. But it helps with escaping, see examples below. Just choose the delimiters that don't appear inside the rege.

The usual choices are // () ## and ""


$x =~ /andrew/;
$x =~ m/andrew/;
$x =~ m#andrew#;
$x =~ qr(andrew);

#matches speachmarks
$x =~ m/"andrew"/;

#matches HTML tags - there's no need to escape the / in </b>
$x =~ qr# <b>andrew</b> #;

Regex : Using whitepace and comments

2008-09-28T07:27:00.000-07:00

The /x modifier

Using whitespace and comments inside your regular expression is a brilliant idea, especially as regex often look like line noise, even to yourself a week later..

If you need a space, you'll need to escape it as \ (that \ then a SPACE ) or \s (which means any whitespace character)

$x =~ / andrew /ix

# these last 2 are the same
$x =~ m# <b> \s* andrew \s* </b> #ix;

$x =~ qr"                 # " is the delimiter to start and end the regex
      <b>
        \s* andrew \s*  # allow optional whitespace (0+ chars) at start and end
      </b>
   "ix;                 # ignore case

Regex : Some simple examples

2008-09-26T06:28:00.000-07:00

Some simple examples to start


$x = 'Andrew';

print 'match' if $x =~ /rew/;
match

print 'match' if $x =~ /And/;
match

# no match as and doesn't match And
print 'false' if $x =~ /and/;

print 'true' if $x =~ /and/i; # the 'i' is ignore case
match

Regex : How many times to match?

2008-09-25T08:41:00.000-07:00

Place these modifiers after a character:

+ matches 1+ times
* matches 0+ times
{3} after a charater matches it 3 times
{2,3} after a charater matches it 2 to 3 times

Greedy and Not Greedy


$x = " one 
  two 
";

#.* is greedy, so this matches from the first  to the last 


$x =~ m#  .* 
 #x

#.*? isn't greedy, so this matches  one  only

$x =~ m#  .*? 
 #x

Regex : Predefined Groups of characters

2008-09-24T08:45:00.000-07:00

Groups of characters

/s matches a whitespace character
/S matches anything except a whitespace character
/w a word charaters (a-z, A-Z, 0-9, Persian, Greek, Chinese alphabets, etc)
/W anything except a word character
/d matches 0..9
/D matches anything EXCEPT 0..9

Regex : Groups

2008-09-23T08:47:00.000-07:00

Use ( ) for groups.


$html =~ (andrew) 
# matches andrew

((andrew)|(john)|(peter)) 
# matches andrew OR john or peter

# the same, but with whhitespace
# m# - use # as the delimiter
# #x - allow whitespace
$html =~ m#
 (
   ( andrew   )  |
   ( peter    )  |
   ( john     )
 )#x;

Perl Complex Variables

2008-09-11T10:12:00.000-07:00

Perl has 3 types, scalar, array and hash. These can be combined to make complex variables.

Unfortunately, the syntax isn't kind.

Defining Scalars, Arrays and Hashes.


$name = 'andrew' ;

# these 2 are the same, qw() is 'syntax sugar'
@names = qw( andrew john peter );
@names = ('andrew', 'john', 'peter');

%age = (
 'andrew' => 20,
 'peter'  => 30,
 'paul'   => 17,
 );

# note the outside hash uses (
# - but the inside hash uses {

%names = (
 'andrew' => { 
      'age'    => 20, 
      'height' => 1.76,
      },
 'peter'  => { 
      'age'    => 30, 
      'height' => 1.2, 
      'friends'=> ['andrew', 'paul'], 
      }
 );

Scalars


$x =  $names{'andrew'}{'age'};
# now $x = 20 

$names{'andrew'}{'age'} = 35 ;

Arrays

Arrays are a bit more complex - use @{ } to access them and [ ] to set them


@x = @{ $names{'peter'}{'friends'} } ;

$names{'peter'}{'friends'} = [ 'bill', 'john', 'fred' ] ;

Hashes

Hashes are similar, use %{ } to access them and { } to set them


%hash = %{ $names{'andrew'} } ;

$names{'andrew'} = { 'age' => 20, 'height' => 1.76 } ;

Important Gotcha

When using the %hash = %{} and @array = @{} syntax, you are making a reference to the data, NOT a copy of it;


%hash = %{ $names{'andrew'} } ;
# %hash is a reference, change %hash and you change %names, and visa versa

%new_hash = %hash;
# now safe

Regex : Doesn't match

2008-09-11T09:58:00.000-07:00


# $x does not contain andrew
$x !~ /andrew/

Regex : Match on word Boundary

2008-09-11T09:57:00.000-07:00

\b match on a word boundary

\B matches on not a word boundary

Regex : . and the /s modifier

2008-09-11T09:44:00.000-07:00

A . matches anything except a newline

To match a newline, use the /s modifer (treat as a Single line)


$x = qq(
andrew
was
here
);

$x =~ /andrew .* here/xs;
# matches as the /s carries the search over the newlines

Regex : Anchors and the /m modifier

2008-09-11T09:24:00.000-07:00

Anchors
^ matches at the start of a string
$ matches at the end of a string


$x = /andrew/

$x =~ /^a/ ;
# matches as andrew starts with an a

$x =~ /rew$/;
# matches as andrew ends with a rew

But what if you have a string containing several lines (e.g. a file with \n characters) ?

The /m modifier.

This treats the string as multiple lines, so ^ and $ will match the start and end of each line in the file


$x = qq(
andrew
john andrew
andrew
);
$x =~ s/^andrew/ANDREW/mg;
# will replace 1st and 3rd andrew
# without the /m, it would only replace the third
This treats the string as a single line, so

Regex : Global Replace

2008-09-11T09:21:00.001-07:00

The /g modifier

This repeats the regex until there are no more matches


$x = /andrew andrew andrew/;

$x = s/a/A/;
# Andrew andrew andrew (1 replace)

$x = s/a/A/g;
# Andrew Andrew Andrew (all the 'a' get replaces)

Regex : Using expressions

2008-09-11T08:54:00.001-07:00

This is the /e modifier

Example: make HTML tags upper case


# ( ) - capture the text
# .*? - match the least amount of text, i.e. an HTML tag
# uc ($1) - make the matches text uppercase
# gxe - g is global replace
# gxe - x is allow whitespace and comments
# gxe - the bit on the right is an expressions to be evaluates

$html =~ s# ( < .*? > ) # print uc($1) #gxe ;

Example : find hi byte characters


# find hi-byte characters in HTML
# - and keep a record of all the hi byte chars found in %found
my %found ;

sub find {
 my ($char) = @_ ;
 $found{ $chars } ++; ## keep a record
 return '[' . ord( $chars ) . ']';
 }

## this will find euro and £ symbols, but not €
## \x80 is hex (ascii char 128)
## \x80-\xffff is a range - if the file is utf8, it will match hi bytes chars as well
## gxe - expression, whitespace, and global replace
$html =~ s# [\x80-\xffff] # &find( $1 ) #gxe;

Regex : Substitutions

2008-09-11T08:53:00.000-07:00

Regex : Capture Matched text

2008-09-11T08:52:00.000-07:00

Use brackets to capture text, and $1, $2 ... to refer to it


$x = 'andrew john';

# swap the first and second word around

$x=~ s/ (\w+) \s+ (\w+*)
      / $2 $1
      / x;

# s/    - substitute
# (\w+) - 1+ word charcters, captures as $1
# \s+   - 1+ whitespace characters
# (\w+) - 1+ word chars, captured as $2
# /x    - use whitesapce and comments

# $1 and $2 survive
print $2;
# prints john

Regex: Ranges

2008-09-11T08:51:00.000-07:00

Use [ ] for ranges - match any 1 char inside the range


$x =~ /[0-9]/;
$x =~ /[a-z]/i;   # so match a..z and A..Z as /i is ignore case
$x =~ /[\n\r]/;

#match 1+ chars in the range 0..9 + - and .
#\. escape the dot so it doesn't mean "any 1 char"
$x=~ /[0-9+-\.]+/

Perl Poetry

2008-09-11T06:20:00.000-07:00

This isn't mine, but its quite cool.

$_ = reverse sort qw p ekca lre uJ reh ts
p, $/.r, map $_.$", qw e p h tona e; print