Commit 3f5be827 authored by matsduf's avatar matsduf
Browse files

Merge pull request #1 from mattias-p/v1.0.0

Initial import
parents b0ef32c2 b18233e3
Copyright (c) 2015 IIS (The Internet Foundation In Sweden).
All rights reserved.
The Whois Selftest Tool and the use hereof is subject to the following licensing
conditions.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THE SOFTWARE IS PROVIDED AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
ANY KIND CONCERNING THE WORK, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE,
INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS
FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER
DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT
DISCOVERABLE.
IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
OF THE POSSIBILITY OF SUCH DAMAGE.
# Whois Selfttest Tool
Purpose
=======
The purpose of the Whois Selftest Tool is to help gTLD applicants prepare for
[Pre-Delegation Testing]( http://newgtlds.icann.org/en/applicants/pdt) (PDT) by
providing pre-PDT Whois output validation.
The Whois Selftest Tool is planned to be released on Thursday 2015-12-03 at 17:00 UTC.
It will then be available for download.
Scope
=====
While Whois Selftest Tool _does_ validate Whois output and it _does_ strive to
reflect the state of PDT Whois output validation, it _is not_ authoritative on
the outcome of PDT and it _is_ subject to change.
Disclaimer
----------
The Whois Selftest Tool and the actual Whois testing under PDT are not equal.
We strive to make the two as equal as possible, but here is no guarantee that
successfully running the Whois Selftest Tool means
that the same Whois system will pass the Whois testing under PDT. For example,
the parts of Whois tests under Whois that include DNS lookups and TCP
connections are not included in the Whois Selftest Tool. For a complete
reference of the Whois tests under PDT see the PDT Whois documents.
Version history
===============
* v1.0.0 - Initial public release (2015-12-03)
Specification compatibility matrix
----------------------------------
Refer to this compatibility matrix when deciding which version of Whois Selftest
Tool to use.
<table>
<tr><th>Whois Selftest Tool version</th><th>PDT Test Specifications</th></tr>
<tr><td>v1.0.0</td><td>v.2.8</td></tr>
</table>
Roadmap
=======
The plan is to solve know issues and any bugs of importance before the
stricter Whois testing is enforced at 2016-01-31. New versions will be released
when fixes are stable.
References
==========
The [Pre-Delegation Testing]( http://newgtlds.icann.org/en/applicants/pdt)
microsite hosts the following documents relevant to the Whois Selftest Tool:
* The PDT\_Whois\_TC\_CLI and PDT\_Whois\_TC\_Web documents, within the PDT Test
Specifications zip, specifies the test cases that the Whois Selftest Tool
partially implements.
* The PDT\_Whois\_TP document, within the PDT Test Specifications zip, specifies
the format specification that the Whois Selftest Tool implements.
In the PDT\_Whois\_TP you can find references to other useful documents.
Licensing
=========
Whois Selftest Tool is distributed under the terms of [this license]( LICENSE).
Dependencies
============
* Ubuntu Linux version 12.04
* Perl, version 5.14 or higher
* Standard Perl libraries found on CPAN.org
* DateTime
* File::Slurp
* File::Which
* Net::IDN::Encode
* Net::IP
* Readonly
* Regexp::IPv6
* Test::Differences
* Test::MockObject
* Text::CSV
* Text::CSV\_XS
* URI
* YAML::Syck
* wget
The Whois Selftest Tool has been developed on Unbuntu Linux, but we have tried to
avoid Linux specific coding. There is, however, no guarantee that it works on
other OSs.
Installation
============
Clone the project repository and choose version according to the specification
compatibility matrix.
$> git clone https://github.com/dotse/Whois-Selftest-Tool.git <installdir>
$> cd <installdir>
$> git checkout <version>
`<installdir>` is assumed to be in the PATH in code examples throughout the
rest of this document.
Create a program directory `<programdir>` to install Whois Selftest Tool in,
and copy the scripts and libraries there.
$> cd <somewhere>
$> mkdir <programdir>
$> cp <installdir>/script/* <programdir>/
$> cp -r <installdir>/lib/PDT <programdir>/
Now the script is installed and can be run from `<programdir>`. To check the
installation run the scripts with `--help`.
$> cd <programdir>
$> ./whois-test --help
$> ./whois-fetch-epp-repo-ids --help
Before use
==========
Before you use the tool, make sure that you have read the documents listed
in the reference above. Some error messages may be difficult to understand
without referring to the PDT\_Whois\_TP document.
Usage
=====
The Whois Selftest Tool provides the two commands `whois-fetch-epp-repo-ids`
and `whois-test`. If you have followed the installation above, always go to
your `<programdir>` and run the commands from there or else the scripts will
not be able to find its libraries in the `PDT` directory. You probably have to
prepend the commands with `./` just as in the instructions above.
`whois-fetch-epp-repo-ids` fetches the EPP Repository Identifiers registered
with IANA and stores them in a text file inside the user's home directory.
`whois-test` validates the Whois responses according to the format specification
in the PDT\_Whois\_TP document.
The database of EPP Repository Identifiers is a prerequisite for running
`whois-test` command, so `whois-fetch-epp-repo-ids` must be run at least once
before `whois-test` is used for the first time. After that, run
`whois-fetch-epp-repo-ids` again to update the database every time
the Whois Selfttest Tool is to be used.
See the man pages for the respective commands for details on how to run them.
(You can use the `--man` option to view the man pages)
Known issues
============
* The Perl library that converts between IDN U-label and IDN A-label,
Net::IDN::encode, will magically make upper-case characters into its
equivalent lower-case characters. The upper-case characters are not valid
in a U-label. This issue can make false positive validations of Whois
responses when the "Internationalized Domain Name" field is present in a
Domain Object response.
* Perl 5.14 only supports Unicode 6.0.0. If an "Internationalized Domain Name"
field contains code points available in Unicode 6.3.0 but not in Unicode 6.0.0, and valid for IDNA,
then they will incorrectly be reported as invalid.
Reporting bugs
--------------
If you think you've found a bug, please search both the list of known issues and
the [issue tracker](https://github.com/dotse/Whois-Selftest-Tool/issues) to see
if this is a known bug. If you cannot find it, please report it to the issue
tracker.
package PDT::TS::Whois;
use strict;
use warnings;
use 5.014;
=pod
=encoding utf8
=head1 NAME
PDT::TS::Whois - Validates Whois output
=head1 DESCRIPTION
This module validates Whois output strings according to the ICANN specification.
It consists of the following sub-modules:
=over 4
=item L<PDT::TS::Whois::Lexer>
Takes a string and produces a token/value/errors triplet for each line.
=item L<PDT::TS::Whois::Grammar>
Exports a datastructure representing the ICANN specification.
=item L<PDT::TS::Whois::Types>
Type checker providing most rules required by the ICANN specification and a means for the user to go the last mile.
=item L<PDT::TS::Whois::Util>
Various utility functions.
=item L<PDT::TS::Whois::UnicodeData>
Character property tests for Unicode 3.1.0.
=item L<PDT::TS::Whois::Validator>
Validates the output of a lexer according to a grammar and types.
=back
=head1 VERSION
Version 1.01
=cut
our $VERSION = '1.01';
=head1 SYNOPSIS
use PDT::TS::Whois::Grammar qw( $grammar );
use PDT::TS::Whois::Lexer;
use PDT::TS::Whois::Types;
use PDT::TS::Whois::Validator qw( validate );
my $types = PDT::TS::Whois::Types->new();
$types->add_type( 'query domain name', sub { return ( lc( shift ) ne 'domain.example' ) && ( 'expected exact domain name' ) || () } );
$types->add_type( 'query name server', sub { return ( lc( shift ) ne 'ns1.domain.example' ) && ( 'expected exact name server' ) || () } );
$types->add_type( 'query registrar name', sub { return ( shift !~ /Example Registrar/ ) && ( 'expected exact registrar name' ) || () } );
my $lexer = PDT::TS::Whois::Lexer->new( `whois domain.example` );
my @errors = validate(
rule => 'Domain Name Object query',
lexer => $lexer,
grammar => $grammar,
types => $types,
);
say join( "\n", @errors ) || "OK";
=head1 AUTHOR
Mattias Päivärinta, <mattias.paivarinta@doxwork.com>
=head1 LICENSE AND COPYRIGHT
Copyright (C) 2015 IIS (The Internet Infrastructure Foundation).
All rights reserved.
This module is subject to the following licensing conditions.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THE SOFTWARE IS PROVIDED AS-IS AND MAKES NO REPRESENTATIONS OR
WARRANTIES OF ANY KIND CONCERNING THE WORK, EXPRESS, IMPLIED,
STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF
TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE,
NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY,
OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE.
IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
=cut
1; # End of PDT::TS::Whois
package PDT::TS::Whois::Grammar;
use strict;
use warnings;
use 5.014;
use YAML::Syck;
=head1 NAME
PDT::TS::Whois::Grammar - A data representation of the ICANN Whois specification
=cut
require Exporter;
our @ISA = 'Exporter';
our @EXPORT_OK = qw( $grammar );
=head1 EXPORTS
=head2 $grammar
A set of rules represented by a HASHREF. Each key-value-pair in the HASHREF
represents a rule.
Rules are represented by key-value-pairs where the key (string) is the rule
name.
There are two types of rule values:
=over 4
=item Sequence rule
Represented by an ARRAYREF. Each element represents a subrule using a single-
key-value-pair HASHREF.
=item Choice rule
Represented by a HASHREF. Each key-value-pair represents a subrule.
=back
Subrules are represented by key-value-pairs where the key (string) is the
subrule name and the value (HASHREF).
There are two types of subrules:
=over 4
=item Line rules
Represented by a HASHREF with a 'line' key.
=item Section rules
Represented by a HASHREF without a 'line' key.
=back
A subrule value HASHREF may have the following keys:
=over 4
=item optional
Values: 'y'|'n' (default: n)
=item repeatable
Values: non-negative integer|'unbounded' (default: 1)
=item line
Values: 'any line'|'awip line'|'empty line'|'field'|'last update line'|'multiple
name servers line'|'non-empty line'|'roid line'.
'any line' is a special wildcard value matching any one of the other line types.
=item type
Values: string
Specifies the type name that the field value must match. Only applicable if this
subrule contains line => 'field'.
=back
=cut
our $grammar = LoadFile( *DATA );
__DATA__
---
Domain Name Object query:
- Domain name reply: { }
Name Server Object query:
Name server reply type 1: { }
Name server reply type 2: { }
Registrar Object query:
- Registrar reply: { }
Domain name reply:
- Domain name details section: { }
- Domain name subsection 1: { optional: y }
- Empty line: { repeatable: 3, line: empty line }
- AWIP footer: { }
- Legal disclaimer: { }
- EOF: { line: EOF }
Domain name subsection 1:
- Empty line: { line: empty line }
- Domain name subsection 2: { }
Domain name subsection 2:
Domain name subsection 3: { }
Empty line: { line: empty line, optional: y, repeatable: 2 }
Domain name subsection 3:
- Subsequent domain name details section: { }
- Domain name subsection 1: { optional: y }
Domain name subsection 1:
Last updated footer: { }
Domain name subsection 2: { }
Domain name subsection 2:
- Empty line: { line: empty line }
- Domain name subsection 3: { }
Domain name subsection 3:
Domain name subsection 4: { }
Last updated subsection 1: { }
Domain name subsection 4:
- Subsequent domain name details section: { }
- Domain name subsection 5: { }
Domain name subsection 5:
Last updated footer: { }
Domain name subsection 2: { }
Registrar reply:
- Registrar details section: { }
- Registrar subsection 1: { optional: y }
- Empty line: { repeatable: 3, line: empty line }
- AWIP footer: { optional: y }
- Legal disclaimer: { }
- EOF: { line: EOF }
Registrar subsection 1:
Last updated footer: { }
Registrar subsection 2: { }
Registrar subsection 2:
- Empty line: { line: empty line }
- Registrar subsection 3: { }
Registrar subsection 3:
Registrar subsection 4: { }
Last updated subsection 1: { }
Registrar subsection 4:
- Subsequent registrar details section: { }
- Registrar subsection 5: { }
Registrar subsection 5:
Last updated footer: { }
Registrar subsection 2: { }
Name server reply type 1:
- Name server details section: { }
- Name server subsection 1: { optional: y }
- Empty line: { repeatable: 3, line: empty line }
- AWIP footer: { optional: y }
- Legal disclaimer: { }
- EOF: { line: EOF }
Name server subsection 1:
- Empty line: { line: empty line }
- Name server subsection 2: { }
Name server subsection 2:
Name server subsection 3: { }
Empty line: { line: empty line, optional: y, repeatable: 2 }
Name server subsection 3:
- Subsequent name server details section: { }
- Name server subsection 1: { optional: y }
Name server subsection 1:
Last updated footer: { }
Name server subsection 2: { }
Name server subsection 2:
- Empty line: { line: empty line }
- Name server subsection 3: { }
Name server subsection 3:
Name server subsection 4: { }
Last updated subsection 1: { }
Name server subsection 4:
- Subsequent name server details section: { }
- Name server subsection 5: { }
Name server subsection 5:
Last updated footer: { }
Name server subsection 2: { }
Name server reply type 2:
- Multiple name servers section: { }
- Empty line: { optional: y, repeatable: 3, line: empty line }
- Last updated footer: { }
- Empty line: { repeatable: 3, line: empty line }
- AWIP footer: { optional: y }
- Legal disclaimer: { }
- EOF: { line: EOF }
Subsequent registrar details section:
- Registrar details section: { }
Registrar details section:
- Registrar Name: { line: field, type: query registrar name }
- Street: { line: field, type: postal line }
- City: { line: field, type: postal line }
- State/Province: { optional: y, line: field, type: postal line }
- Postal Code: { optional: y, line: field, type: postal code }
- Country: { line: field, type: country code }
- Phone Number: { line: field, type: phone number }
- Phone Ext: { optional: y, line: field, type: token }
- Fax number section: { optional: y }
- Email: { line: field, type: email address }
- WHOIS Server: { optional: y, line: field, type: hostname }
- Referral URL: { line: field, type: http url }
- Admin contact section: { optional: y, repeatable: unbounded }
- Technical contact section: { optional: y, repeatable: unbounded }
Admin contact section:
- Admin Contact: { line: field, type: postal line }
- Phone Number: { line: field, type: phone number }
- Phone Ext: { optional: y, line: field, type: token }
- Fax number section: { optional: y }
- Email: { line: field, type: email address }
Technical contact section:
- Technical Contact: { line: field, type: postal line }
- Phone Number: { line: field, type: phone number }
- Phone Ext: { optional: y, line: field, type: token }
- Fax number section: { optional: y }
- Email: { line: field, type: email address }
Fax number section:
- Fax Number: { line: field, type: phone number }
- Fax Ext: { optional: y, line: field, type: token }
Subsequent domain name details section:
- Domain name details section: { }
Domain name details section:
- Domain Name: { line: field, type: query domain name }
- Internationalized Domain Name: { optional: y, line: field, type: u-label }
- Domain ID: { line: field, type: epp repo id }
- WHOIS Server: { optional: y, line: field, type: hostname }
- Referral URL: { line: field, type: http url }
- Updated Date: { optional: y, line: field, type: time stamp }
- Creation Date: { line: field, type: time stamp }
- Registry Expiry Date: { line: field, type: time stamp }
- Sponsoring Registrar: { line: field, type: token }
- Sponsoring Registrar IANA ID: { line: field, type: positive integer }
- Domain Status: { repeatable: unbounded, line: field, type: domain status }
- Registrant ID: { line: field, type: roid }
- Registrant Name: { line: field, type: postal line }
- Registrant Organization: { optional: y, line: field, type: postal line }
- Registrant Street: { repeatable: 3, line: field, type: postal line }
- Registrant City: { line: field, type: postal line }
- Registrant State/Province: { optional: y, line: field, type: postal line }
- Registrant Postal Code: { optional: y, line: field, type: postal code }
- Registrant Country: { line: field, type: country code }
- Registrant Phone: { line: field, type: phone number }
- Registrant Phone Ext: { optional: y, line: field, type: token }
- Registrant Fax: { optional: y, line: field, type: phone number }
- Registrant Fax Ext: { optional: y, line: field, type: token }
- Registrant Email: { line: field, type: email address }
- Admin ID: { line: field, type: roid }
- Admin Name: { line: field, type: postal line }
- Admin Organization: { optional: y, line: field, type: postal line }
- Admin Street: { repeatable: 3, line: field, type: postal line }
- Admin City: { line: field, type: postal line }
- Admin State/Province: { optional: y, line: field, type: postal line }
- Admin Postal Code: { optional: y, line: field, type: postal code }
- Admin Country: { line: field, type: country code }
- Admin Phone: { line: field, type: phone number }
- Admin Phone Ext: { optional: y, line: field, type: token }
- Admin Fax: { optional: y, line: field, type: phone number }
- Admin Fax Ext: { optional: y, line: field, type: token }
- Admin Email: { line: field, type: email address }
- Tech ID: { line: field, type: roid }
- Tech Name: { line: field, type: postal line }
- Tech Organization: { optional: y, line: field, type: postal line }
- Tech Street: { repeatable: 3, line: field, type: postal line }
- Tech City: { line: field, type: postal line }
- Tech State/Province: { optional: y, line: field, type: postal line }
- Tech Postal Code: { optional: y, line: field, type: postal code }
- Tech Country: { line: field, type: country code }
- Tech Phone: { line: field, type: phone number }
- Tech Phone Ext: { optional: y, line: field, type: token }
- Tech Fax: { optional: y, line: field, type: phone number }
- Tech Fax Ext: { optional: y, line: field, type: token }
- Tech Email: { line: field, type: email address }
- Billing ID: { optional: y, line: field, type: roid }
- Billing Name: { optional: y, line: field, type: postal line }
- Billing Organization: { optional: y, line: field, type: postal line }
- Billing Street: { optional: y, repeatable: 3, line: field, type: postal line }
- Billing City: { optional: y, line: field, type: postal line }
- Billing State/Province: { optional: y, line: field, type: postal line }
- Billing Postal Code: { optional: y, line: field, type: postal code }
- Billing Country: { optional: y, line: field, type: country code }
- Billing Phone: { optional: y, line: field, type: phone number }
- Billing Phone Ext: { optional: y, line: field, type: token }
- Billing Fax: { optional: y, line: field, type: phone number }
- Billing Fax Ext: { optional: y, line: field, type: token }
- Billing Email: { optional: y, line: field, type: email address }
- Name server section: { optional: y, repeatable: unbounded }
- DNSSEC: { line: field, type: dnssec }
- Additional fields section: { optional: y }
Name server section:
- Name Server: { line: field, type: hostname }
- IP Address: { optional: y, repeatable: unbounded, line: field, type: ip address }
Multiple name servers section:
- Multiple name servers line: { line: multiple name servers line }
- ROID line: { line: roid line }
- ROID line: { line: roid line, repeatable: unbounded }
Subsequent name server details section:
- Name server details section: { }
Name server details section:
- Server Name: { line: field, type: query name server }
- IP Address: { repeatable: unbounded, line: field, type: query name server ip }
- Registrar: { optional: y, line: field, type: postal line }
- WHOIS Server: { optional: y, line: field, type: hostname }
- Referral URL: { optional: y, line: field, type: http url }
Additional fields section:
- Additional field: { repeatable: unbounded, line: field }
Last updated subsection 1:
Last updated footer: { }
Last updated subsection 2: { }
Last updated subsection 2:
- Empty line: { repeatable: 2, line: empty line }
- Last updated footer: { }
Last updated footer:
- Last update line: { line: last update line }
AWIP footer:
- AWIP line: { line: awip line }
- Empty line: { repeatable: 3, line: empty line }
Legal disclaimer:
- Non-empty line: { line: non-empty line }
- Any line: { optional: y, repeatable: unbounded, line: any line }
package PDT::TS::Whois::Lexer;
use strict;