Archive for the 'Construct internals' Category

Speaking Construct Protocols

Published
by
graeme
on July 9, 2008
in Construct internals
. 0 Comments

All messages that are exchanged between Construct services and applications take the following format: [3 bytes][10 bytes][payload]

  • The first 3 bytes correspond to the message type identifier (see below).

The next 10 bytes correspond to the length of the message payload in bytes. For example, a message with a 64 byte payload would be written as 0000000064.

  • Finally, the message payload consists of a number of bytes indicated by the previous part of the message. The contents of the payload are protocol specific.

The message identifier codes used are as follows:

The protocol identifier for querying QUERY 200
The protocol identifier used for responding to queries QUERY_RESPONSE 210
The protocol identifier used for sending rdf statements to the data port RDF_ADD 100
The protocol identifier used for responding to an add rdf statements request RDF_ADD_RESPONSE 110
The protocol identifier used for sending a service descriptor SERVICE_DESCRIPTOR_RESPONSE 310

When most service send a response to the client, the payload also takes the form of a code:

The operation succeeded OK 600
An error occured during the operation ERROR 610
The service code was unrecognised UNKNOWN 650

Working with the discovery service

Open a connection to the host/port given in the bonjour resolution (there is no need to send any data). You will be sent an XML descriptor file of the form -

  1. <services>
  2. <servicecomponentdescriptor>
  3. <name>Construct DataPort</name>
  4. <description>Raw data port: Connect via a socket and send N-TRIPLE RDF strings. Response string will be ok or error if it fails.</description>
  5. <host>erdinger</host>
  6. <port>3528</port>
  7. <misc>See example applications.</misc>
  8. </servicecomponentdescriptor>
  9. <servicecomponentdescriptor>
  10. <name>Construct QueryService</name>
  11. <description>The query service: Connect via a socket and send SPARQL queries. Response string will be a SPARQL result set in RDF.</description>
  12. <host>erdinger</host>
  13. <port>3531</port>
  14. <misc>See example applications.</misc>
  15. </servicecomponentdescriptor>
  16. </services>

This provides you with all the information required to open a connection to the data port or query service directly.

Working with the data port

Use the protocol described above. This is an example of adding a single line:

  • 1000000000043<http://hello> <http://construct> "world" .

NOTE: The data RDF Triple must have the trailing full stop . Example responses might be:

  • 1100000000003600 (OK)
  • 1100000000003610 (ERROR)
  • 1100000000003650 (UNKNOWN)

Working with the query service

The point of contact for application developers to Construct is the Query Service. It takes a SPARQL query as an input. This query is run on the data store (a list of RDF triples). If the query is valid and answerable, a string of data will be returned. This section will try to help you write a SPARQL query, and make sense of the returned information.

Writing a SPARQL query

The presence of a query service implies that there must be something to query. The data in Construct is stored in the data store. All the data is represented in RDF triples. Here’s a few examples of these: For readability, we will replace the URIs with the prefixes “sighting:” and “person:”. NOTE: The prefixes in the query(below) is valid SPARQL syntax.

  1. PREFIX: sighting:http://srg.ucd.ie/construct/sighting#
  2. PREFIX: person:http://www.pervasive-ontologies.org/ontologies/context/person.owl#
  3. <sighting:Waldo109> <sighting:person> <person:Waldo>
  4. <sighting:Waldo109> <sighing:computer> <http://srg.cs.ucd.ie/construct/computer/waldo.ucd.ie>
  5. <sighting:Waldo109> <sighting:time> <2006-10-04T13:17:59Z>
  6. <sighitng:Waldo109> <sightingstatus> <Active>

These four statements represent a sighting from a computer activity sensor.

The Query

To retrieve information from construct, a query must be written in the SPARQL format. This has a similar “Select X From Y Where Z” form to an SQL query. Here is an example query which relates to the RDF Triples above. It finds all the times of sightings of Waldo(note the time triple above).

  1. String personQuery =  “PREFIX sighting:<http://srg.ucd.ie/construct/sighting#> “
  2. + “PREFIX user:<http://www.pervasive-ontologies.org/ontologies/context/person.owl#> “
  3. + “SELECT ?time “
  4. + “WHERE {”
  5. + “?sighting sighting:person user:Waldo . “
  6. + “?sighting sighting:status sighting:Active . “
  7. + “?sighting sighting:time ?time”
  8. + “}”;

For a more comprehensive tutorial on SPARQL, visit the website at: http://www.w3.org/TR/rdf-sparql-query/

In order to send a query to the query service we must use the protocol described above. Below is an example.

  • 200[payload length][SPARQL QUERY]
The Response

You will be sent back a string of the form

  • 210[payload length][SPARQL result set in RDF form]
  • OR 2100000000003610 (if an error occured)

This String is a SPARQL ResultSet. This is a valid RDF string, represented in N-TRIPLE format. This means that way in which it is processed is dependent on the RDF parsing capabilities of the language used. For example, in java, a new Jena model (or indeed a ResultSet object) can be created from it, and it can be then easily traversed.

Bookmark and Share
[del.icio.us] [Digg] [Google] [StumbleUpon] [Windows Live] [Yahoo!] [Email] More »
Powered by Bookmarkify™

Installing Bonjour for *nix Users

Published
by
graeme
on July 9, 2008
in Advanced and Construct internals
. 0 Comments

Every time Ubuntu’s Synaptic Package Manager updates the Avahi layer for Bonjour support it stomps on my Apple Bonjour install and breaks my Construct install. My guess is that the update puts Avahi first in the list of Bonjours to run when Construct starts. So I get this error:

*** WARNING *** The programme ‘java’ uses the Apple Bonjour compatiblity layer of Avahi.

*** WARNING *** Please fix your application to use the native API of Avahi!

*** WARNING *** For more information see

*** WARNING *** The programme ‘java’ called ‘DNSServiceQueryRecord()’ which is not supported (or only supported partially) in the Apple Bonjour compatiblity layer of Avahi.

*** WARNING *** Please fix your application to use the native API of Avahi!

*** WARNING *** For more information see

One way to address this is just to turn off the Avahi warning. Set an environmental variable list this export AVAHI_COMPAT_NOWARN=1 This isn’t a fix though it is just hiding the warning.

The best I have figured out so far is just to reinstall Bonjour:

Instructions for Installing Apple’s Bonjour on *nix

  1. Download the latest version of the Bonjour source code here.
  2. Decompress the downloaded archive
  3. cd to the mDNSPosix directory below the mDNSResponder-xxx directory (where xxx is the Bonjour version number).
  4. Edit Makefile and change line JDK = /usr/jdk to make it point at your Java installation (e.g., /usr/lib/jvm/java-6-sun)
  5. Type sudo make os=linux Java
  6. Type sudo make os=linux install — if you have problems with this, see below. Most likely you will need to install a specific version of gcc
  7. Copy the java specific files to your jre lib’s ext directory (cp build/prod/* /path/to/jre/lib/ext/)
  8. Open up /etc/nsswitch.conf and ensure that the ‘mdns’ switch appears on the “hosts:” line. My “hosts:” line looks like this:
hosts:          files dns mdns

Remember, whenever you use a new jre, make sure to include dns_sd.jar (Bonjour JAR file) in your jre/lib/ext directory

Problems with the install

Problems with stdlib.h

Lorcan had some problems with this install on Ubuntu (Edgy). He says: when running sudo make os=linux Java I got a lot of errors, starting with ../mDNSShared/dnssd_clientlib.c:71:20: error: stdlib.h: No such file or directory.

I installed the libc6-dev package. When I tried again everything worked fine

Problems with __stack_chk_fail_local

I had some major problems with this install on Ubuntu (Edgy), specifically when using gcc4.1 (to find out which version of gcc you are using type gcc --version).

lorcan@comp:~/Desktop/mDNSResponder-107.6/mDNSPosix$ sudo make os=linux install
Stopping Apple Darwin Multicast DNS / DNS Service Discovery daemon: mdnsd.
cp build/prod/mdnsd /usr/sbin/mdnsd
/usr/sbin/mdnsd installed
cp mdnsd.sh /etc/init.d/mdns
chmod ugo+x /etc/init.d/mdns
/etc/init.d/mdns start
Starting Apple Darwin Multicast DNS / DNS Service Discovery daemon: mdnsd.
ln -s -f /etc/init.d/mdns /etc/rc2.d/S52mdns
ln -s -f /etc/init.d/mdns /etc/rc3.d/S52mdns
ln -s -f /etc/init.d/mdns /etc/rc4.d/S52mdns
ln -s -f /etc/init.d/mdns /etc/rc5.d/S52mdns
ln -s -f /etc/init.d/mdns /etc/rc0.d/K16mdns
ln -s -f /etc/init.d/mdns /etc/rc6.d/K16mdns
/etc/init.d/mdns installed
cp build/prod/libdns_sd.so /usr/lib/libdns_sd.so.1
ln -s -f /usr/lib/libdns_sd.so.1 /usr/lib/libdns_sd.so
/usr/lib/libdns_sd.so.1 /usr/include/dns_sd.h installed
/usr/share/man/man8/mdnsd.8 installed
make[1]: Entering directory `/home/lorcan/Desktop/mDNSResponder-107.6/Clients’
mkdir build
cc dns-sd.c -L../mDNSPosix/build/prod/ -ldns_sd -I../mDNSShared -o build/dns-sd
/usr/bin/ld: build/dns-sd: hidden symbol `__stack_chk_fail_local’ in /usr/lib/libc_nonshared.a(stack_chk_fail_local.oS) is referenced by DSO
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: ld returned 1 exit status
make[1]: *** [build/dns-sd] Error 1
make[1]: Leaving directory `/home/lorcan/Desktop/mDNSResponder-107.6/Clients’

make: *** [../Clients/build/dns-sd] Error 2

Solution:
If the version of gcc is not 4.0.x Follow these instructions (EXACTLY!!!) to correct the problem:

  1. Ensure that gcc version 4.0.x is installed. This is a little tricky as the synaptic package manager stopped supporting this version of gcc since edgy. You’ll have to download the following packages:  gcc-4.0-base, cpp-4.0 , gcc-4.0
  2. Open the terminal and cd to where you saved the package.
  3. Do the following for each of the three packages: dpkg -i PACKAGE.deb Install them in the same order the are above. gcc base first, then cpp, then gcc — This should put gcc-4.0 into your /usr/bin directory.
  4. cd /usr/bin
  5. sudo mv gcc gcc-backup
  6. sudo ln -s gcc-4.0 gcc

Now completely rerun the instructions above (at the top of this page) to install Bonjour (you must completely rerun them from step 2). When installation is complete continue these instructions.

  1. sudo cd /usr/bin
  2. sudo rm gcc
  3. sudo mv gcc-backup gcc

Now gcc should point to the original version of gcc (i.e. that version that you were using before you installed Bonjour).

Bookmark and Share
[del.icio.us] [Digg] [Google] [StumbleUpon] [Windows Live] [Yahoo!] [Email] More »
Powered by Bookmarkify™

HTTP Port - Part 2

Published
by
steve
on July 9, 2008
in Construct internals
. 0 Comments

In part 1 of this post I described the new HTTP Port in Construct. In this part of the post I’ll explain how to write a web form and style the return values.

SPARQL Query

You can send a query to Construct with GET or POST. Here is how we do it in HTML with GET:

  1. <form method=“GET” action=“http://duvel.ucd.ie:8888/”>
  2. <textarea name=“q” cols=“64″ rows=“10″>SELECT ?subject ?predicate ?object WHERE {?subject ?predicate ?object}</textarea><br/>URL of XSLT to apply (optional):<br/><input type=“text” name=“xsl” size=“56″ value=“http://www.construct-infrastructure.org/stylesheets/default.xsl type=”submit” value=”Submit Query“>
  3. </form>

the only thing the HTTP Port wants is the query field to be called either “q” or “query”. The returned XML will be sent back to your browser. Something like this:

  1. <?xml version=“1.0″?>
  2. <?xml-stylesheet type=“text/xsl” href=“http://duvel.ucd.ie:8888/xsl/www.construct-infrastructure.org/stylesheets/default.xsl”?>
  3. <sparql
  4. xmlns:rdf=“http://www.w3.org/1999/02/22-rdf-syntax-ns#”
  5. xmlns:xs=“http://www.w3.org/2001/XMLSchema#”
  6. xmlns=“http://www.w3.org/2005/sparql-results#”>
  7. <head>
  8. <variable name=“subject”/>
  9. <variable name=“predicate”/>
  10. <variable name=“object”/>
  11. </head>
  12. <results>
  13. <result>
  14. <binding name=“subject”> <uri>http://www.pervasive-ontologies.org/ontologies/sensors/bluetooth#reading00:19:63:96:56:01@00:80:98:94:AE:4B@1201631502</uri>
  15. </binding>
  16. <binding name=“predicate”>
  17. <uri>http://www.pervasive-ontologies.org/ontologies/sensors/bluetooth#spotted</uri>
  18. </binding>
  19. <binding name=“object”>
  20. <literal>00:19:63:96:56:01</literal>
  21. </binding>
  22. </result>
  23. </results>
  24. </sparql>

To make the results look neater you can apply and XSL stylesheet. Send the URL for the XSL stylesheet as a field “xsl” in the form. This stylesheet must be web accessible. We’ve made a default example stylesheet you can use to get started.

Inserting N-3 RDF Data

You can insert new data into Construct using a form like this (we’re using POST this time for variety but GET works too):

  1. <form method=“POST” action=“http://duvel.ucd.ie:8888/”>
  2. <textarea name=“i” cols=“64″ rows=“10″><http://www.pervasive-ontologies.org/ontologies/sensors/bluetooth#reading00:19:63:96:56:01@00:80:98:94:AE:4B@1201631502><http://www.pervasive-ontologies.org/ontologies/sensors/bluetooth#spotted>“00:19:63:96:56:01″.</textarea><br/>Expiry time for data (optional):<br/><input type=“text” name=“expiry” size=“15″ value=“20000″><br/><input type=“submit” value=“Submit Data”>
  3. </form>

in this form you send the data to be inserted as a field “i” or “insert”.

The returned data from an insert is an XML document with a status code and any messages given back from Construct. It looks something like this:

  1. <response>
  2. <status>OK</status>
  3. <description>The data <http://www.pervasive-ontologies.org/ontologies/sensors/bluetooth#reading00:19:63:96:56:01@00:80:98:94:AE:4B@1201631502><http://www.pervasive-ontologies.org/ontologies/sensors/bluetooth#spotted>“00:19:63:96:56:01″. with given expiry 20000 was passed to the Construct data store without error.
  4. </description>
  5. </response>

Well, that is about it. You can make calls to Construct over HTTP from browsers or web applications using the HTTP Port. Have fun.

Bookmark and Share
[del.icio.us] [Digg] [Google] [StumbleUpon] [Windows Live] [Yahoo!] [Email] More »
Powered by Bookmarkify™

HTTP Port - Part 1

Published
by
steve
on July 9, 2008
in Construct internals
. 2 Comments

The latest release of Construct has the new HTTP Port enabled by default. The HTTP Port has similar functionality to the data port — you can use it to insert and query for data in Construct. The difference is that the data port works over raw TCP sockets, whereas the HTTP Port works over HTTP.

Now we can query Construct through a web browser and web apps… neat, huh?

First off, the construct.properties file now contains:

  1. <!– http port –>
  2. <component interface=“org.construct_infrastructure.component.httpport.HttpPort” implementation=“org.construct_infrastructure.component.httpport.HttpPortImpl”>
  3. <property name=“port” value=“8888″ />
  4. </component>

these lines activate the HTTP Port on port 8888. When you have Construct running, open your web browser and go to http://localhost:8888/ and you should a simple set of forms for querying and inserting data using HTTP GET and HTTP POST. It looks something like this:

Screenshot of the HTTP Port

From this page you can send RDF data in N-3 format or SPARQL queries to Construct. Try clicking on one of the buttons to test it out.

This is the end of Part 1 on the HTTP Port. Part 2 of this post will explain how to build your own web forms and use of XSL stylesheets for your results.

Bookmark and Share
[del.icio.us] [Digg] [Google] [StumbleUpon] [Windows Live] [Yahoo!] [Email] More »
Powered by Bookmarkify™

Building a Client Library for Construct

Published
by
steve
on July 9, 2008
in Construct internals
. 0 Comments

This post contains a guide for building a data port and query service client proxy for Construct in a language of your choice. The purpose of proxy objects is to disguise as much as possible the distributed nature of Construct from the developer — taking care of locating and accessing Construct services over the network.

The following sections describe (at a high level) the steps involved in building proxies for the data port and query service. If you plan to implement a client proxy, take a look at the Java, Python, and Ruby code in the codebase for examples. See the Construct protocols on the tutorial page for protocol details.

Layer 1: Finding and Connecting to Services - Connect to the Construct Bonjour Proxy

The Construct Bonjour Proxy maintains a list of active Construct instances that it can see. Assuming you have set it up on your machine, it will be running on port 3826. Connect to this port to be sent a list of service descriptors.

Layer 2: Connecting to a Service

Assuming that some services are present, pseudocode for this process is a follows:

  • iterate through each available service instance
    • if the service descriptor contains the desired service, attempt to create a connection to it
    • if successful, return the connection. If not, continue the iteration

If no suitable services are available, you may wish to throw an exception.

Layer 3: Wrapping the socket in an appropriate client object

The final step is to provide code that can be wrapped round the socket to handle interaction with the services and client. Code is required for each Construct service that the client requires to access (the data port and the query service).

Retain a reference to the socket as it can be reused for multiple interactions. You should only need to obtain a new set of service descriptors when your connection fails.

Below are examples of the sort of functionality you may wish to provide in your proxy:

The Data Port proxy

Methods:

  • Constructor (boolean scan_localhost_only)
    • start the process to scan for Construct instances
  • add(String rdf)
    • Take RDF statement(s) in String format
    • Create a connection to a data port (see Layer 2)
    • Format the data using the correct protocol (see Construct protocols tutorial) and send it to the data port
    • Get the response from the data port (using the correct protocol) and return true/false to indicate successful submission
  • add(String rdf, long expiryTime)
    • As above, but add the expiry time to the end of concatenated N-Triple string
  • add(LanguageSpecificConstruct model)
    • Convert the (Jena, rdflib etc.) model to N-Triples and continue as add(String rdf)
  • add(Model model, long expiryTime)
    • Convert the (Jena, rdflib etc.) model to N-Triples and continue as add(String rdf, long expiryTime)
  • Destructor
    • stop the process scanning for Construct instances

The Query Service proxy

Methods:

  • Constructor (boolean scan_localhost_only)
    • start the process to scan for Construct instances
  • query(String sparqlQuery)
    • Takes in a String containing a SPARQL query
    • Create a connection to the query service (see Layer 2), format the query using the correct protocol (see Construct protocols tutorial page) and send the query.
    • Get the response from the data port (using the correct protocol)
    • If an error occurred, return null
    • If successful, transform the RDF into the appropriate language specific construct representing a SPARQL result set (Jena, rdflib etc.)
  • Destructor
    • stop the process scanning for Construct instances

[adapted from original material by Graeme]

Bookmark and Share
[del.icio.us] [Digg] [Google] [StumbleUpon] [Windows Live] [Yahoo!] [Email] More »
Powered by Bookmarkify™

How to build a new component

Published
by
steve
on July 9, 2008
in Construct internals
. 0 Comments

Overview

In this post we’ll describe the construction and runtime operation of Construct Components. These are the building blocks that combine to form the Construct infrastructure. If you want to modify how Construct works internally (i.e. write a new data dissemination mechanism, change the internal data model, add a reasoner…) then this is the place to start.

The number and type of components in any given deployment is customisable. At runtime, each of the components defined in the construct.properties file is loaded and instantiated by the Component Manager. The Component Manager is also responsible for the correct shutdown of the components when the software is terminated.

In order to facilitate this process, components must implement two interfaces. The first interface is the ConstructComponent interface, which defines all the methods required for communication between a component and the Component Manager. The second interface is component specific, and is used to register the component with Construct, and allow other components to gain access to it. A default implementation of the ConstructComponent interface, the AbstractConstructComponent, is provided in order to simplify the process of creating new components. Developers may extend from this class if desired, or implement directly from the ConstructComponent interface.

The ConstructComponent Interface

The ConstructComponent interface extends the Runnable interface and defines 6 additional methods that all components must implement in order that the Component Manager can talk with them. Below is an overview of each of these methods.

  • shutdownComponent(): Called by the Component Manager in order to instruct the component to shutdown. Implementing components should ensure that this method performs any cleanup operations and then terminates.
  • getGraphicalInterface(): Gives access to the graphical interface defined by this component. The default value is null – meaning no interface provided
  • setGraphicalInterface(): Sets the graphical interface for this component. Null values are valid and indicate that no interface exists.
  • setupComponentLinks(): This template method is where the component should access the Component Registry in order to retrieve references to other components that it talks to. The Component Manager calls this method after instantiating all components, but before running their threads.
  • getLogger(): This method gives access to the logger for the component. Subclasses should call this method in order to report information, warnings, or errors.

Of these methods, a component should only require to call the setLogger(), getLogger() and setGraphicalInterface() methods. The rest of the methods are called by the Component Manager when required.

The AbstractConstructComponent Class

The AbstractConstructComponent class provides the expected implementation for the get/set methods defined within the ConstructComponent interface and provides a default (blank) implementation for the setupComponentLinks() method.

In order to keep the component thread alive, the run() method waits until a notification is triggered by a call to the shutdownComponent() method (which sets a flag to indicate that the component should terminate)

In addition to providing a default implementation for the methods defined in the ConstructComponent interface, the AbstractConstructComponent class defines two template methods for developer use: the onRun() method which is called when the component’s thread is started, and the onShutdown() method which is called when the component is requested to shutdown. For both methods, implementing classes should perform any necessary actions and ensure that the method returns normally.

The Component Manager

The Component Manager is responsible for loading, running, and stopping all the components in any given Construct deployment. The set of components to be started is defined in an external file (construct.properties) which is read by the Component Manager. The startup process is as follows:

  1. Initialise all the components. For each component, this involves making a call to the Component Registry, which is responsible for ensuring that only one copy of a component exists at any given time. If the component does not exist, it is created (using reflection). If the component already exists, a reference to the existing component is returned.
  2. Obtain and display the graphical interface for each component. This calls the getGraphicalInterface() method for each component, and adds the return value (if not null) to a tab in the main GUI.
  3. Allow components to interact. Next, the setupComponentLinks() method of each component is called in order that they may obtain references to the other components that they interact with from the Component Registry. Should any requested component be unavailable, the initialisation process will be terminated.
  4. Start the components. Finally, the thread for each component is started.

When shutting down a Construct deployment, the Component Manager iterates through all active components, calling their shutdownComponent() method. Once completed, the Component Manager removes the entry from the Component Registry.

The construct.properties File

The construct.properties file in the Construct root directory specifies the components they should be loaded, and the properties that should be passed to components on startup. The construct.properties file also contain properties that can be used to refine logging details. An example file construct.properties.tmp is included with the distribution. You should copy this file to construct.properties before running Construct.

Component entires in the file take one of two forms depending on whether the component requires properties on startup.

Components without properties are specified as follows:

<component interface="[ComponentInterfaceName]” implementation=”[ImplementationClass]” />

Components with properties are specified as follows:

<component interface="[ComponentInterfaceName]” implementation=”[ImplementationClass]” />
<property name="[PropOneName]” value=”[PropOneValue]” />
<property name="[PropTwoName]” value=”[PropTwoValue]” />
</component>

[adapted from original material by Graeme]

Bookmark and Share
[del.icio.us] [Digg] [Google] [StumbleUpon] [Windows Live] [Yahoo!] [Email] More »
Powered by Bookmarkify™