Metadata/UCD tutorial

Introduction

The goal of this tutorial is to understand where UCDs can be used in the VO, how to manipulate them, attach metadata to tabular data, and build services to automate metadata management.

Because UCD1+ are not yet widely used in the VO, we will have to play with both old-style UCD1 and the new recommended UCD1+: we will take this opportunity to show how to migrate to the newer version.

For exercises requiring programming, places where the source code is to be modified/completed are indicated by the CHANGE_ME keyword. Other parts of the source code shoudn't need changes.

1 UCD documentation and tools

UCD1+-related documentation and tools can be found online http://cdsweb.u-strasbg.fr/UCD/.

UCDs are mainly used for describing the contents of columns in VOTable documents, with a ucd="" attribute in the FIELD element.

The first step for data providers is to identify the relevant UCDs describing the data they want to publish to the VO.

1.1 Manual search

Try to find the relevant UCDs for the following quantities, using the UCD builder (http://cdsweb.u-strasbg.fr/UCD/cgi-bin/descr2ucd):

right ascension
declination
K magnitude
heliocentric redshift
trigonometric parallax

Feel free to try more complex queries!

1.2 Using Web Services

For large collections, it is desirable to automate the process of finding UCDs corresponding to descriptions. We can use the assign method of the UCD Web Services (http://cdsweb.u-strasbg.fr/cdsws/ucdClient.gml).

Edit the PERL script src/assign.pl to find UCDs corresponding to the descriptions in the file data/descriptions.dat. You can use hints from http://cdsweb.u-strasbg.fr/cdsws/tucdClient2.gml. Run your script to see the result.

An example solution can be found in src/assign_solved.pl.

2 Locating resources in a registry

If registry entries (see example in appendix A) contain a description of their contents with UCDs, one can easily locate which resources contain some specific measurements. We will perform such a content-based search of resources, using the AstroGrid Hydra registry (or NVO's Carnivore registry, which would return the same result).

The UCD information is stored at the following location in the VOResource schema: vs:table/vs:column/vs:ucd (see example in appendix A).

Note: the registry entries currently contain UCD1, not UCD1+. Therefore, we will have to use UCD1 to make the query. In the application below, use POS_PARLX_TRIG and PHOT_JHN_K.

Go to the advanced query form http://hydra.star.le.ac.uk:8080/astrogrid-registry/sadqlquery.jsp
Select version 0.10, and search resources containing both a K magnitude measurement and parallaxes (the SQL query below can be found in the file data/query_hydra.txt -- simply copy/paste it in the SQL part of the form, but NOT in the ADQL upload).
```
select * from Registry where 
        vs:table/vs:column/vs:ucd = "POS_PARLX_TRIG" 
        and 
        vs:table/vs:column/vs:ucd = "PHOT_JHN_K"
```

Alternatively, one could do the same with the NVO Carnivore registry, located at http://mercury.cacr.caltech.edu:8080/carnivore/advancedquery, using the XQuery in file data/query_carnivore.txt).

declare namespace vr = "http://www.ivoa.net/xml/VOResource/v0.10"; 
for $vr in //vr:Resource 
        where $vr//vs:column/vs:ucd = 'POS_PARLX_TRIG' 
        and $vr//vs:column/vs:ucd = 'PHOT_JHN_K' 
        return $vr//vr:interface[@xsi:type="vr:WebBrowser"]

The result should consist of 11 tables which contain both a measurement of a trigonometric parallax, and a K-band magnitude, among over 11,000 resources in the registry

3 Upgrading a VOTable document

We have extracted 3 complete tables from the above matching resources.

J/AJ/123/2806/table3 : data/AJ_123_2806_table3.xml
J/AJ/123/2822/table1 : data/AJ_123_2822_table1.xml
J/A+A/393/183/catalog : data/A+A_393_183_catalog.xml

You can notice that the contents of the ucd attribute of the FIELD elements are UCD1. We will convert these into UCD1+ using the translate method of the UCD Web Services. We will do so from a Java application this time, using the SAVOT parser to handle the VOTable structure.

Launch eclipse for editing a new java project
Include the package src/UCD_pkg and the file src/Convert.java (from within eclipse, or simply from copy these files into your workspace/project directory, and press F5 to update your project)
Include links to the SAVOT jar files (src/savot/*) and AXIS libraries in the project properties
Edit the file src/Convert.java to do the translation from UCD1 to UCD1+ (see http://cdsweb.u-strasbg.fr/cdsws/tucdClient.gml for an example of Web Service call in java).
Define the program arguments (simply 2 arguments: inputFile.xml outputFile.xml), and run the program to convert each of the three tables stored in your data directory, and save the new tables.

Check the contents of the output to see that UCD1 have been replaced by the corresponding UCD1+.

An example program solution can be found in src/Convert_solved.java

4 Using UCDs in applications

4.1 Use in Aladin

You can create in Aladin some filter planes (documentation: http://vizier.u-strasbg.fr/boch/doc/filters.htx) that will alter the display of a catalogue plane. Using UCDs instead of column names in the filter syntax, one can build generic filters that will work with tables from different origins (different column names).

You can see such a filter in data/magplx_ucd1+.ajs. This one will create symbols with a size proportional to the magnitude, and color corresponding to the distances.

Launch Aladin
Load > MyData > browse and load data/allsky.fits for a nice background
Load > MyData > browse and load some of the tables with UCD1+ that we just created
Load > MyData > browse data/magplx_ucd1+.ajs
You can select some of the sources to see the measurements.

4.2 Create your own application

It is possible to use UCDs to easily build services transforming VOTable documents. We will try, as an example, to add a column containing the distance (d) to a star for a VOTable document containing a trigonometric parallax (π).

d (pc) = 1/π(arcsec)

Note: you can of course do something more sophisticated, compute the associated error, etc!

The program will take as an input a VOTable document having a column being a trigonometric parallax. The program will identify automatically this column using the UCD.
The output will be new VOTable with one extra column: the distance d.

The idea is that the UCDs allow us to locate unambiguously a given quantity in the table, whatever the column name.

Identify the UCDs for the quantities involved.
Edit src/Distance.java for performing the actions (creation of a new FIELD and computation of the TD values for the new column)
Run the program and check the result.

An example solution can be found in src/Distance_solved.java

A Example of registry entry

    <vor:Resource xmlns="http://www.ivoa.net/xml/VOResource/v0.10" 
        xmlns:cea="http://www.ivoa.net/xml/CEAService/v0.2" 
        xmlns:ceab="http://www.astrogrid.org/schema/CommonExecutionArchitectureBase/v1" 
        xmlns:ceapd="http://www.astrogrid.org/schema/AGParameterDefinition/v1" 
        xmlns:cs="http://www.ivoa.net/xml/ConeSearch/v0.3" 
        xmlns:sia="http://www.ivoa.net/xml/SIA/v0.7" 
        xmlns:vc="http://www.ivoa.net/xml/VOCommunity/v0.2" 
        xmlns:vg="http://www.ivoa.net/xml/VORegistry/v0.3" 
        xmlns:vs="http://www.ivoa.net/xml/VODataService/v0.5" 
        xsi:schemaLocation="http://www.ivoa.net/xml/VOResource/v0.10       
        http://www.ivoa.net/xml/VOResource/VOResource-v0.10.xsd                 
        http://www.ivoa.net/xml/VODataService/v0.5                 
        http://www.ivoa.net/xml/VODataService/VODataService-v0.5.xsd       
        http://www.ivoa.net/xml/ConeSearch/v0.3                 
        http://www.ivoa.net/xml/ConeSearch/ConeSearch-v0.3.xsd" 
        xsi:type="vs:TabularSkyService">
        <title>Photometry of southern NLTT stars (Reid+, 2002) - 
                Photometry and Astrometry of NLTT Stars</title>
        <shortName>J/AJ/123/2822/ta</shortName>
        <identifier>ivo://CDS/VizieR/J/AJ/123/2822/table1</identifier>

        <curation>
            <publisher ivo-id="ivo://CDS/VizieR">VizieR</publisher>
            <creator>
                <name>Reid I.N., Kilkenny D., Cruz K.L.</name>
            </creator>

            <contributor>Greg Schwarz [AAS], Patricia Bauer [CDS]</contributor>
            <date role="creation">2003-05-05T23:26:36Z</date>
            <version>21-Jun-2002</version>
            <contact>

                <name>CDS support team</name>
                <address>CDS, Observatoire de Strasbourg, 11 rue de l'Universite, 
                        F-67000 Strasbourg, France</address>
                <email>question@simbad.u-strasbg.fr</email>
            </contact>

        </curation>
        <content>
            <subject>Stars</subject>
            <subject>Photometry:wide-band</subject>
            <description>We present BVRI photometry of 180 bright, southern 
                nearby-star candidates. The stars were selected from the New Luyten 
                Two-Tenths proper-motion catalog (Cat. <I/98>) based on optical/infrared 
                colors, constructed by combining Luyten's m_r_ estimates with near-infrared 
                photometry from the Two Micron All Sky Survey (Cat. <B/2mass>) . 
                Photometric parallaxes derived from V-Ks, V-I, and I-J colors, combined 
                with the limited available astrometry, show that as many as 108 stars may 
                lie within 20pc of the Sun. Of these, 53 are new to nearby-star catalogs, 
                including three within 10pc of the Sun.</description>

            <source format="bibcode">2002AJ....123.2822R</source>
            <referenceURL>http://vizier.u-strasbg.fr/cgi-bin/Cat?J/AJ/123/2822/table1</referenceURL>
            <type>Catalog</type>
            <contentLevel>Research</contentLevel>

        </content>
        <interface xsi:type="vr:WebBrowser">
            <accessURL use="full">
            http://vizier.u-strasbg.fr/cgi-bin/VizieR-2?-source=J/AJ/123/2822/table1
            </accessURL>
        </interface>
        <interface qtype="GET" xsi:type="vs:ParamHTTP">

            <accessURL use="base">
            http://vizier.u-strasbg.fr/viz-bin/votable/-dtd/-A?-source=J/AJ/123/2822/table1
            </accessURL>
            <vs:resultType>text/xml+votable</vs:resultType>
        </interface>
        <vs:coverage>

            <vs:spectral>
                <vs:waveband>Optical</vs:waveband>
                <vs:waveband>Infrared</vs:waveband>
            </vs:spectral>
        </vs:coverage>

        <vs:table xmlns="http://www.ivoa.net/xml/VODataService/v0.5">
            <vs:name>J/AJ/123/2822/table1</vs:name>
            <vs:description>Photometry and Astrometry of NLTT Stars</vs:description>
            <vs:column>
                <name>recno</name>

                <description>Record number within the original table (starting from 1)</description>
                <unit/>
                <ucd>RECORD</ucd>
            </vs:column>
            <vs:column>

                <name>NLTT</name>
                <description>Designation from the NLTT <I/98> Catalogue (1)</description>
                <unit/>
                <ucd>ID_MAIN</ucd>

            </vs:column>
            <vs:column>
                <name>f_NLTT</name>
                <description>[1234, ] Flag on NLTT (2)</description>
                <unit/>

                <ucd>CODE_MISC</ucd>
            </vs:column>
            <vs:column>
                <name>AName</name>
                <description>Alternative name (3)</description>

                <unit/>
                <ucd>ID_ALTERNATIVE</ucd>
            </vs:column>
            <vs:column>
                <name>LHS</name>

                <description>? Designation from Luyten's LHS catalog</description>
                <unit/>
                <ucd>ID_ALTERNATIVE</ucd>
            </vs:column>
            <vs:column>

                <name>RAJ2000</name>
                <description>Hour of Right Ascension from 2MASS (J2000)</description>
                <unit>"h:m:s"</unit>
                <ucd>POS_EQ_RA_MAIN</ucd>

            </vs:column>
            <vs:column>
                <name>DEJ2000</name>
                <description>Degree of Declination from 2MASS (J2000)</description>
                <unit>"d:m:s"</unit>

                <ucd>POS_EQ_DEC_MAIN</ucd>
            </vs:column>
            <vs:column>
                <name>rmag</name>
                <description>The NLTT r-band magnitude</description>

                <unit>mag</unit>
                <ucd>PHOT_PHG_R</ucd>
            </vs:column>
            <vs:column>
                <name>Vmag</name>

                <description>The V-band magnitude</description>
                <unit>mag</unit>
                <ucd>PHOT_JHN_V</ucd>
            </vs:column>

            <vs:column>
                <name>B-V</name>
                <description>? The (B-V) color on the Johnson system</description>
                <unit>mag</unit>

                <ucd>PHOT_JHN_B-V</ucd>
            </vs:column>
            <vs:column>
                <name>V-Rc</name>
                <description>? The (V-Rc) color on the Cousins system</description>

                <unit>mag</unit>
                <ucd>PHOT_COUS_V-R</ucd>
            </vs:column>
            <vs:column>
                <name>V-Ic</name>

                <description>? The (V-Ic) color on the Cousins system</description>
                <unit>mag</unit>
                <ucd>PHOT_COUS_V-I</ucd>
            </vs:column>

            <vs:column>
                <name>Nobs</name>
                <description>? Number of optical observations</description>
                <unit/>
                <ucd>NUMBER</ucd>

            </vs:column>
            <vs:column>
                <name>Jmag</name>
                <description>The 2MASS J-band magnitude</description>
                <unit>mag</unit>

                <ucd>PHOT_JHN_J</ucd>
            </vs:column>
            <vs:column>
                <name>Hmag</name>
                <description>? The 2MASS H-band magnitude</description>

                <unit>mag</unit>
                <ucd>PHOT_JHN_H</ucd>
            </vs:column>
            <vs:column>
                <name>Kmag</name>

                <description>The 2MASS K_S_ band magnitude</description>
                <unit>mag</unit>
                <ucd>PHOT_JHN_K</ucd>
            </vs:column>

            <vs:column>
                <name>plx</name>
                <description>? Trigonometric parallax</description>
                <unit>mas</unit>

                <ucd>POS_PARLX_TRIG</ucd>
            </vs:column>
            <vs:column>
                <name>e_plx</name>
                <description>? Uncertainty in plx</description>

                <unit>mas</unit>
                <ucd>ERROR</ucd>
            </vs:column>
            <vs:column>
                <name>r_plx</name>

                <description>Source of the astrometry (4)</description>
                <unit/>
                <ucd>REFER_CODE</ucd>
            </vs:column>
            <vs:column>

                <name>Luyten</name>
                <description>NLTT original data</description>
                <unit/>
                <ucd>DATA_LINK</ucd>

            </vs:column>
            <vs:column>
                <name>2MASS</name>
                <description>results of 2MASS Catalog for this source</description>
                <unit/>

                <ucd>DATA_LINK</ucd>
            </vs:column>
        </vs:table>
    </vor:Resource>

B Related files

B.1 `data` directory

allsky.fits All-sky FITS image

descriptions.dat Text file with example plain text description of parameters, one per line

ucd1_magplx.ajs Aladin filter with UCD1 syntax

magplx_ucd1+.ajs Aladin filter with UCD1+ syntax

query_carnivore.txt Sample Carnivore Registry query (XQuery)

query_hydra.txt Sample Hydra Registry query (SQL)

A+A_393_183_catalog.xml Full table A+A/393/183 with UCD1 in FIELDS

A+A_393_183_catalog-new.xml Same as above with UCD1+

AJ_123_2806_table3.xml Full table AJ/123/2806 with UCD1 in FIELDS

AJ_123_2806_table3-new.xml Same as above with UCD1+

AJ_123_2822_table1.xml Full table AJ/123/2822 with UCD1 in FIELDS

AJ_123_2822_table1-new.xml Same as above with UCD1+

B.2 `src` directory

assign.pl PERL script (incomplete) for automated UCD assignment

assign_solved.pl Example solution for above exercise.

Convert.java Java program (incomplete) for translating UCD1 into UCD1+

Convert_solved.java Example solution for above exercise.

Distance.java Java program (incomplete) for adding a new column to a VOTable

Distance_solved.java Example solution for above exercise.

Workshop on VO Standards and Systems for Data Centres and Large Projects

Metadata/UCD tutorial

Contents:

Introduction

1 UCD documentation and tools

1.1 Manual search

1.2 Using Web Services

2 Locating resources in a registry

3 Upgrading a VOTable document

4 Using UCDs in applications

4.1 Use in Aladin

4.2 Create your own application

A Example of registry entry

B Related files

B.1 `data` directory

B.2 `src` directory

Workshop on VO Standards and Systems for Data Centres and Large Projects

Metadata/UCD tutorial

Contents:

Introduction

1 UCD documentation and tools

1.1 Manual search

1.2 Using Web Services

2 Locating resources in a registry

3 Upgrading a VOTable document

4 Using UCDs in applications

4.1 Use in Aladin

4.2 Create your own application

A Example of registry entry

B Related files

B.1 data directory

B.2 src directory

B.1 `data` directory

B.2 `src` directory