$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ Migrating From Previous Versions of Khoros

E.1 How do I migrate my Khoros 1 software to Khoros 2?

The Migration toolbox provides a number of programs that will help you in migrating Khoros 1 code to Khoros 2. See the Migration Manual or the man pages for these routines to find out how to use them. The key programs included in the Migration toolbox are:

kfix Performs any possible updates of source code from Khoros 1 library calls to Khoros 2

kfixpane Updates a Khoros 1 *.pane file to a Khoros 2 *.pane file

uisupdate Updates a Khoros 1 *.form file to a Khoros 2 *.form file

mkproto Creates ANSI prototypes for functions

kdarwin An application that provides a front end to the programs listed above (and others), in order to help evolve a Khoros 1 program to a Khoros 2 software object.

The conversion programs available in the Migration toolbox aid in the migration of K1 software to K2 software, but they cannot do everything; depending on the contents of your Khoros 1 toolbox, you may have to do more work or less work to upgrade your software. The following summaries give an idea of the migration process for each category:

Cantata Workspaces

There is no automatic conversion. You must start Cantata with an empty workspace, and recreate the K1 workspace using the K2 operator equivalents. The reason for this is that all the operators in Cantata have been rewritten under 2; names and arguments have changed so much that a migration script would have been extremely difficult to write, and results would have been uncertain at best. Also, Cantata itself has change considerably since Khoros 1.

Xvroutines

There is a bit of automatic conversion provided. Use uisupdate to convert *.form and *.pane to a Khoros 2 version, use the K2 software development tools to generate the program framework, and finally, re-code the application by hand. Kfix can help to upgrade Khoros 1 function calls and bsd/non-POSIX.1 OS calls to their K2 equivalent. Since the interface to the GUI and visualization capabilities of Khoros 2 is so different from that of Khoros 1, a migration script would have been extremely difficult to write, and results would have been uncertain at best. Also, the functionality provided by GUI and Visualization is much more comprehensive than that which was provided in Khoros 1.

Vroutines

Most of the process can be done with automatic conversion. Use kdarwin to convert the K1 vroutine to a K2 kroutine. Kdarwin takes a Khoros 1 *.prog and *.pane, and creates a Khoros 2 kroutine software object. This type of conversion is the most standardized.

Libraries

The amount of automatic conversion possible varies with library contents. If the files are l*.c's from Khoros 1 vroutines, simply use kdarwin on each vroutine (see above). Otherwise, you must create an empty library software object with the Khoros 2 Craftsman, and upgrade the code by hand. Kfix can help to move old Khoros 1 function calls and bsd/non-POSIX.1 OS calls to their K2 equivalent.

E.2 How do I migrate my Khoros 2.0.x software to Khoros 2.1?

You will need to run kmigrate (found in the MIGRATION toolbox) for any toolboxes you have developed with an earlier version of Khoros 2 in order to update them to use the improved Imake rules and API changes. In most cases, this will fully update your toolboxes and objects to function in the new system.

Warning:


You should not run kmigrate on any of the toolboxes in


this release (i.e. those delivered with Khoros 2.1).



It is a good idea to make backup copies of your toolboxes before running kmigrate. If there is a problem with your toolbox that kmigrate can not recover from, you will need your backup copy so that you can fix the problem in the original toolbox and rerun kmigrate.

An example of how to run kmigrate is:


> kmigrate -tb <toolbox1>,<toolbox2>,<toolbox3>


Note: Spaces are not allowed between toolbox names.

Some caveats to migration:

E.3 VIFF Khoros format replaced by Khoros Data Format (KDF)

Background

The Khoros 2.1 release contains a new standard file format for the Khoros system, the Khoros Data Format (KDF). This format is intended to replace the Khoros 2.0.x Viff format.

The K2 Viff was a modified gdbm database capable of storing data segments as well as arbitrary attributes. The format was too complex to be documented, given the dependence on the gdbm database, meaning that reading the Viff files outside of the Khoros system was not possible. Furthermore, there was too much overhead associated with the gdbm code, making this format one of our least efficient ones.

The new KDF format was designed to solve all these problems. It is as structurally flexible as the old K2 Viff, and is well defined. Efficiency and simplicity were the driving factors in its design. It has proven to be much more maintainable, and much more efficient than the K2 Viff was.

The Khoros 1 Xvimage format is now again called Viff. The old Khoros 2 VIFF has been renamed to Gdbm and is still supported for input.

Note:


Please use the kformats program in DATAMANIP to begin


migrating your Gdbm data files to KDF.



With data services, we've tried to abstract away the notion of file formats. Instead, we encourage people to access data in terms of the segments and attributes of a data model. You access your data with data services library calls which operate on an abstract data object. The data object can then be stored in any supported format.

Basically, the K2 VIFF was a database capable of storing arbitrary data segments of any dimensionality, size, and index order as well as arbitrary attributes. There is a lot of overhead associated with the K2 Viff, and not much of the system can use its flexibility. The overhead and flexibility is tied to the use of using the dbm access routines in which to store and retrieve data.

We have designed a new format which is structurally flexible (like the K2 Viff), but is more structured in terms of its storage (like the K1 Xvimage). We wanted something that was a cross between xvimage and viff.

Some of the design objectives for this format :

Flexible

Since data services can store arbitrary attributes and segments which means fixed formats (like xvimage, rast, xwd, gif) will not work. We need to have a format that can completely represent all attributes and segments.

Efficient

The K2 Viff is not very efficient. The other formats are fairly efficient but not flexible. For data processing routines that take a long time this is not really important, but for light weight processes that are processing small data sets, the overhead associated with Viff becomes fairly substantial. This becomes particularly noticeable in Cantata.

Maintainable

The K2 Viff has had some problems with machine compatibility and with memory management. We wanted a simple format which is easy to maintain. The dbm code has exhibited some memory leaks which have not been trackable.

Streamable

We wanted a format that can be streamed. None of the existing formats can be streamed. The K2 Viff is flexible, but its dependence on dbm and its random access capability means that it can't be streamed.

Large Data Sets

Only xvimage and viff were written so that they could do partial reads and writes. So another criteria was that the format be able to support partial reads and writes.

Minimal Storage Overhead

When storing small data sets both xvimage and viff have a minimal header size, even if the data set is to store a single value data point. To store a single byte (value only) data set created using kgconst the sizes are:

kdf: 173 bytes

viff: 1025 bytes

gdbm: 4379 bytes

KDF Format Specification

The Big Picture

The format for a KDF file consists of a header, a series of attribute blocks, and a series of data blocks. It can be pictured roughly as follows:


KDF Header               

Attribute Block for      
Object                   

Attribute Block for      
Segment 1                

Attribute Block for      
Segment 2                

...                      

Data Segment 1           
                         
                         
                         

Data Segment 2           
                         
                         
                         

...                      
                         
                         
                         



KDF Header

The KDF header consists of 7 bytes followed by three integers. Note that all information contained in a KDF file will be stored in a machine specific format. The seventh byte indicates the machine storage type for the particular file.


   4 bytes       2 bytes         1 byte       1 integer      1 integer    

   magic #      version #     machine type    # of data      # of attr.   
                                                 sets          blocks     



The magic number and version number: 01 03 19 94 00 02

The machine type follows the numbers defined in: $BOOTSTRAP/include/machine/kmachine.h

The number of data sets is currently hardwired to 1. This will be used in the future to store multiple data objects per file.

There is always at least one attribute block for storing the object level attributes. Beyond that, there is one attribute block for each segment in a data object. The attribute blocks follow sequentially from the header.

Attribute Blocks

The format for the beginning of an attributes block is:


     1 NULL-terminated string                   1 integer              

           segment name              number of attributes for segment  



The segment name will simply be a NULL ('\0') for the object level attribute block. The number of attributes describes the number of user-defined attributes for a segment and may be 0.

If the segment name is NULL, then none of the following information will be present. If the segment name is not NULL, then all the physical segment characteristics will be stored next. The segment characteristics are the dimension, datatype, size, and index order. They are stored as follows:


        1 integer          1 NULL-terminated string  

        dimension                  datatype          




                     dimension integers for size                      

     size[0]           size[1]              ...           size[dim-1]    




                    dimension integers for order                      

 index order[0]     index order[1]          ...              index       
                                                         order[dim-1]    




        1 integer                  1 integer         These should be          
                                                     hardwired to  -1         

     fixed dimension            dimension index      for now.  In the         
                                                     future, these            

                                                     fields will be used for  
                                                     storing                  
                                                     multi-resolution data.   



Datatypes are stored as a string representation of the datatype. The routine kdefine_to_datatype can be used to convert from the standard Khoros type identifiers to strings and the routine kdatatype_to_define can be used to convert the other direction. (Note: These routines may not be available in Khoros 2.0.2)

Valid datatypes are: bit, byte, unsigned byte, short, unsigned short, integer, unsigned integer, long, unsigned long, float, double, complex, and double complex.

Following this will be all other attributes for this segment. There will be exactly the number of attributes as was indicated at the top of the attribute block. Each attribute will follow the format described in the next section.


attribute 0              

attribute 1              

...                      



Attributes

The format for attributes is:


    1 string          1 integer         1 integer          1 string      

 attribute name    # of arguments     argument size        datatype      

                                                                      
                           attribute data                             
                                                                      

  end of attr.                                                           
    (eoa) tag                                                            



The attribute name is a string corresponding to the name of the attribute. The number of attribute arguments and the size of each arguments follow. The datatype of the attribute is stored next. Datatypes are stored in a string. The routine kdefine_to_datatype can be used to convert from the standard khoros type identifiers to strings and the routine kdatatype_to_define can be used to convert the other direction. (Note: These routines may not be available in Khoros 2.0.2)

The valid standard datatypes are: bit, byte, unsigned byte, short, unsigned short, integer, unsigned integer, long, unsigned long, float, double, complex, double complex, and string. Any arbitrarily-defined datatype is also allowed, although it is dependent on the datatype being defined in order to be read.

The attribute data follows, stored in a large contiguous block. The attribute data is followed by an end of attribute (eoa) tag.

The end of attribute tag is a string: "<>" . This tag is only really used when reading past attributes of an unknown data type (an undefined structure datatype for instance).

Data Segments

The data segments are stored following the attribute blocks in a series of blocks with all the data for that particular segment. There is one block per segment. The order of the data segment blocks must match the order of the corresponding attribute blocks.


data segment 1           
                         
                         

data segment 2           
                         
                         

...                      
                         
                         



The amount of data present for each segment must be consistent with the physical segment characteristics specified in the attribute block that corresponds to this segment.

For instance, a segment with the characteristics of dimension = 2, size = [32, 24], and datatype = "byte," dataset would be expecting 32x24 bytes of data to be stored for its segment data block.

Glossary

$

$BOOTSTRAP
Location of the bootstrap toolbox; not an environment variable that you should set.

$DATAMANIP
Location of the datamanip toolbox; not an environment variable that you should set.

$DATASERV
Location of the dataserv toolbox; not an environment variable that you should set.

$DESIGN
Location of the design toolbox; not an environment variable that you should set.

$ENVISION
Location of the envision toolbox; not an environment variable that you should set.

$GEOMETRY
Location of the geometry toolbox; not an environment variable that you should set.

$IMAGE
Location of the image toolbox; not an environment variable that you should set.

$MIGRATION
Location of the migration toolbox; not an environment variable that you should set.

$SAMPLEDATA
Location of the sampledata toolbox; not an environment variable that you should set.

$SUPPORT
Location of the support toolbox; not an environment variable that you should set.

A

alias
An alias created in the Khoros system is a way to define a shorthand for input file locations. For example, a file called ~scientist/data/images/chest.mri.2103x3000.short could be aliased to image:chest1 .

animate
Application for interactive animation of image sequences.

API
Application Programming Interface

AthenaIncludedir
The variable in the Site.arch that indicates where the Athena widget set includes are located

AthenaLibdir
The variable in the Site.arch that indicates where the Athena widget set library is located

B

Basic Services
Utilities located in the kutils library of the bootstrap toolbox, including a large number of utility functions to handle memory allocation, string manipulation, string parsing, and message reporting.

BinDir
The variable in the Site.arch that indicates the directory for Khoros binaries.

BOOTSTRAP
Lowest-level toolbox, containing base Khoros utilities such as installation script, Imakefile/Makefile generators, and foundation services libraries

C

Cantata
Khoros visual language; an advanced data flow visual language that supports coarse grain distributed processes and both stream and block data. Its visual hierarchy, iteration, flow control, and expression based parameters make it a powerful simulation and prototyping system.

CLUI
see Command Line User Interface

Command Line User Interface
The command line user interface (CLUI) is supported by all Khoros programs.

Conductor
Automatic code generation tool for generating code for interactive X Window System based Khoros applications

Craftsman
Software object editor; allows you to modify, debug, compile, and document software objects.

D

Data Services
Routines located in the kdataman, kappserv, and kdatafmt libraries of the dataserv toolbox, comprising a powerful system for accessing and manipulating data independent of format or type.

DATAMANIP
Toolbox containing data manipulation routines.

DATASERV
Toolbox containing data services

DESIGN
Toolbox containing software development system, visual language, GUI & visualization services.

DoFullWarn
The variable in the Site.arch that indicates whether all the compiler warnings should be displayed
when compiling

E

Editimage
Interactive image display, examination, and manipulation tool.

ENVISION
Toolbox containing data visualization applications.

Expression Services
A symbolic expression parser located in the kexpr library of the bootstrap toolbox, which may be used to evaluate mathematical equations and functions.

F

Foundation Services
A term used to encompass several distinct program services; together, these services fulfill all the requirements of the Khoros software infrastructure that do not deal with either data I/O or data display. Foundation services includes Basic, Expression, Math, Operating System, Software, and User Interface Services.

G

GEOMETRY
Toolbox containing 3D vector/polygon based visualization routines.

ghostwriter
Automatic code generation tool for generating code and documentation for all Khoros programs.

Graphical User Interface
The graphical user interface (GUI) utilizes the X Toolkit and one of the supported widget sets. Every program in Khoros has a graphical user interface; xvroutines have a graphical user interface of their own, which is displayed when they are executed from the command line; kroutines have a graphical user interface which is only displayed when they are accessed via Cantata, or when their graphical user interface is deliberately displayed with preview or Guise.

GUI
See Graphical User Interface

GUI & Visualization
Routines provided by all the libraries in the design toolbox, taken together. They include functions to create and maintain a GUI, plus a large variety of visual objects which can be used to display images, graphics, plots, etc.

Guise
Direct manipulation GUI design tool.

H

HasAthena
The variable in the Site.arch that indicates whether the system has the Athena widget set.

HasCenterLineC
The variable in the Site.arch that indicates whether to use clcc to compile the code.

HasCodeCenter
The variable in the Site.arch that indicates whether the system has the program codecenter.

HasCplusplus
The variable in the Site.arch that indicates whether the system has a C++ compiler.

HasFortran
The variable in the Site.arch that indicates whether the system has a FORTRAN compiler.

HasGcc
The variable in the Site.arch that indicates whether to use gcc to compile the code

HasGcc2
The variable in the Site.arch that indicates whether to use gcc 2.x or higher to compile the code.

HasLargeTmp
The variable in the Site.arch that indicates whether the system has a large TMPDIR or /tmp.

HasMotif
The variable in the Site.arch that indicates whether the system has the Motif widget set.

HasOlit
The variable in the Site.arch that indicates whether the system has the Olit widget set.

HasPureCov
The variable in the Site.arch that indicates whether the system has the program purecov.

HasPurelink
The variable in the Site.arch that indicates whether the system has the program purelink.

HasPurify
The variable in the Site.arch that indicates whether the system has the program purify.

HasQuantify
The variable in the Site.arch that indicates whether the system has the program quantify.

HasSBrowser
The variable in the Site.arch that indicates whether the system has the SPARCworks
development software.

HasSentinel
The variable in the Site.arch that indicates whether the system has the program sentinel.

HasSharedLibraries
The variable in the Site.arch that indicates whether the system supports shared libraries.

HasTestCenter
The variable in the Site.arch that indicates whether the system has the program testcenter.

I

IMAGE
Toolbox containing image processing routines.

K

karith1
Command to performs single-operand pointwise arithmetic operations.

karith2
Command to performs dual-operand pointwise arithmetic operations.

kbuild
Script to compile all libraries, programs, and documentation from the Khoros source code in your installation directory (This may take a long time.) Provided for use on non-supported platforms.

kconfigure
Script to set up user environments after installation.

kfft
Command to performs Fast Fourier Transforms (both forward and reverse).

kgenimake
Program to automatically generate an architecture-independent Imakefile for a software object.

kgenmake
Program to automatically generate an architecture dependent Makefile from an Imakefile; correct & current Makefiles are required by make to compile a software object.

kinstall
Script to copy, uncompress, and unpack all distribution files contained on the CD-ROM into your installation directory.

Khoros Consortium
A collaboration between universities, industry, government, and Khoral Research, Inc. to advance scientific software.

.khoros_env
The .khoros_env file typically contains settings for Khoros environment variables.

kman
Command to access to Khoros on-line help in the form of man pages.

L

LibDir
The variable in the Site.arch that indicates the directory for Khoros libraries

M

make
Program to compile a software object.

Math Services
Utilities located in the kmath library of the bootstrap toolbox, including machine independent implementations of common mathematical operations; offers a variety of useful extensions to the standard mathematical functions; designed to be highly portable and efficient.

MIGRATION
Toolbox containing utilities to migrate from Khoros 1.0.5 to Khoros 2.0, and from Khoros 2.0.x to Khoros 2.1.

MotifIncludedir
The variable in the Site.arch that indicates where the Motif widget set includes are located

MotifLibdir
The variable in the Site.arch that indicates where the Moth widget set library is located

O

OlitIncludeDir
The variable in the Site.arch that indicates where the Olit widget set includes are located

OlitLibDir
The variable in the Site.arch that indicates where the Olit widget set library is located

Operating System Services
Routines found in the kutils library of the bootstrap toolbox, which isolate Khoros from the operating system, and extend the capabilities of the operating system to support distributed computing.

S

SAMPLEDATA
Toolbox containing sample data for use with the Khoros software.

SentinelLibdir
The variable in the Site.arch that indicates where the system has the sentinel libraries

software services
Routines located in the kcms library of the bootstrap toolbox, designed to give the toolbox programmer a coherent view of the various components of a toolbox or software object.

spectrum
Application for interactive signal/image classification.

SUPPORT
Toolbox containing various & sundry software development utilities

T

Toolbox
A collection of programs and/or libraries that are managed as an entity.

U

UIS
User Interface Specification file which define a command line (CLUI) and/or graphical (GUI) user interface for Khoros programs.

User Interface Services
Private routines located in the kforms, kcodegen, and kclui libraries that support UIS interpretation, CLUI support, code generation, documentation generation, and UIS management.

V

visual language
A naturally visible approach to problem solving in which a visual program is described as a directed graph, where each node represents an operator or function and each directed arc represents a path over which data flows.

W

Widget
X Window System terminology for a Graphical User Interface component, such as a push-button, scroll-bar, or dialog box.

WidgetSet
The variable in the Site.arch that indicates which widget set to use by default to compile xvroutines

X

X11IncludeDir
The variable in the Site.arch that indicates where the X11 includes are located

X11LibDir
The variable in the Site.arch that indicates where the X11 libraries are located

xprism
Application for 2D and 3D plotting.